--- title: "Quality Checks Overview" slug: "quality-checks-overview" updated: 2026-02-21T21:15:55Z published: 2026-02-21T21:15:55Z --- > ## Documentation Index > Fetch the complete documentation index at: https://help.gladly.com/llms.txt > Use this file to discover all available pages before exploring further. # Quality Checks Overview Quality checks are automated guardrails that run before Gladly AI’s response is sent to the customer. They're designed to catch messages that don't meet predefined conditions and ensure Gladly only sends accurate, helpful, and appropriate responses to your Customers. ## How Quality Checks Work 1. A Customer sends a message 2. Gladly generates a response 3. **Quality Check runs:**The response undergoes automated quality checks 4. **Pass or fail:** 1. If the response **passes** all checks, it's sent to the Customer 2. If the response **fails** any check, Gladly hands off to an Agent instead > [!NOTE] > Quality Checks include a mix of: > > - **LLM-based checks**: Use AI to evaluate response quality and accuracy > - **Code-based checks:** Use programmatic rules to catch specific issues ## Quality Check categories When you review Gladly AI Conversations, Quality Checks are organized into three Customer-facing categories. Each category represents multiple internal algorithms working together to ensure response quality. ### How Quality Checks are categorized | Category | What It Checks | Algorithms | | --- | --- | --- | | **Missing Action** | Flags claims where Gladly will perform or has performed actions not outlined in the Guide | - Implied Mutation - Invalid Phrase - Unfulfilled Action - Transfer Claim | | **Missing Information** | Flags when Gladly lacks adequate information to respond to the Customer's situation | - Contact Us | | **Unverified Information** | Flags unverified information like dates, policies, or contact information | - Hallucination - Email and Phone Hallucinations - Invalid URL | ### Quality Check algorithms #### Missing Action check These checks ensure Gladly doesn't make claims about actions it can't actually perform. 1. **Implied Mutation (LLM)** **Purpose**: Detects if Gladly AI implies it changed something without actually doing it. **How It Works**: Reviews the response against actual system actions that occurred and determines whether Gladly AI claimed it did something it didn't actually do. **Failure Examples** - "Your order has been cancelled" (when no cancel action ran) - "I've updated your address" (when no update action ran) 1. **Unfulfilled Action (LLM)** **Purpose**: Detects if Gladly AI claims it will personally do something in the future without requiring Customer action. **How It Works:** Evaluates whether the response promises future actions that Gladly AI cannot complete on its own. **Failure Examples** - "I will send you an email, look out for it" - "I will investigate this further" **Pass Examples** - "I can cancel your order" (capability statement) - "Give me your order number and I'll look it up" (requires customer action first) 1. **Transfer Claim (LLM)** **Purpose:** Detects instances where Gladly AI responds that it will transfer the Customer to a human Agent, yet no handoff occurs. **How It Works:** This is a specialized subset of Unfulfilled Action, focused specifically on transfer promises. 1. **Invalid Phrase (Code)** **Purpose:** Blocks responses containing technical terms, system artifacts, or placeholder text that should never appear to customers. **How It Works:** Code-based check that scans for configured phrases. **Failure Examples** - "[Your Name]" or other bracketed placeholders - References to internal variables (e.g., "handoff_claim = true") #### Missing Information check 1. **Contact Us (LLM)** **Purpose:** Prevents redirecting Customers away from the current Conversation for support. **How It Works:** Evaluates whether the response unnecessarily separates Gladly AI from the brand's support team. **Failure Examples** - "Reach out to our customer service team" - "Please start a new conversation with support" **Pass Examples** - Directs to specific resources (websites, email addresses, physical stores) > [!TIP] > Why Missing Information check is important > > Customers often see phrases like "contact customer service" on company websites. When > > Gladly AI searches these sites, it may repeat this language, but the Customer is already talking to customer > > service! This check prevents that confusing experience. #### Unverified Information check This check ensures Gladly AI only shares information that's supported by reliable sources. - **1. Hallucination (LLM)** **Purpose:** Ensures Gladly AI’s statements are supported by available context. > [!NOTE] > Context sources for Hallucination Quality Check > > - Action results from external apps > - Search results from knowledge sources > - Guide content **How It Works:** 1. Splits the response into individual sentences 2. For each sentence, checks: 1. Does it make a factual claim? 2. Is that claim supported by the context (action results, search results, Guide content)? 3. Flags sentences that make unsupported factual claims **Failure Examples** - "Our return policy is 90 days" (when the actual policy is 30 days and this isn't in the context) > [!NOTE] > Auto-Correction > > Auto-Correction is a default setting which reviews messages flagged as a Hallucination and attempts to “correct” the message before immediately handing off. Auto-Correction does the following: > > - Removes hallucinated sentences (up to a configured threshold) > - Calls an LLM to fix grammar issues caused by removed sentences > - Integrates URL correction > - Re-checks the corrected response - **2. Email Hallucination (Code)** **Purpose:** Prevents responses containing email addresses not found in available knowledge sources. **How It Works:** Code-based check that extracts email addresses (matching pattern: something@something.something) and verifies they exist in: - Customer-provided information - Action results - Search results - Guide content **Failure Examples:**Response includes any email address not present in the approved source material. 1. **Phone Number Hallucination (Code)** **Purpose:** Prevents responses containing phone numbers not found in available knowledge sources. **How It Works:** Code-based check that extracts phone numbers and verifies they exist in: - Customer-provided information - Action results - Search results - Guide content **Failure Examples:**Response includes any phone number not present in the approved source material. 1. **Invalid URL** **Purpose:**Validates URLs in responses and corrects broken links. **How It Works:** - Checks if URLs actually exist in the context - Can automatically fix some broken URLs ## How Quality Checks appear in the Gladly AI Conversation Review When you review a Gladly AI Conversation, Quality Check results appear in the reviewer within the flow of the Conversation Timeline. ![Customer chat discussing priority shipping upgrade and quality check issues.](https://cdn.us.document360.io/7047b671-c4f2-4df0-bb0a-b9b511fd2452/Images/Documentation/Group 2675.png) ### Passed Quality Checks **What you'll see:** - "Quality check completed with no issues flagged" - Categories displayed (Missing Action, Missing Information, Unverified Information) ![Chat interface showing quality check results and missing information alerts.](https://cdn.us.document360.io/7047b671-c4f2-4df0-bb0a-b9b511fd2452/Images/Documentation/Group 2676.png) ### Failed Quality Checks **What you'll see:** - "Missing action and unverified information quality checks flagged" (or whichever categories failed) - **Proposed Response**: The message Gladly AI wanted to send but couldn't - **Specific explanations** for each failed category - "Gladly handed off the conversation due to flagged quality checks" ![Chat conversation discussing identity verification and missing information quality check.](https://cdn.us.document360.io/7047b671-c4f2-4df0-bb0a-b9b511fd2452/Images/Documentation/Group 2677.png)