Quality Checks Overview

Quality checks are automated guardrails that run before Sidekick's response is sent to the customer. They're designed to catch messages that don't meet predefined conditions and ensure Sidekick only sends accurate, helpful, and appropriate responses to your Customers.

How Quality Checks Work

1. A Customer sends a message

2. Sidekick generates a response

3. Quality Check runs: The response undergoes automated quality checks

4. Pass or fail:

   1. If the response passes all checks, it's sent to the Customer

   2. If the response fails any check, Sidekick hands off to an Agent instead

Quality Checks include a mix of:

• LLM-based checks: Use AI to evaluate response quality and accuracy

• Code-based checks: Use programmatic rules to catch specific issues

Quality Check categories

When you review Sidekick Conversations, Quality Checks are organized into three Customer-facing categories. Each category represents multiple internal algorithms working together to ensure response quality.

How Quality Checks are categorized

Quality Check algorithms

Missing Action check

These checks ensure Sidekick doesn't make claims about actions it can't actually perform.

1. Implied Mutation (LLM)

Purpose: Detects if Sidekick implies it changed something without actually doing it.

How It Works: Reviews the response against actual system actions that occurred and determines whether Sidekick claimed it did something it didn't actually do.

Failure Examples

• "Your order has been cancelled" (when no cancel action ran)

• "I've updated your address" (when no update action ran)

2. Unfulfilled Action (LLM)

Purpose: Detects if Sidekick claims it will personally do something in the future without requiring Customer action.

How It Works: Evaluates whether the response promises future actions that Sidekick cannot complete on its own.

Failure Examples

• "I will send you an email, look out for it"

• "I will investigate this further"

Pass Examples

• "I can cancel your order" (capability statement)

• "Give me your order number and I'll look it up" (requires customer action first)

3. Transfer Claim (LLM)

Purpose: Detects instances where Sidekick responds that it will transfer the Customer to a human Agent, yet no handoff occurs.

How It Works: This is a specialized subset of Unfulfilled Action, focused specifically on transfer promises.

4. Invalid Phrase (Code)

Purpose: Blocks responses containing technical terms, system artifacts, or placeholder text that should never appear to customers.

How It Works: Code-based check that scans for configured phrases.

Failure Examples

• "[Your Name]" or other bracketed placeholders

• References to internal variables (e.g., "handoff_claim = true")

Missing Information check

1. Contact Us (LLM)

Purpose: Prevents redirecting Customers away from the current Conversation for support.

How It Works: Evaluates whether the response unnecessarily separates Sidekick from the brand's support team.

Failure Examples

• "Reach out to our customer service team"

• "Please start a new conversation with support"

Pass Examples

• Directs to specific resources (websites, email addresses, physical stores)

Why Missing Information check is important

Customers often see phrases like "contact customer service" on company websites. When

Sidekick searches these sites, it may repeat this language, but the Customer is already talking to customer

service! This check prevents that confusing experience.

Unverified Information check

This check ensure Sidekick only shares information that's supported by reliable sources.

• 1. Hallucination (LLM)

Purpose: Ensures Sidekick's statements are supported by available context.

Context sources for Hallucination Quality Check

• Action results from external apps

• Search results from knowledge sources

• Guide content

How It Works:

1. Splits the response into individual sentences

2. For each sentence, checks:

   1. Does it make a factual claim?

   2. Is that claim supported by the context (action results, search results, Guide content)?

3. Flags sentences that make unsupported factual claims

Failure Examples

• "Our return policy is 90 days" (when the actual policy is 30 days and this isn't in the context)

Auto-Correction

Auto-Correction is a default setting which reviews messages flagged as a Hallucination and attempts to “correct” the message before immediately handing off. Auto-Correction does the following:

• Removes hallucinated sentences (up to a configured threshold)

• Calls an LLM to fix grammar issues caused by removed sentences

• Integrates URL correction

• Re-checks the corrected response

• 2. Email Hallucination (Code)

Purpose: Prevents responses containing email addresses not found in available knowledge sources.

How It Works: Code-based check that extracts email addresses (matching pattern: something@something.something) and verifies they exist in:

• Customer-provided information

• Action results

• Search results

• Guide content

Failure Examples: Response includes any email address not present in the approved source material.

3. Phone Number Hallucination (Code)

Purpose: Prevents responses containing phone numbers not found in available knowledge sources.

How It Works: Code-based check that extracts phone numbers and verifies they exist in:

• Customer-provided information

• Action results

• Search results

• Guide content

Failure Examples: Response includes any phone number not present in the approved source material.

4. Invalid URL

Purpose: Validates URLs in responses and corrects broken links.

How It Works:

• Checks if URLs actually exist in the context

• Can automatically fix some broken URLs

How Quality Checks appear in the Conversation Review

When you review a Sidekick Conversation, Quality Check results appear in the reviewer within the flow of the Conversation Timeline.

Passed Quality Checks

What you'll see:

• "Quality check completed with no issues flagged"

• Categories displayed (Missing Action, Missing Information, Unverified Information)

Failed Quality Checks

What you'll see:

• "Missing action and unverified information quality checks flagged" (or whichever categories failed)

• Proposed Response: The message Sidekick wanted to send but couldn't

• Specific explanations for each failed category

• "Sidekick handed off the conversation due to flagged quality checks"