Build a Signal

Navigate to Signals

From the left navigation, go to Gladly AI > Analyze > Signals. The Signals dashboard lists every Signal your team has created, split into Active and Archived tabs. Search by name, or filter by agent or Channel to narrow the view.

Gladly AI interface showing Signals section for analyzing agent sessions and conversation complexity.

Criteria Library — a library of the reusable criteria your org has created. Each entry shows its current version and when it was created.
New Signal — click here to start building a new evaluation job.

Create a new Signal

Click New Signal to open the builder. It's a three-step wizard: Define, Scope, and Run. Each step is its own screen with Back and Next controls, and a Signal summary at the bottom of the last step recaps what you've configured in plain language before you create it.

Dashboard displaying AI agent session quality metrics and options for new signal creation.

Step 1: Define your signal

Defining a signal with name, description, and evaluation criteria for customer feedback.

Name the outcome you want to measure, then add 1–5 criteria. Each one becomes a judgment applied to every session this Signal evaluates.

[A] Signal name — name the outcome you're measuring, e.g. "Sizing and fit questions."
[B] Description (optional) — a short note on what this Signal is for.
[C] Evaluation criteria — search or browse the library and click + to add up to 5 existing criteria, or Create new criteria to build one from scratch. Each criterion becomes a judgment applied to every evaluated session.

When browsing existing criteria to add, click the info icon next to any entry to open a Criterion details panel — a quick, read-only view of its description, labels, and full evaluation prompt, with an Edit criterion link if you need to make changes. This is the fastest way to confirm a criterion measures what you think it does before adding it to a Signal.

Step 2: Scope — choose which sessions to evaluate

Select evaluation options for AI sessions, focusing on all or specific sessions.

Evaluate all AI sessions (Recommended) — the most representative read, and the most flexibility to slice and dice the data later.
Narrow to specific sessions — focus on:
- Channels — Chat, Email, SMS (check any combination).
- Agents and guides — "Evaluate sessions from [agents] using [guides]," defaulting to all agents and all guides.
- Keywords (optional) — only evaluate sessions containing one of the keywords you add, separated by commas.

Step 3: Run — one-off or ongoing

There's no schedule to set beyond what you choose here. Pick how the Signal runs:

Options for running a signal, including one-off and ongoing sessions, with settings.

One-off — looks at and labels a window of sessions once, right when you create it, then stops.
- First run covers — From today, Last 7 days, Last 30 days, or Last 90 days.
- Sessions per run — the most sessions this run evaluates: 100, 1,000, or 10,000. Choose 100 to quickly test your Signal.
- Sampling rate — the percentage of matching sessions the run evaluates: 5%, 20% (the default), 50%, or 100%.
Ongoing — looks at and labels a window of sessions, then keeps it fresh on a cadence you pick.
- First run covers — From today, Last 7 days, Last 30 days, Last 90 days, or No backfill. This sets how much history the first run backfills so there's data on day one; every run after that covers whatever's new since the last one.
- Then refreshes — Daily or Weekly.
- Sessions per run and Sampling rate — the same options as One-off, applied to every recurring run.

Review the Signal summary at the bottom of the page, then click Create signal.

Build a criterion

If you clicked Create new criterion in Step 1, a three-step wizard opens: Describe, Labels, Review. You can go back at any point to refine your answers.

1. Describe what you want to measure

First, you’ll be prompted to answer "What do you want to measure?". Describe the quality aspect you want to evaluate in sessions, and be specific about what good vs. bad looks like. The AI uses your description to draft classification logic; no prompt engineering required.

Instructions for defining product sizing criteria and evaluating customer inquiries about fit.

Tips for good descriptions:
Be specific about what you're measuring (e.g., "politeness" rather than "overall quality").
Mention what success looks like.
Include any edge cases or exceptions the AI should acc

Click Generate. The AI drafts classification labels and an evaluation prompt for you to review.

2. Review the classification labels

The AI proposes mutually exclusive labels the criterion will use to classify each session, each with a name and a description of what qualifies for it. Review, edit, regenerate, or add labels as needed — up to 5 labels, each up to 30 characters.

Edit any label's name or description inline, click Regenerate for a fresh set, or add a custom label with + Add Label. Hover a label to reveal a trash icon and remove it.
Goal or Risk label — optionally mark one label as your Goal, the result you're aiming for, or as your Risk, the result you want to avoid. This doesn't change anything about how Signals evaluates sessions: it's a wording change only, so a negative outcome you're tracking reads as a Risk in reporting instead of being called a Goal. Gladly uses whichever one you mark to calculate the occurrence rate shown throughout reporting.
Color — pick a color for each label from the six swatches provided, for quick visual identification in results. The AI won't pick colors or a goal for you — set both yourself.
Label tips shown in the UI: use 2–5 labels that cover all possible outcomes, keep them mutually exclusive, and order them from best to worst (or positive to negative).

Reviewing classification labels for sizing questions and customer inquiries in a user interface.

3. Review & Create

The final step shows the full criterion before you save it: name, description, classification labels, and the AI-generated evaluation prompt (the exact instructions the AI follows against each session transcript). Review everything (you can edit any field, including the evaluation prompt itself) then click Create. The criterion is saved to your Criteria Library and becomes available to add to any Signal.

Manage your Criteria Library

Open Criteria Library from the Signals dashboard to see every criterion your org has created.

Gladly Signals interface showing session evaluation options and highlighted Criteria Library button.

Each row shows its version and creation date; click Use to add one to a new Signal, or the pencil icon to edit it.

Criteria list for Gladly with highlighted option to edit specific criteria.

Editing a criterion never rewrites history
Each criterion is versioned, and past runs keep the results from the version that was active when they ran. Only future runs pick up your edits, nothing re-runs automatically.

Best practices for building Criterion

Prefer fewer labels. Binary (Yes/No) or a short list of named categories (Positive/Neutral/Negative) produces more consistent, accurate results than a longer list — and never use a 1–5 numeric scale. It also reads better in the UI and gives the AI more information to reason with than a numeric scale does.
Be specific in your criterion description. Name what "good" and "bad" look like, and call out edge cases the AI should account for.
Favor structural questions if content is repetitive. In a test org where scenarios repeat, a criterion based on a structural property of the conversation (how many exchanges it took, how the conversation ended) tends to produce more varied, believable results than a quality or sentiment judgment, which can collapse toward one dominant label when the underlying content doesn't vary much.