Build a Signal

Navigate to Signals

From the left navigation, go to Gladly AI > Analyze > Signals. The Signals dashboard lists every Signal your team has created, split into Active and Archived tabs. Search by name, or filter by agent or Channel to narrow the view.

Gladly AI interface showing Signals section for analyzing agent sessions and conversation complexity.

Criteria Library — a library of the reusable criteria your org has created. Each entry shows its current version and when it was created.
New Signal — click here to start building a new evaluation job.

Create a new Signal

Click New Signal to open the builder. It's a four-step wizard: Define, Scope, Schedule & Run, and Review. At each step, you can go back to any earlier step before creating the Signal.

Dashboard displaying AI agent session quality metrics and options for new signal creation.

Step 1: Define your signal

Form fields for defining signal related to sizing and fit questions.

[A] Signal name — name the outcome you're measuring, e.g. "Return handling — damaged goods."
[B] Description (optional) — a short note on what this Signal is for.
[C] Evaluation criteria — click Add criteria to choose up to 5 existing criteria from the library, or Create new criterion to build one from scratch. Each criterion becomes a judgment applied to every evaluated session.

When browsing existing criteria to add, click the info icon next to any entry to open a Criterion details panel — a quick, read-only view of its description, labels, and full evaluation prompt, with an Edit criterion link if you need to make changes. This is the fastest way to confirm a criterion measures what you think it does before adding it to a Signal.

Step 2: Scope — narrow the sessions

Overview of return handling process for damaged goods with session narrowing options.

Leave everything unselected to evaluate all sessions, or narrow the sample by:

[A] AI agents — pick specific agents, or leave "No filter (all agents)."
[B] Guides — narrow further to specific guides under an agent.
[C] Channels — Chat, Email, SMS, or no filter (all channels).
[D] Keywords (optional) — only evaluate sessions containing one of the keywords you add.

Narrowing by agent, guide, channel, or keyword focuses the Signal and controls how many sessions it scores. Leaving everything unselected evaluates all sessions.

Step 3: Schedule & run

Set how often this Signal runs. Creating it does not run it automatically — you separately choose whether to kick off a one-time test.

Options for scheduling tests and selecting date ranges for signal analysis.

[A] Run on a schedule — turn this on to have the Signal run automatically (daily or weekly). Leave it off for a manual, one-off Signal.
[B] Run a first test now — a one-time test that scores the Signal against a date range you pick (Today, Yesterday, Last 7 days, Last 30 days, or Custom), so you can preview results right away. This is separate from the recurring schedule above.

Step 4: Review

Confirm the Name, Description, Criteria, Scope, Schedule, and First test run settings, then click Create Signal.

Review section for sizing and fit questions, confirming details before creating a signal.

Build a criterion

If you clicked Create new criterion in Step 1, a three-step wizard opens: Describe, Labels, Review. You can go back at any point to refine your answers.

1. Describe what you want to measure

First, you’ll be prompted to answer "What do you want to measure?". Describe the quality aspect you want to evaluate in sessions, and be specific about what good vs. bad looks like. The AI uses your description to draft classification logic; no prompt engineering required.

Instructions for defining product sizing criteria and evaluating customer inquiries about fit.

Tips for good descriptions:
Be specific about what you're measuring (e.g., "politeness" rather than "overall quality").
Mention what success looks like.
Include any edge cases or exceptions the AI should acc

Click Generate. The AI drafts classification labels and an evaluation prompt for you to review.

2. Review the classification labels

The AI proposes mutually exclusive labels the criterion will use to classify each session, each with a name and a description of what qualifies for it. Review, edit, regenerate, or add labels as needed — up to 5 labels, each up to 30 characters.

Edit any label's name or description inline, click Regenerate for a fresh set, or add a custom label with + Add Label. Hover a label to reveal a trash icon and remove it.
Goal label — optionally mark one label as your Goal, the result you're aiming for. Gladly uses this to calculate the occurrence rate shown throughout reporting.
Color — pick a color for each label from the six swatches provided, for quick visual identification in results. The AI won't pick colors or a goal for you — set both yourself.
Label tips shown in the UI: use 2–5 labels that cover all possible outcomes, keep them mutually exclusive, and order them from best to worst (or positive to negative).

Reviewing classification labels for sizing questions and customer inquiries in a user interface.

3. Review & Create

The final step shows the full criterion before you save it: name, description, classification labels, and the AI-generated evaluation prompt (the exact instructions the AI follows against each session transcript). Review everything (you can edit any field, including the evaluation prompt itself) then click Create. The criterion is saved to your Criteria Library and becomes available to add to any Signal.

Manage your Criteria Library

Open Criteria Library from the Signals dashboard to see every criterion your org has created.

Gladly Signals interface showing session evaluation options and highlighted Criteria Library button.

Each row shows its version and creation date; click Use to add one to a new Signal, or the pencil icon to edit it.

Criteria list for Gladly with highlighted option to edit specific criteria.

Editing a criterion never rewrites history
Each criterion is versioned, and past runs keep the results from the version that was active when they ran. Only future runs pick up your edits, nothing re-runs automatically.

Best practices for building Criterion

Prefer fewer labels. Binary (Yes/No) or a short list of named categories (Positive/Neutral/Negative) produces more consistent, accurate results than a longer list — and never use a 1–5 numeric scale. It also reads better in the UI and gives the AI more information to reason with than a numeric scale does.
Be specific in your criterion description. Name what "good" and "bad" look like, and call out edge cases the AI should account for.
Favor structural questions if content is repetitive. In a test org where scenarios repeat, a criterion based on a structural property of the conversation (how many exchanges it took, how the conversation ended) tends to produce more varied, believable results than a quality or sentiment judgment, which can collapse toward one dominant label when the underlying content doesn't vary much.