Set Up A/B Testing

A/B testing allows you to split incoming Conversation traffic across two different Gladly Agent configurations. You can compare two different Gladly Agents or test two versions of the same agent to evaluate which setup performs better before committing to a change.

A/B testing is configured per Entry Point, giving you control over which specific Channels and Entry Points are part of a test. You can run tests at any traffic split you choose, and end the test at any time.

Before you begin

A/B testing is set up from the Entry Points page and requires at least one entry point already connected to a Gladly Agent. If you haven't done this yet, see Connect or Disconnect Gladly Agents from an Entry Point.

Set up an A/B test

1. From the Guides page, click the Gladly Agent name dropdown at the top of the page, then select Connect Entry Points.
   

2. Locate the entry point you want to test.

3. Click + Add A/B test below the entry point row.
   

4. Two rows appear — one for each leg of the test. For each row, configure the following:

   • %: the percentage of traffic routed to that leg. The two values must add up to 100. Adjusting one automatically updates the other.

   • Agent: select any live Gladly Agent. Both legs can use the same agent or different Gladly Agents.

   • Version: select a version of the chosen Gladly Agent. Choose Always latest to always use the most recently saved version, or select a specific saved version by timestamp.
     

5. Click Save.

The entry point row updates to show A/B test in the Agent column and the configured traffic split in the Traffic column.

Edit an active A/B test

To adjust the Gladly Agents, versions, or traffic split for an active test:

1. Select Entry Points from the left sidebar under DEPLOY.

2. Locate the entry point with an active A/B test.

3. Update the %, Agent, or Version values directly on the variant rows.

4. Click Save.

End an A/B test

To stop an A/B test and return the entry point to a single Gladly Agent:

1. Select Entry Points from the left sidebar under DEPLOY.

2. Locate the entry point with an active A/B test.

3. Click the delete icon on the variant row you want to remove.

4. Click Save.

The remaining variant becomes the sole configuration for that entry point.

Review A/B test results

Once an A/B test is running, results are available in two places:

• Gladly Improvement Opportunities dashboard

• Gladly AI Sessions report

Gladly Improvement Opportunities dashboard

The Gladly Improvement Opportunities dashboard includes two filters specifically for A/B testing: Entry Point and Experiment Traffic Split.

Use the Entry Point filter to narrow the dashboard to a specific entry point where a test is running. Use the Experiment Traffic Split filter to select a specific test.

Once filtered, the existing dashboard tiles update to reflect only the sessions from that experiment. Each Gladly Agent version in the test appears as a separate row across tiles, making it possible to compare resolution rates, handoff rates, response rates, and recontact rates side by side.

The Gladly AI Agent Outcomes tile and the Handoff Opportunities Prioritization table are particularly useful for assessing which configuration is performing better.

Historical data is not available prior to creating a test

Entry Point and Experiment Traffic Split data is only available for sessions created after the A/B testing feature was enabled. Historical session data will not include these fields.

Gladly AI Sessions report

The Gladly AI Sessions report, accessible via Reporting in Gladly Team, also includes Entry Point and Traffic Split as filterable fields. This report provides session-level data and can be exported as a CSV for more granular analysis outside of the dashboard.