Monitor and maintain Answer Threads performance
It’s helpful to think of training your generative AI assistant as you would train an Agent. Like ramping a new Agent, we suggest reviewing Answer Threads daily for 1-2 weeks after implementation. Like an Agent, once Answer Threads is fully launched and situated within your setup, monitor the AI assistant weekly to identify areas for improvement.
Train AI like you would a human Agent to to get better
AI acts as your digital Agent, and you should treat it like your human Agents. Teach AI new information (e.g., ensure you have the correct information documented as Public Answers). You can teach it how to behave or engage with Customers by configuring its voice and tone, words, and terminologies to avoid, and more.
Performance metrics
You’ll want to make sure you are monitoring your Answer Threads metrics regularly by using the Sidekick Dashboard.
Open the Sidekick Dashboard.
Take a look at resolutions and engagement by Thread, in particular, “Chat – Answer Thread” and “SMS – Answer Thread.”
Do you want Answer Threads to engage in more? If so, think about:
What other parts of your Chat experience could benefit from Answer Threads?
Are you transferring too many interactions?
Are there top topics (see the Topics dashboard) for which you still need to write Public Answers?
Maintenance
The metrics above provide you with a helpful gut check as to how Answer Threads is performing, while evaluating individual Conversations within the Conversations page offers a more in-depth look at specific Answer Threads engagements. For routine maintenance, we suggest analyzing the Conversations page.
Log into Gladly and then click on Sidekick Beta.
Go to Conversations.
Here, you will see a list of Conversations that Answer Threads have had with your Customers.
Review a handful of Conversations where Answer Threads has engaged. Use Manage Answer Threads Contacts Page to evaluate responses.
Daily Answer Threads Management
Review key metrics and Answer Threads engagements on a daily basis following initial implementation. These steps help ensure your assistant is delivering expected results and aligning with your company’s standards in the first 1-2 weeks post-launch:
Review the Sidekick Dashboard by going to Reports > Sidekick Dashboard.
In addition to a general review of the dashboard, pay attention to the following two metrics:
Number of Assists in comparison to SMS and/or Chat volume
Number of Resolutions
With up-to-date Answers, customers can see ~10-15% of Assisted Conversations resolved following initial Answer Threads implementation
Customers may see as high as 25% of Assisted Conversations resolved with consistent augmentation of Public Answers. The best way to increased Number of Assists and Number of Resolutions is to optimize Public Answers.
Once you have a sense of how Answer Threads is performing, you can explore individual engagements more deeply on the Conversations page.
Identify a handful of Conversations with a Sidekick resolved or a Sidekick handed off status.
Consider how the Quality Check has been evaluating generated replies.
Has the Knowledge Check been failing?
Create or update Public Answers to make responses more accurate and/or available.
Consider adding Draft Answers.
Has the Hallucination Check been failing?
Review tips for avoiding Hallucinations.
Manage Answer Threads Conversations Page
Before you start
Get to know Answer Threads Concepts
If you’re brand new to Answer Threads, review New to Answer Threads? Start Here.
Search Answer Threads interactions
As a Gladly Team Manager or Administrator, you can review every Customer interaction Answer Threads engaged in through the Conversations page.
This is a great place to understand how Answer Threads interacts with Customers, how it responds, and why it hands off certain interactions to agents. This is information you need to understand how Answer Threads is performing and where you could make improvements.
On this page, you can search for interactions by entering a customer’s phone number, email address, or name. Icons indicate if Answer Threads interacted with the customer through SMS or Chat.
Quality Check review
If any of the quality checks — Knowledge, Hallucination, or Unsupported Actions — fail, Answer Threads will automatically handoff the Customer to an Agent.
Avoid Hallucinations
Hallucinations occur when generative AI attempts to generate a response but Answer Threads believes that the response is not accurate because it’s unable to find a suitable Public Answer.
Hallucinations can be avoided by:
Ensuring you have Public Answers for as many answers to questions that Customers may ask.
Advice is fine-tuned to align with the expected brand tone and overall behaviors you’d like the generative AI assistant to adopt. Review tips for managing Advice to ensure it’s configured for success.
The Transfer Reasons list includes many scenarios where you want Answer Threads to handoff to an Agent instead of attempting to generate a possible inaccurate reply.