Monitor and maintain Sidekick Answers performance
It’s helpful to think of training your generative AI assistant as you would train an Agent. Like ramping a new Agent, we suggest reviewing Sidekick Answers daily for 1-2 weeks after implementation. Like an Agent, once Sidekick Answers is fully launched and situated within your setup, monitor the AI assistant weekly to identify areas for improvement.
Train AI like you would a human Agent to to get better
AI acts as your digital Agent, and you should treat it like your human Agents. Teach AI new information (e.g., ensure you have the correct information documented as Public Answers). You can teach it how to behave or engage with Customers by configuring its voice and tone, words, and terminologies to avoid, and more.
Performance metrics
You’ll want to make sure you are monitoring your Sidekick Answers metrics regularly by using the Sidekick Dashboard.
Open the Sidekick Dashboard.
Take a look at resolutions and engagement by Thread, in particular, “Chat – Answer Thread” and “SMS – Answer Thread.”
Do you want Sidekick Answers to engage in more? If so, think about:
What other parts of your Chat experience could benefit from Sidekick Answers?
Are there top topics (see the Topics dashboard) for which you still need to write Public Answers?
Maintenance
The metrics above provide you with a helpful gut check as to how Sidekick Answers is performing, while evaluating individual Conversations within the Conversations page offers a more in-depth look at specific Sidekick Answers engagements. For routine maintenance, we suggest analyzing the Conversations page.
Log into Gladly and then click on Guides.
Go to Conversations.
Here, you will see a list of Conversations where Sidekick has engaged with your Customers.
Review a handful of Conversations where Sidekick has engaged. Use the Conversations Page Overview to evaluate responses.
Daily Sidekick Answers Management
Review key metrics and Sidekick Answers engagements on a daily basis following initial implementation. These steps help ensure your assistant is delivering expected results and aligning with your company’s standards in the first 1-2 weeks post-launch:
Review the Sidekick Dashboard by going to Reports > Sidekick Dashboard.
In addition to a general review of the dashboard, pay attention to the following two metrics:
Number of Assists in comparison to SMS and/or Chat volume
Number of Resolutions
With up-to-date Answers, customers can see ~10-15% of Assisted Conversations resolved following initial Sidekick Answers implementation
Customers may see as high as 25% of Assisted Conversations resolved with consistent augmentation of Public Answers. The best way to increased Number of Assists and Number of Resolutions is to optimize Public Answers.
Once you have a sense of how Sidekick Answers is performing, you can explore individual engagements more deeply on the Sidekick Conversations page.
Identify a handful of Conversations with a Sidekick resolved or a Sidekick handed off status.
Consider how the Quality Check has been evaluating generated replies.
Has the Knowledge Check been failing?
Create or update Public Answers to make responses more accurate and/or available.
Has the Hallucination Check been failing?
Review tips for avoiding Hallucinations.
Review Suggested Answers daily
In instances where Sidekick does not have enough information to fully resolve the Customer’s inquiry and hands off to an Agent, Sidekick will generate a batch of new Answers based on the Agent’s response to the Customer. Carving out time each day to review these recommendations will strengthen your existing Answer repository and improve Sidekick’s ability to respond accurately and completely to Customers.
Manage Sidekick Conversations Page
Before you start
Get to know Sidekick Answers Concepts.
If you’re brand new to Answer Threads, review New to Answer Threads? Start Here.
Search Sidekick Answers interactions
As a Gladly Team Manager or Administrator, you can review every Customer interaction Sidekick Answers engaged in through the Conversations page.
This is a great place to understand how Sidekick Answers interacts with Customers, how it responds, and why it hands off certain interactions to agents. This is information you need to understand how Sidekick Answers is performing and where you could make improvements.
On this page, you can search for interactions by entering a customer’s phone number, email address, or name. Icons indicate if Sidekick Answers has interacted with the customer through SMS or Chat.
Quality Check review
If any of the Quality Checks — Knowledge, Hallucination, or Unsupported Actions — fail, Sidekick will automatically handoff the Customer to an Agent.
Avoid Hallucinations
Hallucinations occur when generative AI attempts to generate a response but Sidekick believes that the response is not accurate because it’s unable to find a suitable Public Answer.
Hallucinations can be avoided by:
Ensuring you have Public Answers for as many answers to questions that Customers may ask.
Advice is fine-tuned to align with the expected brand tone and overall behaviors you’d like the generative AI assistant to adopt. Review tips for managing advice to ensure it’s configured for success.
The list of handoff scenarios includes instances where you want Sidekick to handoff to an Agent instead of attempting to generate a possible inaccurate reply.