Monitoring AI Performance Over Time

Let's be honest, AI is great, but it's not perfect, especially in the beginning. The real magic happens when you know how to monitor its performance and continuously improve it. Today, we are diving into how to measure and track your AI assistant performance over time. We will look at real world techniques including a case study from Tin System to show how to focus on what matter the most like low feedback conversations so you can keep your AI sharp, efficient, and always improving. Let's get started. Okay. Okay, as Aura, your AI assistant handles more conversation. Reviewing every single one just isn't realistic, especially if you're managing thousands of interactions daily.

But here's the thing, you don't need to monitor everything. Instead, you can focus on low feedback conversations or trend that shows something might be off. In this lesson, we'll walk you through how to use Seesat customer satisfaction insight to track AI performance. Then how to filter and review negative feedback to find improvement opportunities and what to do if you have fewer conversation like 50 per day and needs a more hands-on approach. Let's reconnect with the team assistant case study we covered earlier. They handle thousands of conversations daily with their AI assistant Aura. But instead of reviewing every conversations aura touches, they focus only on those flagged with low sea salt scores like bad or terrible responses.

This allow them to own in the areas where aura can improve without getting overwhelmed by the sheer volume of conversations. Of course, now I'll show you exactly how to do this for your own AI. All right, head over to your inside dashboard and navigate to sees customer satisfaction over here. And here you will find the overall satisfaction score for all conversation broken down by team and AI performance. Of course you can see all the teammates and team and also you can check only aura conversations. Once you filtered for aura or your AI take note of the following incoming sea score check the average rating over time are scores improving staying steady or dropping.

Drill down by feedback category. So if you scroll down over here, you will be able to see all the feedback and you can filter down by terrible, poor or like okay. Now here you can go and review what happened uh with this ticket. When you click on the conversation side, you can see maybe what was the issue and if aura gathered the information from the article. If she didn't, you can provide maybe a better uh solution for this. If you want to, you can reopen, of course, the ticket if you know if something were wrong with with the customer. You can reopen the ticket directly with this person.

Send a new message of course saying you know this happened and I wanted to follow up with the issue you provided and this is how to resolve it. But you can also tweak over here as a can response. For example, if you want to create a cannon response here, you can tweak the response, save it as a new response. And of course, this will be used as a new AI training. This means that the next time whenever the same question is arising or I will have the information at hand and will be able to better handle this conversation. In this case the user reported that they couldn't find any link to create invoice.

So probably would be better to provide more context in this uh document in this article and you can provide for example the link based on the customer feedback. Now if you notice a trend or frequent negative feedback it's a sign that something in the AI responses needs to be fixed. Here's where you take action. Filter the seesat results to show only terrible or bad feedback. And for each flagged conversation, open the conversation clicking here. Look at what the AI said and how the customer responded. Identify patterns in the feedback and are certain intents or mission consistently receiving negative scores and not opportunities for improvement. Maybe the misunderstood the intent, provided incomplete answers, or escalated unnecessarily because probably was missing or having some missing information.

By focusing on these low feedback cases, you can refine specific areas without needing to review every single interaction. Now, if your business isn't managing thousands of conversation per day, say it's closer to 50, your approach will a little different. Okay, here's what to do. Create a dedicated escalation team like Czech AI. Whenever conversations get flagged for review or like it's escalated, assign it to this team. Manually review flag conversation. Instead of relying on seesat data alone, review all escalated conversation individually. Now track record recurring issues. Use these reviews to identify common problems with your AI assistant and make targeted improvements with fewer conversations. This hands-on approach ensures you're still catch every potential issue.

Finally, let's talk about benchmarking. Your AI doesn't operate in a vacuum. It works alongside your human teams. To see how your AI stacks up, compare its CES score against your human agents. Here's how to do it. Right? You will go back to insights under customer satisfaction. So if you select all teammates and teams, this will of course include also aura. But if you want to compare against your human team and your AI, then you will be able to do it by for example selecting customer support and then selecting aura and see how the percentages changes. Okay. So this is how you do it. This is how you can check against the performance.

And another thing that you can do is checking individually how your teammates are performing in terms of first reply average time, the conversation rating for each account of yours, for each teammates of yours. You will be able to see how many replies they've been sending out, how many conversation they closed. Okay, so there you have it. A clear, actionable way to monitor your AI's performance over time. Whether you're managing thousands of conversations like Team System or just a few dozen, these tools and techniques will help you stay on top of your AI's performance and keep improving it. Okay, in the next lesson, we'll talk about how to use this feedback to retrain and refine your AI system for even better results.

If this lesson was helpful, hit the like button, subscribe, and let's keep building smarter AI system together. I'll see you in the next