15 questions to ask before you scale AI on your helpdesk

Before you scale an AI agent on your helpdesk, check five things: whether your team can step in at any moment, whether the AI understands the question or only finds the nearest article, how much work each new intent costs, how many bots you maintain per channel, language and brand, and what each resolved conversation really costs. The 15 questions below turn those checks into a list you can take to your current vendor or to any new one. They come from what support teams running AI agents at high volume run into once the first launch is behind them.

Last updated: 13 October 2026

Use the list in two ways. Ask your vendor each question and note the answer. Then test the answers on a sample of your own conversations: a vendor's demo shows the best case, your own tickets show the real one.

Control: can your team step in?

The first test of an AI agent is whether a person can take over any conversation at any time, without waiting for the customer to ask.

1. Can an agent take over an AI conversation at any moment?

Why it matters: when the AI is going wrong, the fastest fix is a person stepping in. If only some roles can take over, or the customer has to ask for a human first, mistakes last longer.

A good answer: any agent with access to the inbox can take over a live AI conversation, and the customer sees one continuous conversation.

2. Are AI-only conversations full tickets that your rules, tags and reports can act on?

Why it matters: if conversations the AI handles alone live in a separate, read-only space, your automations, tags and reports miss part of your support.

A good answer: an AI conversation is the same object as a human one, from the first message to the close.

3. Who decides that a conversation is resolved, and can you audit it?

Why it matters: if a vendor bills or reports on "resolutions", someone has to classify them. You should be able to open any conversation and see why it counted.

A good answer: the resolution status is visible per conversation, split into resolved, assumed resolved and escalated.

Understanding: does the AI answer the question that was asked?

A good AI agent reasons over your content to answer the specific question; a weaker one returns a summary of the nearest article.

4. Does it answer better than a general-purpose model on the same help center?

Why it matters: if a general AI assistant given your help center link answers your customers' questions better than your AI agent, the problem is not your knowledge base.

A good answer: on 20 real questions, your AI agent answers at least as accurately as the general-purpose model, and stays within your content.

5. Can it read the screenshots and files customers send?

Why it matters: many support requests start with a screenshot of an error. An AI that reads only text sends all of them to a person.

A good answer: the AI reads images and common documents, such as PDF files, and uses them in its answer.

6. When it isn't confident, what does it do?

Why it matters: an AI that guesses gives wrong answers; an AI that escalates too early saves nothing.

A good answer: you can set what happens at low confidence, for example asking the customer to rephrase before escalating.

Building: how much work is each new intent?

Count the places you have to edit to change one answer. That number decides how fast your team can improve the AI.

7. How do you describe what the AI should do?

Why it matters: if each intent is a short fixed description, overlapping requests end up in the wrong place and admins build large catch-all intents.

A good answer: procedures written in natural language, with branches ("if the customer says X, ask Y") and examples.

8. What happens when two intents overlap?

Why it matters: customers phrase the same request in many ways. If overlaps confuse the AI, every new intent makes the others worse.

A good answer: the vendor shows how the AI picks between similar requests, on your own examples.

9. How many automation rules does the AI depend on, and who maintains them?

Why it matters: when an AI agent needs triggers and rules around it to behave, those rules grow and start to collide.

A good answer: the AI handles routing and data collection itself, and the rules you keep fit on one page.

Scale: channels, languages and brands

Every extra bot you maintain is a cost that grows with each channel, language and brand you add.

10. How many AI agents or flows do you maintain per channel?

Why it matters: if chat, email and voice each need their own AI agent, every change is made two or three times.

A good answer: one AI agent, trained once, answers on every text channel.

11. Does it answer in the customer's language from one knowledge base?

Why it matters: if every reply needs a translated copy, each new language multiplies the work.

A good answer: the AI reads and answers in the customer's language from one knowledge base, with no setup per language.

12. Can one AI agent behave differently per brand without cloning it?

Why it matters: groups with several brands either clone the AI per brand or add brand rules to every intent. Both grow with each brand.

A good answer: brand-specific content and behaviour on top of one shared setup.

Measurement and cost

If you can't state the cost of one resolved conversation, you can't defend the budget.

13. Do AI and human conversations live in one report?

Why it matters: when AI results and human tickets are reported in different places, escalated conversations get counted twice or not at all.

A good answer: one report where the AI appears next to your human agents, with escalations counted once.

14. What does one AI-resolved conversation cost you, and one escalated ticket?

Why it matters: per-resolution, per-conversation and per-seat pricing all look cheap on a slide. Only your own volumes show the real cost.

A good answer: the vendor calculates both numbers on your last three months of volume, with every assumption written down.

Hand-off quality

Measure not only how much escalates, but how it escalates.

15. When the AI hands off, does the agent receive what they need to solve it?

Why it matters: an escalation that asks the same standard question every time leaves the agent to start from zero. A good hand-off saves minutes on every ticket.

A good answer: the AI collects the details each case needs, such as the order number, the account and what was already tried, and hands them over with the conversation.

How Customerly Aura answers these 15 questions

Customerly Aura is the AI agent in Customerly's inbox. Here is how it answers each question; test any of them on your own conversations in a demo.

If you run Zendesk AI agents today, our guide to Zendesk AI agents explains how to check these questions in Zendesk, and Zendesk AI alternatives compares the options.

FAQ

Who should use this checklist?

Support leaders and the people who build and maintain the AI agent, before they add channels, languages or brands, or before a renewal.

Is the checklist specific to one vendor?

No. The 15 questions apply to any AI agent or helpdesk. Customerly's own answers are in a separate table at the end.

What AI resolution rate is realistic?

It depends on your content and your mix of requests, so measure it on your own conversations. Ask for resolved, escalated and wrong answers separately. Across all Customerly customers, Aura resolves 72% of conversations on average.

How do we test an AI agent on our own tickets?

Load your public help center into the AI agent, ask it 20 to 50 real questions from last month, including screenshots, follow-ups and a second language, and score the answers against what your team replied.

Do we need to change helpdesk to fix these issues?

Not always. Some issues are configuration and content. Others, such as one AI agent per channel or text-only input, come from the product's architecture and need a different tool.

Test your AI agent with us

We can run Aura next to your current AI agent on your own help center and answer the 15 questions with your data. Book a demo, or read how Aura, the AI customer support agent works.