Everyone has met the bad version. You type a clear question, the widget replies with three help-center links you already read, and the only way to reach a person is to type "agent" four times in a row. The dashboard behind that widget probably shows a deflection rate the vendor is proud of. The customer just churned a little.
AI customer support automation can be much better than that, but the gap between the two versions is mostly design and measurement, not model choice. Below is how we approach these projects, roughly in the order the decisions come up.
Start with triage, not a chatbot
The cheapest win in most support operations has nothing to do with answering customers. It is classifying the ticket correctly the moment it arrives. Intent, product area, language, urgency, customer tier and sentiment decide which queue a ticket lands in and how fast someone touches it. Many teams still do this by hand, or with keyword rules that someone wrote years ago and nobody dares to edit.
An LLM classifier with a fixed label set does this well. Give it your actual taxonomy, a short definition and two or three real examples per label, and require structured output so a bad response fails validation instead of silently routing to the wrong team. Pull a few hundred historical tickets that agents already labeled, measure agreement per label, and look hard at the confusion between neighbors like "billing dispute" and "refund request". Those pairs are usually where your own taxonomy is ambiguous, and fixing the definitions helps humans too.
Triage is low risk because a human still reads every ticket. It also gives you the labeled data and the trust you need before letting anything reply to customers directly. If a team cannot agree on what a ticket is about, they are not ready to automate the answer.
Answers must come from your content
When you do start replying, every answer should be built from retrieved material: help-center articles, macros, internal runbooks and, where permitted, resolved tickets with a similar problem. The model's job is to pick the relevant passage and phrase it for this customer, not to recall how refunds work from its training data. Your refund window is 30 days; the model's prior might say 14.
Two details matter more than people expect. First, the content itself. Help centers are full of articles written for a product version from two years ago, and automation exposes every one of them. Budget time for a content audit before launch and give the support team a fast way to flag a bad source. Second, account context. "Where is my order?" cannot be answered from an article. It needs a read-only lookup against your order system, scoped to the authenticated customer. Read actions like order status, plan details or invoice copies are where most real resolution comes from.
Write actions such as issuing refunds, changing addresses or canceling subscriptions are a separate decision. Start with the bot preparing the action and an agent approving it in one click. Move specific, low-value actions to full automation only after you have weeks of logs showing the approvals were rubber stamps.
Escalation rules decide whether customers trust it
The quality of a support bot is judged at the moment it gives up. A good handover is fast, obvious, and passes a summary so the customer never repeats themselves. We write escalation rules as explicit code paths, not as a sentence in the prompt asking the model to "escalate when appropriate".
- The customer asks for a person, in any wording. Hand over immediately, no retention attempt.
- Retrieval finds nothing above the relevance threshold. Say so and route, rather than improvising.
- Two failed attempts on the same issue in one conversation.
- Topics you have ruled out: legal threats, security incidents, chargebacks, account takeover, anything involving vulnerable customers.
- Strong negative sentiment combined with a high-value account or an open complaint.
- Any action above a set amount or outside policy.
The handover note matters as much as the trigger. The agent should see the customer's issue in two lines, what the bot already tried, which articles it used, and any account data it pulled. A handover that dumps a raw transcript wastes the agent's first two minutes and makes the automation look like a cost rather than a help.
Deflection is not resolution
Deflection counts conversations that ended without a human. That includes customers who got their answer, customers who gave up, and customers who left to open a ticket by email instead. Reporting it as success rewards a bot that is hard to escape.
Measure resolution instead, and define it strictly. A conversation counts as resolved when the customer confirms it, or when they do not contact you again about the same issue within a window such as 72 hours or seven days, across any channel. That last part requires matching contacts by customer ID across chat, email and phone, which is real work, but without it your numbers are guesses.
| Metric | What it tells you | Watch out for |
|---|---|---|
| Deflection rate | Share of chats with no human involved | Counts abandonment as success |
| Verified resolution rate | Issues solved with no repeat contact | Needs cross-channel customer matching |
| Escalation rate by intent | Where automation struggles | A low rate may mean escalation is hidden |
| CSAT, bot vs human | Whether customers accept the experience | Low survey response on bad chats |
| Handle time after handover | Quality of the handover summary | Compare against the same intents only |
| Cost per resolved contact | The number finance actually wants | Include content and review time |
Read transcripts every week. Pick 30 to 50 at random, plus every conversation with a negative rating, and have a senior agent grade them. That review catches tone problems and wrong answers that no dashboard will surface, and every failure becomes a test case for the next release.
Fitting into Zendesk or Intercom
Both Zendesk and Intercom now ship their own AI agents, and for a team with a clean help center and simple intents those built-in options are often the right first move. They are quick to switch on and the vendor carries the maintenance. Custom work starts to pay off when answers depend on your own systems, when you need control over which model runs and where data goes, or when the vendor's pricing per resolution stops making sense at your volume.
A custom build should still live inside the helpdesk rather than beside it. In practice that means listening to ticket and conversation events through webhooks, writing back tags, internal notes, suggested replies and assignments through the platform's API, and keeping the helpdesk as the system of record. Agents should never need a second tool to see what the bot did. Plan for the boring parts too: API rate limits, idempotent handlers so a retried webhook does not post the same reply twice, and a kill switch that a support lead can flip without calling engineering.
A useful middle step is agent assist. The model drafts the reply inside the agent's composer and the agent edits and sends. You learn which intents produce drafts that get sent unchanged, and those are your candidates for full automation.
Tone is a product decision
Models default to a cheerful, wordy register that grates in support. Nobody wants "Great question!" when their payment failed. Write a short style guide with your support lead: how long replies should be, when to apologize and when not to, how to say no to a refund, whether to use the customer's first name. Then put examples of good and bad replies into the prompt and into your evaluation set.
Regional differences are real. Many UK and Australian customers read heavy American enthusiasm as insincere, and spelling like "color" versus "colour" is noticed. If you serve several markets, set the variant per region and test it. Be plain about what the customer is talking to as well. Label the assistant as automated, and do not give it a human name and photo. Customers forgive a bot that is clearly a bot far more readily than one pretending otherwise.
Done this way, the automation handles the repetitive half of the queue properly and gives agents more time for the conversations that need judgment. That is the outcome worth paying for, and it is measurable if you set up the metrics before launch rather than after.
Questions we get asked
- What share of support tickets can AI resolve?
- It depends heavily on your ticket mix and the systems the bot can read. Order status, account and how-to questions automate well; disputes and complex faults rarely do. Measure verified resolution per intent rather than trusting a headline figure.
- Should we use Zendesk's or Intercom's built-in AI, or build our own?
- Try the built-in agent first if your answers live in the help center. Build custom when answers need data from your own systems, when you need control over models and data location, or when per-resolution pricing gets expensive at volume.
- How do we stop the bot from giving wrong answers?
- Ground every answer in retrieved content, set a relevance threshold below which it escalates, audit your help center before launch, and grade a sample of real transcripts every week.
- Should customers know they are talking to AI?
- Yes. Label it clearly and keep a visible route to a person. Transparency is expected by customers and increasingly by regulators.