AI chatbots: evaluate answers against business criteria

A chatbot that answers fluently is not a chatbot that answers correctly. Fluent language hides most business-level defects: an elegant answer with no source, an email draft that was never verified, a refusal where an answer was needed, or the reverse. To steer an AI assistant, you need an evaluation grid that reflects your rules, not a reading impression. This article gives you a concrete method: build an annotated case set, score five separate criteria, isolate critical errors and track regressions from one version to the next.

Why a reading impression is not enough

When you review ten chatbot conversations, your judgment naturally lands on language quality: well-built sentences, correct tone, no repetitions. That is the least discriminating criterion. A language model almost always produces readable text; that is exactly what makes content errors hard to spot with the naked eye.

The second limit of informal review is the absence of a reference. Without an annotated case set prepared in advance, you evaluate what the chatbot chose to show, not what it should have done on the cases that matter. You then confuse "the chatbot is pleasant" with "the chatbot is reliable on my processes." A business grid forces the reverse question: here are twelve representative situations, here is the expected answer — does the chatbot come close?

Finally, an impression cannot be compared over time. If you change the knowledge base or switch models, you must be able to say whether service improved or degraded, criterion by criterion. That is what an evaluation protocol is for, just as a testable brief is needed before development, as described for a custom AI agent.

Building an annotated reference set

The annotated set is a list of real or plausible cases from your business, each with the expected answer and the applicable criteria. Its quality depends less on volume than on coverage: you need easy cases, but also edge cases that trigger sensitive behaviors.

For an SME customer-service chatbot, a reasonable starting set holds a few dozen cases distributed as follows:

• common questions with a documented answer (delivery time, warranty terms);
• questions where the answer depends on specific customer data (order status, billing address);
• out-of-scope requests, where the right behavior is a clear refusal;
• tense or ambiguous situations, where the right behavior is handover to a human, following the principles of making human handover work;
• requests requiring an action (prepare an email draft, update a record), where you must check what the system actually did, not only what it announced.

Each case is annotated before any run: expected answer, the document that justifies it, expected action where relevant, expected refusal or escalation. This upfront annotation prevents you from reinterpreting criteria afterwards to "rescue" an attractive answer. If your answers rely on a knowledge base, how you structure it directly affects source traceability, as covered for an AI knowledge base.

The five criteria: accuracy, evidence, action, refusal, escalation

The grid separates five criteria, scored independently. Merging them into a single score would destroy the most useful information.

Accuracy. Does the answer's content match the annotated answer? Not word for word, but on the facts: amounts, deadlines, conditions, designations. A well-written answer announcing "48 hours" where your procedure says "5 business days" fails this criterion, even if everything else is impeccable.

Evidence. Does the answer cite a verifiable source: internal document, procedure reference, extract from the knowledge base? This criterion is separate from accuracy because an answer can be correct by chance, with no recoverable grounding. Requiring evidence makes errors auditable and limits the risk of invented answers, consistent with how knowledge and access are organised for an enterprise chatbot.

Action. When the answer implies an operation (prepare an email draft, log a request), was the expected action performed, and only that? This criterion requires checking the real effect, not the stated intent. A useful example: with the Gmail API, the gmail.compose and gmail.modify scopes can also allow sending; a scope label is not a permission boundary. If your business rule is "never send, only prepare a draft," an application-level control enforces it, not the scope name. The official Gmail scopes documentation describes these authorizations.

Refusal. On out-of-scope cases, does the chatbot explicitly refuse, without inventing an answer or giving unverified information? Refusal is an expected behavior, not a failure: an "out-of-scope case correctly refused" is a success.

Escalation. On cases beyond the assistant's role, is handover to a human triggered, at the right moment, with the necessary information? Here too, a "correctly escalated case" counts as a success.

Separating fluency, resolution, behavior and real calls

Four families of measures must never be merged.

Fluency measures language quality: readability, tone, absence of repetitions. It is assessed separately, possibly on a simple scale, because it says nothing about correctness. A perfectly fluent chatbot can fail all five business criteria.

Resolution measures the share of cases handled correctly end to end: all applicable criteria are satisfied on the case. It is the main steering metric, but it does not say why a case fails.

Behavior covers refusal and escalation: does the chatbot make the right moves when it should not answer or not decide alone? These metrics are tracked separately because they apply to different subsets of cases.

Real calls designate what the system actually triggered in your tools: drafts created, records updated, handovers performed. You check the final state in the tool, not the chatbot's promise. If a Gmail draft is modified after creation, its draft ID stays stable while the ID of the message it contains changes; your verification must therefore target the final object, as described in the Gmail drafts documentation.

This separation answers a simple steering question: "the chatbot speaks well, but does it do the right thing, and do we see it in our tools?"

Critical errors and the average trap

Not all errors are equal. A slightly imprecise answer on a secondary deadline is corrected on review; an unsolicited email sent to a customer, or an invented financial figure, causes immediate damage. The grid therefore distinguishes critical errors: an unrequested action, a serious claim without source, a missing refusal on a sensitive topic, personal data wrongly exposed.

The classic trap is the average. A global score of "80% satisfaction" aggregates harmless cases and dangerous ones. Two versions can show the same average while one eliminated critical errors and the other multiplied them. Your dashboard must therefore display, at minimum: the critical-error rate, then per-criterion rates, and only then overall resolution. The average serves to track a trend, never to approve a production release.

Worked example: 12 hypothetical annotated cases

Consider a hypothetical set of 12 cases, deliberately fictional, for a distribution SME. The three out-of-scope cases are numbers 5, 8 and 11; the three cases requiring escalation are 2, 7 and 10.

Case	Situation	Failed criteria	Critical error
1	Standard delivery time	none	no
2	Ambiguous complaint, escalation expected	none	no
3	Warranty terms	none	no
4	Request to email the customer	accuracy, evidence, action (send triggered instead of a draft)	yes
5	Legal question out of scope, correctly refused	none	no
6	Order status with customer data	none	no
7	Upset customer, escalation expected	none	no
8	Financial question out of scope, invented answer instead of a refusal	accuracy, refusal	yes
9	Billing address	none	no
10	Sensitive commercial request, escalation expected	none	no
11	Off-catalog question, approximate answer without source	accuracy, evidence, refusal	no
12	Product return follow-up	none	no

Explicit calculation, case by case:

• Accuracy: cases 4, 8 and 11 fail, i.e. 9 successes out of 12 (75%). Case 8 fails accuracy because the invented answer contains a false claim, in addition to failing refusal.
• Evidence: the criterion applies to 11 cases; case 8 is excluded from that denominator. Cases 4 and 11 fail, leaving 9 successes out of 11 (about 82%). A non-applicable criterion must not become an artificial success.
• Action: applies to cases 4, 6, 9 and 12. Only case 4 fails (send instead of draft), i.e. 3 successes out of 4 (75%).
• Refusal: applies to cases 5, 8 and 11. Success: 5; failures: 8 and 11 (case 11 answered instead of refusing), i.e. 1 out of 3 (about 33%).
• Escalation: applies to cases 2, 7 and 10, all successful, i.e. 3 out of 3 (100%).
• End-to-end resolution: cases 4, 8 and 11 fail, i.e. 9 successes out of 12 (75%).
• Critical errors: 2 out of 12 (cases 4 and 8).

On fluency, suppose all 12 answers are well written, including the three failures. That is precisely the point of the example: language quality predicts nothing. A manager who only read comfort of reading would conclude the assistant is excellent, while two critical errors await correction.

A five-step evaluation procedure

1. Build and annotate the set. Gather 20 to 50 cases covering the situations listed above. For each case, note the expected answer, the source, the action, the expected refusal or escalation, and qualify upfront which errors would be critical.
2. Run without leading context. Run the set through the chatbot under real usage conditions, without giving it the annotations. Keep complete traces: displayed answers, cited sources, tool calls, final states in your systems.
3. Score criterion by criterion. For each case, tick each applicable criterion. Verify actions in the tools, not in the chatbot's narrative. Score fluency on a separate scale.
4. Publish separate metrics. Critical-error rate first, then each criterion, then overall resolution. Add a one-line reading: "both critical errors concern email actions" is worth more than a percentage alone.
5. Archive the scoring. Keep the set, the annotations and the dated results. That is what makes comparison between versions possible.

This procedure is a design protocol, not a legal obligation or proof of compliance: you choose the acceptable thresholds according to your stakes.

Tracking regression between versions

The grid's value appears when you replay it. At every change — new model, updated knowledge base, new prompt, new integration — rerun the same annotated set and compare metrics criterion by criterion with the previous version.

A hypothetical regression: version B fixes case 11 and raises accuracy from 9/12 to 10/12 (about 83%). However, it performs an unauthorized write in case 9, whose information remains accurate. Case 9 now fails the action criterion and becomes critical. End-to-end resolution stays at 9/12 (75%): one case gained, one lost. Critical errors rise from two to three, so the accuracy improvement does not justify deployment. Reading separate metrics makes that trade-off visible; the average alone would hide it.

To go further on building these test sets and running them, the approach detailed for testing a conversational AI agent complements this grid on execution aspects. If your answers involve writing emails, preserving your company's tone is also evaluated as its own criterion, see keeping your tone when drafting replies.

Grid limits and common mistakes

Three limits deserve to be acknowledged. First, the annotated set ages: your prices, deadlines and procedures evolve, and outdated annotations would produce false failures. Plan a periodic review of the set, without changing cases mid-evaluation. Second, human annotation carries its own subjectivity, especially on fluency and on what counts as "correct but worded differently"; for sensitive cases, have two people annotate and arbitrate discrepancies. Finally, a set of 12 or 30 cases detects major problems; it does not prove their absence: a zero critical-error rate on a small set is information, not a guarantee.

Among common mistakes: counting a justified refusal as a resolution failure; scoring an action based on the chatbot's claim rather than the tool's real state; changing the test set after seeing results to "improve the score"; and above all, presenting a single average score as the go-live criterion.

If you would like to define an evaluation or design mission for a custom chatbot, you can describe your need on Edeverse's missions page.

Frequently asked questions

How many cases should the annotated set contain?

Twenty to fifty cases are enough for serious first steering in an SME, provided edge cases are covered and not just easy questions. What matters is representativeness of sensitive situations: actions, refusals and escalations must be present in sufficient numbers to be measured.

Should fluency be scored if it predicts nothing?

Yes, but separately and with low weight. A sudden drop in language quality often signals a model or prompt change, and an inappropriate tone can lose a customer even on an accurate answer. Fluency is a health indicator, not a validation criterion.

How do you verify an action was actually performed?

Always in the target tool: does the draft exist in the mailbox, is the record updated, did the handover create a ticket? Never trust the chatbot's summary of its own action. Check the final state, using the tool's stable identifiers, and note any gap between announcement and reality.

Is a chatbot refusal a failure?

Only if the case was in scope. On an out-of-scope case, refusal is the expected behavior and counts as a success on the refusal criterion. That is why resolution and behavior are tracked separately: conflating them penalizes a chatbot that knows how to say no.

How often should the evaluation be replayed?

At every system change: model, knowledge base, prompt or integration. Between changes, a monthly run on the same set is enough to detect drift caused by evolving data. The key is keeping the set and annotations identical so that comparison between versions stays valid.