Conversational AI testing: a practical method for finding failures before launch

A conversational AI agent can succeed in a polished demonstration and fail on a short, ordinary question. Real users leave out details, correct themselves, refer to earlier messages and expect the system to know whether an action actually happened. Testing needs to cover that whole interaction, not just whether the answer sounds natural.

This guide uses a fictional community library preparing an information assistant. Visitors ask about opening hours, collecting reservations and attending activities. The example is an original testing scenario, not a reported implementation or a promise about Edeverse performance.

The aim is to produce a useful release decision: which requests the agent can handle, which should go elsewhere and what evidence supports that boundary. You can begin with a small manual test notebook and add automated checks when the process becomes repeatable.

Define a successful conversation in operational terms

For the library, a good answer should identify the visitor's purpose, use the applicable information and suggest a feasible next step. If the visitor wants to collect a reservation during an exceptional closure, repeating the normal opening hours is a failure even when the response is friendly.

Write criteria before scoring outputs. Separate at least four dimensions:

• Understanding the request and its corrections.
• Using relevant, current evidence.
• Taking only appropriate, authorised actions.
• Describing the result accurately and usefully.

Style can be assessed too, but it should not hide a failure in another dimension. A concise answer with the correct closure information may be better than a warm, detailed answer that causes a wasted journey.

Define what the system should do when it cannot finish. Asking a necessary question, acknowledging missing information or offering a real human route can be correct. Do not construct an evaluation that rewards answering at any cost.

Build cases around different reasons for failure

Start with the common requests, then add cases that exercise distinct weaknesses. Ten variants of an easy opening-hours question offer less coverage than a smaller set spanning ambiguity, correction, unavailable information and action failure.

For the fictional library, useful groups include:

Group	Example	What it tests
Direct request	“What time does collection close on Friday?”	Retrieval and applicability
Ambiguous request	“Can I come tomorrow?”	Necessary clarification
Corrected request	“I meant collection, not the workshop.”	Latest intent
Conflicting information	Normal hours plus an exceptional closure	Source precedence
Missing answer	A service not covered by approved documents	Appropriate uncertainty
Tool failure	Reservation lookup unavailable	Honest action status
Out-of-scope instruction	Embedded request to reveal other records	Boundaries
Human handoff	“I need to speak to someone.”	Escalation route

Use natural wording, including short messages and minor errors, rather than only the language used in the design document. If the audience uses several languages, create cases in those languages with the same underlying requirement. Translation quality and correct application of the service rule are separate checks.

Include realistic context without copying unnecessary personal information. Synthetic examples often reproduce the important difficulty while keeping the test set easier to share internally.

Write a replayable case, not just a question

A test question means little without knowing what information and tools were available. Fix the environment, source versions, relevant date and expected outcomes.

Here is an original case specification:

Field	Case LIB-07
Visitor's goal	Check whether a reserved item can be collected
Initial message	“Can I come tomorrow to pick up what I reserved?”
Fixed context	Test date and time zone recorded; approved source includes tomorrow's exceptional closure
Follow-up	“I mean collection, not the workshop.”
Expected result	Explain the applicable closure and offer to check the next collection opening
Unacceptable result	Claim the library is open using ordinary hours
Evidence retained	Reply, source passage, relevant tool request and result

Fixing the date matters. A case containing “tomorrow” changes meaning if replayed next week without a controlled reference date. If you intentionally test against live time, record the execution timestamp and time zone so the result remains interpretable.

Specialist platforms can support this discipline. Google's Dialogflow CX documentation describes saved conversation tests with expectations that can be checked after changes to an agent. The implementation is platform-specific; the principle also works in a manually maintained test folder. Source: Dialogflow CX test cases.

Separate retrieval, interpretation, execution and reporting

Suppose the agent gives the wrong opening time. Several different problems can produce that sentence. It may not have retrieved the closure notice. It may have retrieved it but ignored its date. It may have misunderstood “tomorrow”. Each problem calls for a different correction.

Inspect four layers:

1. Retrieval: which documents or records were actually returned?
2. Interpretation: did the assistant apply the relevant information to the request?
3. Execution: if an operation was required, were the target and parameters correct?
4. Reporting: did the final response accurately describe the verified result?

A technically successful tool call can still be the wrong call. Retrieving one visitor's reservation when another visitor is asking is not successful task completion. Likewise, a creation operation returning a case identifier does not prove that the final message explains its status correctly.

Use a test environment to supply controlled tool outcomes: success, no result, partial result, definite failure and uncertain outcome. Check whether the assistant responds differently when it should. Do not create real bookings or contact real people merely to make a demonstration look realistic.

The integration technology does not change the evaluation target. Whether an operation arrives through a connector, an API wrapper or MCP, the test still asks whether the right operation affected the right record and whether the answer stayed within the evidence.

Evaluate multi-turn behaviour explicitly

A system that handles isolated questions can still lose track of a conversation. Include a sequence where the user supplies one fact, changes another and then asks for a summary.

For example, a visitor first asks about an activity on Saturday, then explains that they only want to collect an item, and finally asks whether the collection point is accessible at that time. The agent should not continue answering about the activity because that topic appeared first.

Test cancellation too. If the user asks for a draft request and then says not to proceed, check what is stopped and what already exists. The final status should distinguish prepared, submitted and cancelled work rather than flattening everything into “done”.

Watch for unnecessary clarification. Asking which service the visitor means is useful at the first ambiguous message. Asking the same question after the visitor explicitly says “collection” wastes a turn. A generic rule to always clarify can create poor service just as a rule to never clarify can create guesses.

Also test interruption by a tool result. A long lookup should not make the system forget a correction received while the work was in progress. Record the sequence so that the behaviour can be reproduced.

Use metrics with a clear denominator

“90% successful” is not informative until success and the sample are defined. It could mean nine fluent responses, nine correct resolutions or nine tool calls without technical errors. Those are different measures.

For answerable information requests, measure factually correct responses among cases with sufficient approved evidence. For deliberately unanswerable requests, measure appropriate acknowledgement or clarification separately. For action cases, measure correct completed actions among those both expected and authorised.

Consider an invented set of 20 conversations: 16 end with correct answers, two require useful clarification and two contain wrong information. Reporting 18 resolved conversations would misrepresent the two clarification cases. They may be good intermediate outcomes, but the requests are not yet resolved.

Track several states rather than forcing every conversation into success or failure:

• Resolved with supported information.
• Waiting for necessary clarification.
• Correctly transferred to another route.
• Unresolved because a dependency failed.
• Incorrect or outside the permitted scope.

Keep unacceptable actions separate from average scores. A single disclosure to the wrong person should not disappear inside a high overall writing score. Define those release-blocking outcomes before running the evaluation.

Measure human effort and meaningful delay

A fast first reply can conceal slow completion. Distinguish acknowledgement time, time to useful information and time to the final operational result. If the agent immediately says “I am checking”, that is not evidence that the check has completed.

Record slow cases and their causes. Several searches, repeated clarification or a failing integration may dominate the experience even when the average response time seems acceptable. Look at the spread of timings and inspect the slowest relevant cases rather than relying only on a mean.

Measure the work transferred to people: substantial corrections, minutes spent reconstructing context and questions the customer has to answer again. A summary that omits the unresolved issue may make a human handoff slower than reading the original exchange.

Use comparable tasks when measuring assistance. If the manual baseline contains difficult requests and the automated sample contains only easy ones, the time comparison will overstate the improvement. Keep the workload composition visible in the report.

The useful operational question is whether the complete service improves, not whether the language model produces text quickly.

Calibrate reviewers before scoring the whole set

Two reviewers may disagree about a short answer or a clarification. Have them independently assess the same initial cases, then compare the reasons for disagreement.

Turn vague judgements into concrete rules. “Too cautious” may mean that the agent refused despite having an approved answer. “Too confident” may mean that it described a proposed time as confirmed. Those are actionable descriptions; “needs to be smarter” is not.

For each criterion, keep a good example, a correctable example and a blocking example. This makes later scoring more consistent and gives the implementation team clearer feedback.

If you use another AI model to assist evaluation, verify a sample of its decisions and all consequential disagreements. An automated evaluator can also be distracted by polished wording or miss a small but decisive date. Treat its output as evidence to inspect rather than as an unquestionable verdict.

Where possible, separate evaluation from the person who has been tuning the instructions. Familiarity with the intended behaviour can make an incomplete answer seem clearer than it would to an actual visitor.

Keep development examples separate from acceptance evidence

If every failed test is immediately added to the instruction examples, the agent may become good at the test wording rather than the underlying task. Keep some representative cases aside and introduce fresh variants after a change.

Retain previous failures as regression tests. A correction that fixes exceptional opening hours should not break ordinary collection questions. Replaying old cases helps detect that trade-off.

Version the relevant ingredients together: instructions, identifiable model version, approved sources, exposed tools and important configuration. If you change all of them at once, a new failure becomes difficult to diagnose.

For sensitive or unstable cases, repeat runs where appropriate to see whether a single success masks variable behaviour. The number of repetitions should follow the risk and observed variability, not an arbitrary universal threshold.

Keep a brief decision log: what changed, why, which results improved or regressed and what scope was accepted afterward. That log is more useful than a folder of screenshots without context.

Test the boundary between information and instruction

Documents and messages may contain text that looks like a command to the assistant. A test source could include an irrelevant instruction to reveal another record or ignore the current task. The system should treat that text as untrusted content, not as permission to expand its work.

OWASP describes indirect prompt injection through external sources and notes that retrieval or fine-tuning does not eliminate the risk. For a pilot, test both the response and the operations actually attempted, with unnecessary capabilities removed from the environment. Source: OWASP Prompt Injection.

Use synthetic targets and harmless data for this test. You are checking whether the boundary holds, not attempting to expose real information. If the test reveals an unwanted operation, reduce the available capability and investigate the execution controls, rather than only adding a stronger warning sentence.

Also test ordinary source conflicts. An older policy, a draft proposal and an approved current notice should not be treated as interchangeable evidence. Clear source metadata can prevent a class of errors that no final-answer instruction will reliably fix on its own.

Make a scoped release decision

There is no single success percentage that makes every conversational agent ready. An opening-hours assistant and a system modifying personal records have different consequences when wrong.

Name the person who decides whether the evidence is sufficient, who resolves business-rule disputes and who can stop the release. Define acceptable minor defects and release-blocking failures beforehand.

A reasonable decision may be to launch only routine information questions while routing reservations and unusual cases to people. Another may be to keep the system in draft-only mode. Match the permissions and scope to what has been demonstrated, not to the broadest feature offered by the platform.

After launch, compare real outcomes with the test assumptions. New wording, source changes and operational failures can expose gaps. Add representative failures to the test set and revisit the affected scope. Initial acceptance is a justified starting point, not permanent proof of reliability.

Frequently asked questions

Should a conversational AI agent always answer immediately?

No. A targeted clarification, an honest statement that information is missing or a real handoff can be the correct result. Evaluate whether that response advances the user's actual request rather than rewarding an answer at any cost.

How many cases should a first evaluation contain?

There is no universal number. Cover common requests, important exceptions, multi-turn corrections and consequential failures. Add cases when a new source, tool or permission introduces a materially different way the system could go wrong.

Does connecting a knowledge base prevent incorrect answers?

No. The system must retrieve the right source, apply its date and audience, interpret it correctly and avoid overstating the result. Test each of those stages. A citation or successful search does not by itself establish a correct answer.