How to test an AI assistant’s answers

Testing an AI assistant requires comparing answers against criteria defined before the trial. A fluent answer can be wrong, and a refusal can be correct when information or access is missing. This guide shows how to build an evaluation set, distinguish types of error and decide whether a change improves the service.

By Lyntev. Updated on .
About 5 minutes to read.

From review to publicationA process with approval criteria at every stage.

Translation published on . Original Portuguese guide; source revision .

By the end, you can:

  • Write cases and expected answers
  • Compare versions without contaminating the evaluation
  • Record critical errors, costs and review work

Define what makes an answer acceptable

Choose observable criteria: it answers the question, uses the authorised source, preserves relevant conditions and does not present an assumption as a fact. For actions, add the record that should exist and the permissions required. Do not rely solely on “looks good” or a scale without explaining what distinguishes the scores.

Anthropic’s evaluation documentation recommends specific, measurable criteria. Apply that idea to your service: an answer about availability should state whether it checked the appointment calendar or merely explained the process. A polite answer that invents an available time fails the criterion, even if it follows the brand’s tone.

Define what makes an answer acceptable
DimensionAcceptableObservable failure
ContentFacts and conditions supported by the sourceInvented deadline or service
UsefulnessResolves the question or asks for the missing informationVague answer that forces the person to start again
AccessUses authorised information onlyData from another account
ActionPermitted change confirmed in the systemSuccess message without the operation being completed

Build normal, difficult and missing-information cases

Collect questions representative of the work, removing unnecessary data and respecting authorised use. Add rare cases with serious consequences. Separate questions covered by the sources, ambiguous questions, missing information and attempts to access something outside the permitted scope. A collection of easy questions alone measures too narrow a part of operations.

Keep one part for developing the configuration and another for evaluating the chosen version. Do not keep rewriting the held-out cases until the version passes. When a new error enters the set, record the change; if you compare historical figures, identify that the set is no longer exactly the same.

Write a case another person can assess

In fictitious case T-07, the approved source says: “The quote depends on assessment of the equipment and the work.” The question is “How much does it cost to repair this machine?” and includes neither a model nor a fault. An acceptable answer explains that information is missing and asks for what is needed, without inventing a price or promising a free assessment that was not specified.

The reference answer does not need to be a single sentence to copy. List required statements, prohibited statements and the next action. This allows two different wordings to pass for the same reason. For exact fields or calculations, use exact checks; for explanations, use an editorial rubric with examples of passing and failing answers.

Fictitious evaluation caseYAML
id: T-07
pergunta: How much does it cost to repair this machine?
fonte: procedimento-orcamento-v3
obrigatorio:
  - state that the price depends on assessment
  - ask for identification of the equipment and the problem
nao_admitido:
  - invent a price
  - promise an undocumented free service
acao_externa_permitida: nenhuma

Distinguish retrieval, interpretation and execution

If the answer is wrong, first check what information was retrieved. A missing source, an old version and an incorrect interpretation require different corrections. Microsoft distinguishes evaluators for retrieval, relevance, completeness and grounding in sources; this separation is useful even when you carry out the evaluation in a spreadsheet.

When tools are involved, also examine the proposed arguments, authorisation decision and operation result. A correct final sentence does not undo an unauthorised change already made. Likewise, a technical error that prevented any answer should be recorded as a failure or an inconclusive trial, not as a successful safe refusal.

Repeat the same trial on the versions you want to compare

Record the model, instructions, sources, tools, parameters and date. Use the same set and conditions for the comparison. Run the most sensitive cases several times because answers can vary. Keep the result for each case as well as the total, to discover whether an improvement in one task harmed another.

In a fictitious trial, 20 cases run three times produce 60 answers. If 54 meet the rubric, the observed rate is 90%. If two of the six failures expose data from another account, the 90% total does not justify approval. The small sample describes that trial; it does not prove that 90% of future conversations will be correct.

Repeat the same trial on the versions you want to compare
RecordVersion AVersion B
Set and sourcesSame identified versionSame identified version
Result per caseAccepted, failed or inconclusiveAccepted, failed or inconclusive
Critical errorsCount and evidenceCount and evidence
OperationTime, usage and reviewTime, usage and review

Use automated evaluation with human checking

Checks of structure, field presence and calculations can be automated. A second model can help apply a rubric to many answers, but it can also misinterpret the case. Compare a sample of its evaluations with people who know the operation before accepting classifications in bulk.

Define how to handle disagreements and, where feasible, review without revealing which version generated the answer. Avoid letting the assistant itself decide whether its answer was good through a generic question. A useful evaluation presents the criterion, evidence and reason for the decision; the score alone hides the problem.

Turn the evaluation into a version decision

Before making the change available, establish which failures prevent approval and who reviews them. Record the points not tested. Afterwards, use errors observed in operations to improve the cases without keeping unnecessary personal data. A new source, function or permission should trigger a repeat of the affected trials.

Worksheet for your project

Task and scope of the evaluation:
Versions of the model, instructions, sources and tools:
Development set and held-out set:
Case, question and context:
Expected source and mandatory conditions:
Statements or actions not permitted:
Answer, retrieved data and actions observed:
Criteria met / failed / inconclusive:
Severity, evidence and person responsible for review:
Number of runs and results per case:
Time, usage and manual review:
Version decision and trials to repeat:

Sources and criteria

Content prepared with AI assistance and research using technical sources, with checks of the relevant examples and calculations. The references support the guide’s decisions; examples identified as fictitious let you try out the method.

Wellinton Oliveira

Written by

CEO of Lyntev

Technology with context, original analysis and sources you can consult.