How to test an AI assistant’s answers
Testing an AI assistant requires comparing answers against criteria defined before the trial. A fluent answer can be wrong, and a refusal can be correct when information or access is missing. This guide shows how to build an evaluation set, distinguish types of error and decide whether a change improves the service.
By the end, you can:
- Write cases and expected answers
- Compare versions without contaminating the evaluation
- Record critical errors, costs and review work
Define what makes an answer acceptable
Choose observable criteria: it answers the question, uses the authorised source, preserves relevant conditions and does not present an assumption as a fact. For actions, add the record that should exist and the permissions required. Do not rely solely on “looks good” or a scale without explaining what distinguishes the scores.
Anthropic’s evaluation documentation recommends specific, measurable criteria. Apply that idea to your service: an answer about availability should state whether it checked the appointment calendar or merely explained the process. A polite answer that invents an available time fails the criterion, even if it follows the brand’s tone.
| Dimension | Acceptable | Observable failure |
|---|---|---|
| Content | Facts and conditions supported by the source | Invented deadline or service |
| Usefulness | Resolves the question or asks for the missing information | Vague answer that forces the person to start again |
| Access | Uses authorised information only | Data from another account |
| Action | Permitted change confirmed in the system | Success message without the operation being completed |
Build normal, difficult and missing-information cases
Collect questions representative of the work, removing unnecessary data and respecting authorised use. Add rare cases with serious consequences. Separate questions covered by the sources, ambiguous questions, missing information and attempts to access something outside the permitted scope. A collection of easy questions alone measures too narrow a part of operations.
Keep one part for developing the configuration and another for evaluating the chosen version. Do not keep rewriting the held-out cases until the version passes. When a new error enters the set, record the change; if you compare historical figures, identify that the set is no longer exactly the same.
Write a case another person can assess
In fictitious case T-07, the approved source says: “The quote depends on assessment of the equipment and the work.” The question is “How much does it cost to repair this machine?” and includes neither a model nor a fault. An acceptable answer explains that information is missing and asks for what is needed, without inventing a price or promising a free assessment that was not specified.
The reference answer does not need to be a single sentence to copy. List required statements, prohibited statements and the next action. This allows two different wordings to pass for the same reason. For exact fields or calculations, use exact checks; for explanations, use an editorial rubric with examples of passing and failing answers.
id: T-07
pergunta: How much does it cost to repair this machine?
fonte: procedimento-orcamento-v3
obrigatorio:
- state that the price depends on assessment
- ask for identification of the equipment and the problem
nao_admitido:
- invent a price
- promise an undocumented free service
acao_externa_permitida: nenhumaDistinguish retrieval, interpretation and execution
If the answer is wrong, first check what information was retrieved. A missing source, an old version and an incorrect interpretation require different corrections. Microsoft distinguishes evaluators for retrieval, relevance, completeness and grounding in sources; this separation is useful even when you carry out the evaluation in a spreadsheet.
When tools are involved, also examine the proposed arguments, authorisation decision and operation result. A correct final sentence does not undo an unauthorised change already made. Likewise, a technical error that prevented any answer should be recorded as a failure or an inconclusive trial, not as a successful safe refusal.
Repeat the same trial on the versions you want to compare
Record the model, instructions, sources, tools, parameters and date. Use the same set and conditions for the comparison. Run the most sensitive cases several times because answers can vary. Keep the result for each case as well as the total, to discover whether an improvement in one task harmed another.
In a fictitious trial, 20 cases run three times produce 60 answers. If 54 meet the rubric, the observed rate is 90%. If two of the six failures expose data from another account, the 90% total does not justify approval. The small sample describes that trial; it does not prove that 90% of future conversations will be correct.
| Record | Version A | Version B |
|---|---|---|
| Set and sources | Same identified version | Same identified version |
| Result per case | Accepted, failed or inconclusive | Accepted, failed or inconclusive |
| Critical errors | Count and evidence | Count and evidence |
| Operation | Time, usage and review | Time, usage and review |
Use automated evaluation with human checking
Checks of structure, field presence and calculations can be automated. A second model can help apply a rubric to many answers, but it can also misinterpret the case. Compare a sample of its evaluations with people who know the operation before accepting classifications in bulk.
Define how to handle disagreements and, where feasible, review without revealing which version generated the answer. Avoid letting the assistant itself decide whether its answer was good through a generic question. A useful evaluation presents the criterion, evidence and reason for the decision; the score alone hides the problem.
Turn the evaluation into a version decision
Before making the change available, establish which failures prevent approval and who reviews them. Record the points not tested. Afterwards, use errors observed in operations to improve the cases without keeping unnecessary personal data. A new source, function or permission should trigger a repeat of the affected trials.
Worksheet for your project
Task and scope of the evaluation: Versions of the model, instructions, sources and tools: Development set and held-out set: Case, question and context: Expected source and mandatory conditions: Statements or actions not permitted: Answer, retrieved data and actions observed: Criteria met / failed / inconclusive: Severity, evidence and person responsible for review: Number of runs and results per case: Time, usage and manual review: Version decision and trials to repeat:
Sources and criteria
Content prepared with AI assistance and research using technical sources, with checks of the relevant examples and calculations. The references support the guide’s decisions; examples identified as fictitious let you try out the method.
