---
title: "How to test an AI assistant’s answers · Lyntev"
description: "Build test cases, acceptance criteria and a comparison between versions. Includes a completed rubric, critical errors and the limits of metrics."
url: "https://lyntev.com/en/guides/test-ai-assistant-answers"
language: "en-GB"
date_published: "2026-11-21T17:20:13.047Z"
date_modified: "2026-11-21T17:20:13.047Z"
---

1. [Home](<https://lyntev.com/en>)
2. [Practical guides](<https://lyntev.com/en/guides>)
3. How to test an AI assistant’s answers

# How to test an AI assistant’s answers

Testing an AI assistant requires comparing answers against criteria defined before the trial. A fluent answer can be wrong, and a refusal can be correct when information or access is missing. This guide shows how to build an evaluation set, distinguish types of error and decide whether a change improves the service.

By [Lyntev](<https://lyntev.com/en/company>). Updated on 17 November 2026.
About 5 minutes to read.

[Start reading](<https://lyntev.com/en/guides/test-ai-assistant-answers#leitura>)[Download worksheet](<https://lyntev.com/recursos/en/testar-respostas-assistente-ia.txt>)

**From review to publication**A process with approval criteria at every stage.

Translation published on 21 November 2026. [Original Portuguese guide](<https://lyntev.com/guias/testar-respostas-assistente-ia>); source revision 17 November 2026.

By the end, you can:

- Write cases and expected answers
- Compare versions without contaminating the evaluation
- Record critical errors, costs and review work

In this guide

1. [Define what makes an answer acceptable](<https://lyntev.com/en/guides/test-ai-assistant-answers#criterios-da-resposta>)
2. [Build normal, difficult and missing-information cases](<https://lyntev.com/en/guides/test-ai-assistant-answers#conjunto-de-casos>)
3. [Write a case another person can assess](<https://lyntev.com/en/guides/test-ai-assistant-answers#caso-preenchido>)
4. [Distinguish retrieval, interpretation and execution](<https://lyntev.com/en/guides/test-ai-assistant-answers#localizar-a-falha>)
5. [Repeat the same trial on the versions you want to compare](<https://lyntev.com/en/guides/test-ai-assistant-answers#comparacao-entre-versoes>)
6. [Use automated evaluation with human checking](<https://lyntev.com/en/guides/test-ai-assistant-answers#avaliadores-e-revisao>)
7. [Turn the evaluation into a version decision](<https://lyntev.com/en/guides/test-ai-assistant-answers#ficha-de-avaliacao>)
8. [Sources and criteria](<https://lyntev.com/en/guides/test-ai-assistant-answers#fontes>)

## Define what makes an answer acceptable

Choose observable criteria: it answers the question, uses the authorised source, preserves relevant conditions and does not present an assumption as a fact. For actions, add the record that should exist and the permissions required. Do not rely solely on “looks good” or a scale without explaining what distinguishes the scores.

Anthropic’s evaluation documentation recommends specific, measurable criteria. Apply that idea to your service: an answer about availability should state whether it checked the appointment calendar or merely explained the process. A polite answer that invents an available time fails the criterion, even if it follows the brand’s tone.

Define what makes an answer acceptable

| Dimension | Acceptable | Observable failure |
| --- | --- | --- |
| Content | Facts and conditions supported by the source | Invented deadline or service |
| Usefulness | Resolves the question or asks for the missing information | Vague answer that forces the person to start again |
| Access | Uses authorised information only | Data from another account |
| Action | Permitted change confirmed in the system | Success message without the operation being completed |

- [Anthropic: success criteria and evaluations](<https://platform.claude.com/docs/en/test-and-evaluate/develop-tests>)

## Build normal, difficult and missing-information cases

Collect questions representative of the work, removing unnecessary data and respecting authorised use. Add rare cases with serious consequences. Separate questions covered by the sources, ambiguous questions, missing information and attempts to access something outside the permitted scope. A collection of easy questions alone measures too narrow a part of operations.

Keep one part for developing the configuration and another for evaluating the chosen version. Do not keep rewriting the held-out cases until the version passes. When a new error enters the set, record the change; if you compare historical figures, identify that the set is no longer exactly the same.

## Write a case another person can assess

In fictitious case T-07, the approved source says: “The quote depends on assessment of the equipment and the work.” The question is “How much does it cost to repair this machine?” and includes neither a model nor a fault. An acceptable answer explains that information is missing and asks for what is needed, without inventing a price or promising a free assessment that was not specified.

The reference answer does not need to be a single sentence to copy. List required statements, prohibited statements and the next action. This allows two different wordings to pass for the same reason. For exact fields or calculations, use exact checks; for explanations, use an editorial rubric with examples of passing and failing answers.

Fictitious evaluation caseYAML

```
id: T-07
pergunta: How much does it cost to repair this machine?
fonte: procedimento-orcamento-v3
obrigatorio:
  - state that the price depends on assessment
  - ask for identification of the equipment and the problem
nao_admitido:
  - invent a price
  - promise an undocumented free service
acao_externa_permitida: nenhuma
```

## Distinguish retrieval, interpretation and execution

If the answer is wrong, first check what information was retrieved. A missing source, an old version and an incorrect interpretation require different corrections. Microsoft distinguishes evaluators for retrieval, relevance, completeness and grounding in sources; this separation is useful even when you carry out the evaluation in a spreadsheet.

When tools are involved, also examine the proposed arguments, authorisation decision and operation result. A correct final sentence does not undo an unauthorised change already made. Likewise, a technical error that prevented any answer should be recorded as a failure or an inconclusive trial, not as a successful safe refusal.

- [Microsoft: dimensions for evaluating answers supported by sources](<https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators>)

## Repeat the same trial on the versions you want to compare

Record the model, instructions, sources, tools, parameters and date. Use the same set and conditions for the comparison. Run the most sensitive cases several times because answers can vary. Keep the result for each case as well as the total, to discover whether an improvement in one task harmed another.

In a fictitious trial, 20 cases run three times produce 60 answers. If 54 meet the rubric, the observed rate is 90%. If two of the six failures expose data from another account, the 90% total does not justify approval. The small sample describes that trial; it does not prove that 90% of future conversations will be correct.

Repeat the same trial on the versions you want to compare

| Record | Version A | Version B |
| --- | --- | --- |
| Set and sources | Same identified version | Same identified version |
| Result per case | Accepted, failed or inconclusive | Accepted, failed or inconclusive |
| Critical errors | Count and evidence | Count and evidence |
| Operation | Time, usage and review | Time, usage and review |

## Use automated evaluation with human checking

Checks of structure, field presence and calculations can be automated. A second model can help apply a rubric to many answers, but it can also misinterpret the case. Compare a sample of its evaluations with people who know the operation before accepting classifications in bulk.

Define how to handle disagreements and, where feasible, review without revealing which version generated the answer. Avoid letting the assistant itself decide whether its answer was good through a generic question. A useful evaluation presents the criterion, evidence and reason for the decision; the score alone hides the problem.

## Turn the evaluation into a version decision

Before making the change available, establish which failures prevent approval and who reviews them. Record the points not tested. Afterwards, use errors observed in operations to improve the cases without keeping unnecessary personal data. A new source, function or permission should trigger a repeat of the affected trials.

### Worksheet for your project

```
Task and scope of the evaluation:
Versions of the model, instructions, sources and tools:
Development set and held-out set:
Case, question and context:
Expected source and mandatory conditions:
Statements or actions not permitted:
Answer, retrieved data and actions observed:
Criteria met / failed / inconclusive:
Severity, evidence and person responsible for review:
Number of runs and results per case:
Time, usage and manual review:
Version decision and trials to repeat:
```

## Sources and criteria

Content prepared with AI assistance and research using technical sources, with checks of the relevant examples and calculations. The references support the guide’s decisions; examples identified as fictitious let you try out the method.

- [Anthropic: definir critérios e construir avaliações](<https://platform.claude.com/docs/en/test-and-evaluate/develop-tests>)
- [Microsoft: avaliação de recuperação e respostas RAG](<https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators>)

## Continue to your next decision.

- [Prepare the assistant’s information](<https://lyntev.com/en/guides/prepare-ai-assistant-information>)
- [Create reusable work instructions](<https://lyntev.com/en/guides/create-reusable-work-prompts>)
- [Protect the assistant from unauthorised instructions](<https://lyntev.com/en/guides/protect-ai-assistant-prompt-injection>)

[Explore the guide library](<https://lyntev.com/en/guides>)

## Put this decision into practice.

Steps, examples and worksheets to prepare the work in your business.

[Find a reading path](<https://lyntev.com/en/guides#percursos-title>)

### [How to make AI hand a conversation over to a person](<https://lyntev.com/en/guides/handover-ai-conversation-to-person>)

A useful handover ends with a responsible person, context and no interference from automation.

### [How to keep customer-support answers up to date](<https://lyntev.com/en/guides/keep-customer-support-answers-updated>)

Give answers an approved version and unanswered questions somewhere to go.

## Explore the research behind this decision.

Articles with sources, clearly identified examples and worksheets for assessing websites, AI customer service and business management systems.

[Explore all articles](<https://lyntev.com/en/research>)

Artificial intelligence · About 11 min

### [AI in customer service: what studies show and how to evaluate it in your business](<https://lyntev.com/en/research/ai-in-customer-service>)

See what studies show about AI in customer service. Learn how to test answers, quality, handover to the team and the time spent on the work in your business.

## Would you like to apply this to your business?

Send us your worksheet or explain what you need to improve. We help you choose the next step for your project.

- [Lyntev assistant Immediate answers about services, prices and timescales, with a cost estimate. Read the assistant guide](<https://lyntev.com/en/help/ai-assistant>)
- [WhatsApp +351 928 412 891. The message is only sent when you confirm it in WhatsApp. Write on WhatsApp](<https://wa.me/351928412891?text=Hello!%20I%20came%20from%20the%20Lyntev%20website%20and%20would%20like%20to%20discuss%20a%20project.>)
- [Form Describe what you need and receive the reply by email. Complete the form](<https://lyntev.com/en/contact>)
- [Email contacto@lyntev.com Send an email](<mailto:contacto@lyntev.com?subject=Request%20for%20information>)

![Wellinton Oliveira](<https://lyntev.com/media/autores/wellinton-oliveira.webp>)

Written by

[Wellinton Oliveira](<https://lyntev.com/en/company>)

CEO of Lyntev

Technology with context, original analysis and sources you can consult.
