Evaluate an AI App: Retrieval, Answers and Failures

A document assistant can succeed on one friendly demonstration and still fail on everyday questions. Evaluation turns that uncertainty into specific cases you can inspect, repeat, and use when changing the system.

This lesson separates retrieval quality from answer quality and application reliability. You will design a small evaluation file, run deterministic acceptance checks, and understand why a passing toy fixture is not a production accuracy estimate.

Define success before changing the model

For our handbook assistant, success means retrieving permitted evidence, answering only what that evidence supports, citing the supporting source, and abstaining when the required information is absent. It also means returning a controlled error when the provider or application fails.

Write those requirements before tuning prompts or retrieval thresholds. Otherwise, it is easy to redefine success around whichever output looks best. OpenAI’s evaluation best practices emphasizes task-specific datasets, clear criteria, and human calibration rather than judging quality by impression alone.

A good evaluation case includes a question, relevant source IDs where applicable, user/access context, expected behavior, and notes about why the case matters. It need not demand one exact sentence unless the task genuinely requires it.

Evaluate retrieval separately

If the right source never reaches the generator, investigate retrieval first. Check parsing, chunking, filters, query wording, index version, and ranking before changing the answer prompt.

For questions with labelled relevant passages, recall at k is the number of relevant passages retrieved in the top k divided by the total number of labelled relevant passages. If a question has two relevant passages and the top three results contain one, recall at 3 is 1/2.

This metric depends on the labels. An unlabelled but useful source can make an evaluation misleading. Inspect misses and keep source identifiers aligned with corpus versions.

An unsupported question has no labelled relevant source. Do not divide by zero or treat it as an ordinary recall case. Evaluate whether the application abstains and whether unrelated passages are wrongly used as evidence.

Evaluate answer support and usefulness

For each answer, ask whether it addresses the question, whether its factual claims are supported by the supplied sources, and whether important conditions were omitted. Check the citation next to the actual claim rather than treating any citation as proof.

Answer faithfulness and completeness can conflict. A short answer may be fully supported but omit a condition needed by the user. A polished answer may include unsupported details. Your rubric should make those distinctions visible.

Microsoft’s RAG evaluation guidance discusses multiple dimensions, including relevance and groundedness. Choose criteria that correspond to the task instead of combining unrelated numbers into an unexplained score.

Model-based graders can help organize larger evaluations, but their judgments also need calibration. Review disagreements with a human-labelled sample. A grader using a similar model can share weaknesses with the answer generator.

Include negative and boundary cases

Our capstone includes five deliberately small fixture cases: a refund question, a support-hours question, an unsupported topic, a restricted question as a public identity, and the same restricted question as a staff fixture identity.

The public and staff options simulate test identities. They do not implement authentication. In a real application, include permission checks at the server boundary and prove that the client cannot grant itself a privileged identity.

Add cases for malformed provider JSON, missing citations, invented source IDs, refusals, incomplete responses, timeouts, and rate limits. Those are application behaviors, not merely language-quality questions.

Then add realistic content problems: paraphrases, typos, outdated sources, conflicting policies, irrelevant retrieved text, and instructions embedded in a document. You will not cover every possible attack or failure with a short list, but recording known boundaries is more useful than hiding them behind an average score.

Keep a versioned evaluation record

Record the corpus snapshot, question set, retrieval configuration, prompt version, model identifier, relevant request settings, and execution date. Note whether the run used deterministic fixtures, mocked HTTP, or a live provider.

Track latency and usage for live experiments separately from output quality. Compare distributions where possible; one fast response does not establish tail latency. Our guides to LLM latency and throughput and cost per successful task explain the operational side.

Do not tune repeatedly on every evaluation question and then describe the result as unseen performance. Keep a separate set of cases for checking a proposed change, and expand the dataset with genuine failure examples after careful review and appropriate handling of user data.

Exercise: run checks, then create a failure

In the downloadable capstone, run python evaluate.py and inspect every case. The initial fixture should report 5/5 passing and exit zero. That shows the small, deliberately constructed example behaves as specified; it does not establish a general 100% accuracy rate.

Add a paraphrase such as “How long do I have to get my money back?” Label the refund source as relevant, then run the lexical baseline. If retrieval misses it, record a retrieval failure rather than blaming generation. This gives you a concrete target for a better retrieval experiment.

Add a stale-policy case and decide how version selection should work before implementing it. Add one unsupported question whose wording overlaps with a known policy so you can inspect false-positive retrieval.

You have completed the exercise when a failing case makes the runner exit nonzero, the failure is assigned to the correct component, and your proposed improvement can be tested against a fixed question set. Keep the failing evidence; a useful evaluation system should reveal weaknesses.

Documentation checked November 8, 2026. Capstone code verification covers deterministic fixture behavior and mocked provider handling, not live model accuracy.