Applied AI Capstone: Build a Source-Grounded Assistant

This capstone ties the learning path together with a small, inspectable document assistant. You will load a fictional handbook, retrieve permitted source text, return a cited result or an abstention, and test the boundaries before considering a live model or web deployment.

The default program deliberately uses lexical word overlap for retrieval and extracts source text for its answer. It does not compute embeddings or call a language model. An optional adapter adds live generation through the OpenAI Responses API, but still uses the lexical retriever. That distinction lets you study the application wiring without an account, paid request, or hidden model behavior.

Download and run the project

Download the version 1 project ZIP, extract it into a new directory, and open a terminal in the extracted project folder. It requires Python 3.10 or later and uses only the standard library. No package installation is needed.

Run these commands; use python3 if your operating system uses that name for Python 3:

python --version
python document_assistant.py "What is the refund window?"
python evaluate.py
python -m unittest -v

The first question should produce JSON with mode lexical-extractive-fixture, status answered, citation refunds-v1, and an answer containing “14 calendar days.” The result copies the relevant fictional source text rather than synthesizing a model answer.

The archive contains document_assistant.py, three fictional handbook records in documents.json, five acceptance cases in evaluation.json, an evaluation runner, unit tests, a short README, and an MIT license for the original example code. There are no real customer records or provider credentials.

Follow one question through the system

The loader validates required source fields and rejects duplicate IDs. For a public fixture identity, the retriever excludes staff records before calculating scores. It then finds meaningful query words in each remaining source’s title and text, sorts by overlap score, and keeps at most two hits above the teaching threshold.

The default answer function joins the selected source text and returns its IDs. A separate validator checks the answer contract, citation uniqueness, and citation membership in the retrieved context. If nothing is retrieved, the result is insufficient_context with no citations.

This is a baseline with deliberately visible weaknesses. It can miss paraphrases and can retrieve an irrelevant source whose words overlap. Its threshold is not a calibrated confidence value. The tiny corpus also does not exercise long-document parsing, chunk selection, or large-index behavior.

Check abstention and access boundaries

Run these cases before changing the code:

python document_assistant.py "What is the price of Bitcoin?"
python document_assistant.py "What is the internal escalation code?"
python document_assistant.py "What is the internal escalation code?" --audience staff

The unrelated question and public escalation question should abstain. The staff fixture identity should retrieve the fictional staff record. The code DEMO-ORANGE is synthetic test data, not a real secret.

The audience argument simulates an identity for a local exercise. It is not authentication or production authorization. A real service must derive permissions from a trusted authenticated session; accepting staff from a client-controlled parameter would defeat the boundary.

Now alter one evaluation expectation intentionally. The runner should identify the failing case and exit nonzero. Restore it afterward. A test suite that cannot fail is not a useful release gate.

Inspect the verification evidence

Metarix ran the version 1 project with Python 3.12.14. The commands above completed locally: all 20 unit tests passed and all five deterministic fixture cases passed. Independent review also ran the project from a fresh copied directory, including invocation from another working directory and failure when live credentials were absent.

The unit tests cover default offline behavior, public/staff retrieval separation, malformed answers, unknown source IDs, provider refusal/incomplete-output handling, and mocked HTTP integration. The provider tests use fabricated HTTP responses; they do not call a live model. The five acceptance cases are deliberately constructed for the small corpus, not an estimate of production accuracy.

Valid citation IDs establish that references came from the supplied context. They do not prove semantic support. The offline baseline extracts text; any live generated answer needs a separate faithfulness assessment.

Optional: add live generation explicitly

The adapter follows the OpenAI Responses API reference and Structured Outputs guide, checked November 8, 2026. We tested its code with mocks, not a paid API request.

If you choose to run it, set OPENAI_API_KEY securely in the process environment and OPENAI_MODEL to a currently available model that supports the endpoint and JSON-schema output. Then explicitly select the live provider:

python document_assistant.py "What is the refund window?" --provider openai

This can send retrieved text to an external provider and incur charges. The adapter sets store to false, requests at most 600 output tokens, uses a 20-second client timeout, and makes no automatic retry. Those settings do not establish zero retention under every provider policy or guarantee a fixed cost. Review the terms that apply to your account before using private data.

The adapter rejects incomplete results, malformed JSON, missing citations, and references outside the retrieved context. A provider error produces a controlled nonzero exit. Keep the model ID configurable and record the actual ID, settings, corpus, and date of any live experiment.

Complete the project with a measured extension

First add at least five evaluation cases of your own, including a paraphrase, an unsupported question with overlapping words, a stale policy, conflicting evidence, and an access restriction. Label expected evidence before changing the retriever.

Then choose one improvement. You could compare an approved embedding retriever against the lexical baseline, add source-version selection, or introduce section-aware chunking for a longer fictional document. Keep the old baseline results so the comparison remains inspectable. If you do not run an embedding or live-model experiment, label it as a proposed extension.

For a portfolio handoff, include the problem statement, data provenance, architecture, exact run commands, evaluation cases, observed failures, and a short explanation of the tradeoff you chose. Avoid presenting a handful of fixture passes as proof of general AI reliability.

Definition of done

• The project runs from a fresh extracted directory without a provider account in default mode.
• Known questions cite the expected fixture source; unsupported and restricted public questions abstain.
• Invalid outputs and operational errors fail visibly without printing credentials.
• Your evaluation includes new failure cases and makes a failed expectation exit nonzero.
• Any claimed improvement has recorded inputs, configuration, and results.
• The handoff identifies what remains before web deployment: real authentication, server-side authorization, TLS, rate limits, monitoring, concurrency controls, and rollback.

The CLI is not a hosted service and should not be exposed directly to the internet. A staging plan from the reliability lesson is the next step if you want to operate it for other people.

The finished artifact demonstrates a tested application skeleton and an honest evaluation process. Adding real embeddings, live generation, or a production interface is a further engineering step whose quality must be measured, not assumed.