An LLM eval is a repeatable set of test inputs with checks on the outputs, such as exact matches, JSON validation, code assertions or a rubric graded by another model. You run it whenever you change a prompt, model or app version to catch regressions before users do.

Is Promptfoo or Inspect better for LLM evaluation?

Promptfoo is usually faster to adopt for prompt and model comparisons, app regression tests and CI, because suites are written in YAML. Inspect is stronger for agent and tool-use evals that need sandboxes, and for teams that prefer writing evals in Python. Both are open source under the MIT license.

How many test cases does an LLM eval need?

There is no fixed number, but 30 to 100 well-chosen real inputs with sharp checks is a practical starting point. Add a new case every time a production bug or surprise shows up.

Are Promptfoo and Inspect free?

Both are free, open-source tools, so you pay only for the model calls your evals make. Promptfoo also sells Enterprise and on-premise plans with team dashboards, SSO and support.

Inspect VS Code extension and Inspect View log viewer showing an arc_challenge evaluation — UK AI Security Institute (inspect.aisi.org.uk)

Benchmarks · Guide

How to Build LLM Evals for Your App: Promptfoo vs Inspect

A practical guide to building a private LLM eval suite with Promptfoo or Inspect, plus how to move off OpenAI Evals before its Nov 30 shutdown.

Wajid Ali · Oct 31, 2026 · 10 min read

On this page

To evaluate an LLM for your own app, write down 30 to 100 real inputs your product has to handle, attach a check to each one (exact match, a JSON schema, a code assertion, or a rubric graded by a second model), and run the same suite every time you change a prompt or a model. Public leaderboards tell you which models are worth trying; only your own eval tells you which one to ship. This guide shows how to build one with the two open-source frameworks most teams reach for, Promptfoo and Inspect, and how to pick between them.

The timing matters. OpenAI's hosted Evals platform went read-only for existing users today, Oct 31, 2026, and the dashboard and API are scheduled to shut down on Nov 30, 2026. OpenAI's own migration guide points to Promptfoo. If your regression tests live there, you have a month to move them somewhere you control.

Why leaderboards are not enough

Benchmarks like SWE-bench, Terminal-Bench and Humanity's Last Exam measure a model on someone else's tasks, with someone else's scaffolding and scoring. We broke down what the coding ones actually test in SWE-bench vs Terminal-Bench, and this morning's HLE-Diamond leaderboard is a reminder that the top score can belong to a model you can't even call. Those numbers are useful for shortlisting. They say nothing about whether a model follows your system prompt, returns your JSON shape, refuses the right things, or stays under your latency budget.

A private eval answers narrower questions that matter more to a shipping team:

Did the new prompt fix the bug report without breaking the ten cases that already worked?
Is the cheaper model good enough for this task, or does it fail on the inputs that matter?
Does the agent still call the right tool with the right arguments after an SDK upgrade?
Did a provider's silent model update change behaviour on our traffic?

The anatomy of a useful eval

Every framework uses the same four parts, just with different names.

Test cases. Real inputs, ideally pulled from logs or support tickets, with the variables your prompt template needs. Include the boring common cases and the edge cases that have burned you.
The system under test. A prompt plus a model, or your whole app behind an HTTP endpoint, or an agent with tools in a sandbox.
Checks. Deterministic ones first: exact match, substring, regex, valid JSON, a few lines of code. Use model-graded rubrics only for things code can't judge, such as tone or factual correctness against a reference answer.
A baseline and a threshold. A score you record today and refuse to drop below, so the eval can block a merge instead of just producing a report.

Start small. Thirty well-chosen cases with sharp checks beat five hundred scraped prompts with a vague "is this good?" rubric. Grow the set every time production surprises you: each bug becomes a test case.

Option 1: Promptfoo, config-first and CI-friendly

Promptfoo is an MIT-licensed CLI and library for evaluating and red-teaming LLM apps. You describe prompts, providers and tests in a promptfooconfig.yaml file, run promptfoo eval, and open a local web viewer with promptfoo view. It installs through npm or Homebrew, or runs without installing via npx promptfoo@latest. OpenAI agreed to acquire Promptfoo in March 2026; both companies said the project stays open source and multi-provider, and the repository is still MIT licensed today.

Here is a minimal, illustrative suite for a support-ticket classifier. Swap echo for real providers such as an OpenAI, Anthropic or Bedrock model id once the structure works.

description: Support-ticket triage regression suite
prompts:
- 'Classify this ticket as billing, bug, or other. Reply with JSON {"label": "..."}. Ticket: {{ticket}}'
providers:
- echo
tests:
- vars:
ticket: I was charged twice this month
assert:
- type: is-json
- type: javascript
value: JSON.parse(output).label === 'billing'
- vars:
ticket: The export button crashes the app
assert:
- type: contains
value: crashes

What we actually ran. In this session we installed Promptfoo 0.125.1 and ran exactly that file. promptfoo validate config reported "Configuration is valid." The eval itself used Promptfoo's built-in echo provider, which returns the prompt unchanged, so no model was called: the JSON test failed and the substring test passed, as expected, and the CLI exited with a non-zero code (100). That exit code is what makes it useful as a CI gate. One practical note: the current release refused to start on Node 20 and asked for Node 22.22 or newer, so pin your CI runtime.

Assertions are where Promptfoo earns its keep. The deterministic set covers equals, contains, icontains, regex, starts-with, contains-any, contains-all, is-json, contains-json, and custom javascript or python checks, plus cost and latency. The model-graded set includes llm-rubric, factuality, model-graded-closedqa, answer-relevance and context-faithfulness for RAG, and similar for embedding similarity. You can weight assertions and set a minimum score per test.

Where it fits best:

Comparing several models or prompt variants side by side on the same cases.
Running in GitHub Actions or any CI on every pull request that touches a prompt.
Testing a whole app through an HTTP provider rather than a bare model.
Red teaming. The free Community tier includes red-team scanning with up to 10,000 probes a month; Enterprise and on-premise plans add team dashboards, SSO and support at custom prices.

Option 2: Inspect, code-first and built for agents

Inspect is an open-source Python framework from the UK AI Security Institute, developed with Meridian Labs and also MIT licensed. An Inspect eval is a Task made of three things: a dataset of samples, a solver that produces an answer (from a single generate() call up to a full tool-using agent), and a scorer. You install it with pip install inspect-ai and run tasks with inspect eval; inspect view opens a log viewer, and there is a VS Code extension.

The same triage check, written as an illustrative Inspect task:

from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import generate
@task
def triage():
return Task(
dataset=[
Sample(input="Classify as billing, bug, or other: I was charged twice", target="billing"),
Sample(input="Classify as billing, bug, or other: Export crashes the app", target="bug"),
],
solver=generate(),
scorer=includes(),
)

What we actually ran. We installed Inspect 0.3.286 and ran this task with inspect eval triage.py --model mockllm/model, Inspect's built-in mock model, so again no real model was called. It completed both samples, reported accuracy 0.000 (the mock returns placeholder text, so neither label matched) and wrote a log file you can open in the viewer. Point --model at a real provider, for example openai/<model> or anthropic/<model>, to get a meaningful score.

Where Inspect pulls ahead is agent evaluation. Its built-in react() agent gives a model tools such as bash() and python(), and tasks can run inside a Docker sandbox, with Kubernetes, Modal and others available through extensions. Its docs also show running external coding agents like Claude Code and Codex CLI under evaluation, and the Inspect Evals collection ships more than 200 ready-made benchmark implementations. Model grading supports a separate grader model, for example --model-role grader=<provider/model>, so the model under test doesn't mark its own homework.

Where it fits best:

Agentic tasks with tools, multiple turns and files in a sandbox.
Teams that want evals as Python code with custom solvers and scorers.
Reproducing a public benchmark on your own infrastructure before trusting a vendor's number.

Promptfoo vs Inspect at a glance

Authoring. Promptfoo: YAML config, with JavaScript or Python hooks. Inspect: Python tasks, solvers and scorers.
Best at. Promptfoo: prompt and model comparisons, app-level regression tests, red teaming. Inspect: agent and tool-use evals in sandboxes, benchmark reproduction.
Grading. Both support deterministic checks and model-graded rubrics with a configurable grader.
Results. Promptfoo: terminal table, local web viewer, exportable results. Inspect: log files, Inspect View and a VS Code extension.
Runtime. Promptfoo: Node.js CLI (npm, npx or Homebrew). Inspect: Python package.
License and cost. Both MIT licensed and free to run; you pay only for model calls. Promptfoo also sells enterprise plans.

Moving off OpenAI Evals before Nov 30

OpenAI announced the Evals deprecation on June 3, 2026. Per its deprecations page, existing evals become read-only on Oct 31, 2026, and the Evals dashboard and API are scheduled to shut down on Nov 30, 2026. OpenAI's cookbook guide, "Moving from OpenAI Evals to Promptfoo", describes the path:

Export or copy each eval's definition: the prompts, model, test data and grading criteria.
Recreate it in a promptfooconfig.yaml, mapping each grader to a Promptfoo assertion.
Run promptfoo validate config -c promptfooconfig.yaml, then promptfoo eval -c promptfooconfig.yaml --no-cache and promptfoo view.
Compare the new run against your last known-good results before trusting it. OpenAI notes that similarity scores won't match exactly across systems and that recreated LLM-as-judge graders need re-validation.

Nothing forces you onto Promptfoo specifically. If your evals were mostly agentic or you prefer Python, the same definitions port to Inspect with a little more rewriting. The important part is getting the test data and grading logic out before the shutdown date.

Common mistakes that make evals useless

Letting the model grade itself. Use a different, ideally stronger, model as the grader for rubric checks, and spot-check its verdicts by hand.
Rubrics that are too vague. "Is this answer helpful?" produces noise. "Does the answer mention the refund window and avoid promising a date?" produces signal.
No variance check. Model outputs vary between runs. Run important cases more than once and look at pass rate, not a single pass or fail.
Testing only the happy path. Add malformed input, prompt-injection attempts, empty fields and very long inputs. Our guide to stopping AI coding agents from leaking secrets covers the kind of failures worth turning into permanent tests.
Ignoring cost and latency. A model that wins on quality but doubles your bill is not a free upgrade. Both frameworks can record usage; Promptfoo can assert on it directly.
Leaking production data. Scrub personal data and secrets from test cases before you commit them to a repo.

Verdict

Every team shipping on top of an LLM should have a private eval suite in the repo, run on every prompt or model change. For most product teams, start with Promptfoo: the YAML format is quick to write, the side-by-side model comparison is exactly what you need when a new model lands, and it drops into CI with almost no glue. Reach for Inspect when the thing you're testing is an agent that runs tools, edits files or needs a sandbox, or when you want to reproduce a public benchmark yourself. Plenty of teams will end up with both. If you still have tests in OpenAI Evals, export them this month; after Nov 30 they're scheduled to be gone.

For picking which models to put into the suite in the first place, our comparison of open-weight coding models and the AI gateways guide are good starting points, and the OpenAI company page tracks related platform changes.

Sources

Frequently asked questions

What is an LLM eval?: An LLM eval is a repeatable set of test inputs with checks on the outputs, such as exact matches, JSON validation, code assertions or a rubric graded by another model. You run it whenever you change a prompt, model or app version to catch regressions before users do.
Is Promptfoo or Inspect better for LLM evaluation?: Promptfoo is usually faster to adopt for prompt and model comparisons, app regression tests and CI, because suites are written in YAML. Inspect is stronger for agent and tool-use evals that need sandboxes, and for teams that prefer writing evals in Python. Both are open source under the MIT license.
When does OpenAI Evals shut down?: According to OpenAI's deprecations page, existing evals became read-only on Oct 31, 2026, and the Evals dashboard and API are scheduled to shut down on Nov 30, 2026. OpenAI's migration guide recommends Promptfoo.
How many test cases does an LLM eval need?: There is no fixed number, but 30 to 100 well-chosen real inputs with sharp checks is a practical starting point. Add a new case every time a production bug or surprise shows up.
Are Promptfoo and Inspect free?: Both are free, open-source tools, so you pay only for the model calls your evals make. Promptfoo also sells Enterprise and on-premise plans with team dashboards, SSO and support.