Mahal Systems AOT Evals: the method
How to design, build, calibrate and run evals for a production agent, so the tests predict how it does in production and show what to fix next. Open source under Apache 2.0: read it here, download it, or run it on your own agent with our Claude Code plugin.
Download the method
The method, the process, the practice findings, the benchmark catalog and the templates as plain files, to read or hand to any coding agent. Version 1.0.1.
Run it on your agent
The Claude Code plugin walks your agent through the method in your own repository and shows how far each kind of task can be trusted.
claude plugin marketplace add Mahalanobis-Systems/aot-evals claude plugin install aot-evals@aot-evals
Then, in your agent's repository: /aot-evals:start. More on GitHub
Who it is for. The method is designed for production, user-facing agents on any stack: their work is specific to the product, so no public benchmark measures it; their volume makes a point of correctness, cost or latency worth measuring; and their output reaches users without an expert checking it first. Any agent that can be run from a command can use it. The agent is invoked as a command, so framework, language and model provider do not matter.
What it is not. Not trace collection (it reads exports of what your observability already
records), not a model gateway, not a hosted dashboard and not a leaderboard. There is no service
and no account. Everything it writes is text, diffable and safe to commit; raw traces and PII
never enter git. Every reported number carries n, k and an interval, or states why it cannot
have one. When a number cannot be defended, aot-evals refuses to print it and names the gate.
Principles
- Cases come from the agent’s real traffic and real failures, not from generic benchmarks. Public benchmarks are a source of grading designs, not of cases. Assess matches the agent to a local catalog of them (
aot-evals benchmarks, references/benchmarks.md) without searching; Build may search for newer ones. What the catalog does contribute is the situations (references/situations.yaml): the conditions a request can put an agent in (unanswerable, ambiguous, forbidden, confidential, multi-turn, tool error, …) that benchmarks were built to test and that an agent’s own task types never name. The discovery report lays the agent’s task types (rows, from its traffic) against the situations (columns) as a grid of cases that exist against targets; in Build each situation is a taxonomy entry, so coverage (C14) reports the grid’s empty cells. - Understand the agent before measuring it: inventory, infrastructure, volume and usage first.
- Grade outcomes, with code first. Check what changed in the world. Use a judge only where code cannot decide, and only after calibrating it.
- Cheap sources first, raw transcripts second.
- Keep sensitive data out of version control. Raw traces stay local and ignored; only reviewed, scrubbed cases are committed.
- Record every claim as confirmed or inferred, with its source. When config and runtime disagree, trust the traces.
- Three planes, reported separately, never combined into one score (§2.1): correctness, operational cost, and trust in the instruments.
- Two case sets and two regimes (§2.2, §2.3). A set built to find errors and a set built to estimate quality are different sets. A test that can be re-run and a signal that can only be observed in production are different instruments. Every reported number states which set and regime it came from.
- Measure the suite itself (§2.4): coverage against a list you did not derive from your own cases, and redundancy by whether cases distinguish versions of the agent.
Three planes, never one score
| Plane | Question | Instrument | Reported as |
|---|---|---|---|
| A. Correctness | Did the agent do the right thing? | code state-diff checks; a calibrated binary judge where code cannot decide | pass rate per category with CI; pass^k |
| B. Operational | What did it cost? | counters, not judgements: $ per run and per successful task, p50/p90 latency, input/output/cached tokens, cache-hit rate, tool calls, retries, errors and timeouts by outcome code, runs killed at a limit | values against per-category budgets, across runs |
| C. Instrument trust | Can plane A be believed? | judge TPR/TNR against human labels; judge self-consistency and cross-family agreement; case-validity screens | its own report panel, with reliability and validity in separate column groups |
The process
From agents running in production with no evals to a calibrated suite for each of them, in three phases. The process, step by step.
Commands and their exit gates
Each command has an exit gate. A gate is a trust upgrade, not a lock: a command still runs with an earlier gate unmet, the run records it, and the report greys what that gate covers. Items marked guidance aren't automated yet. How runs and comparisons work.
| # | Command | Does | Exit gate |
|---|---|---|---|
| C0 | import | Import your data as an export (trace dump, dataset download, CSV/JSONL from your observability stack) into data/, with an export.manifest.json entry: source, query, window, row count, date. | Every downstream number traces to an export manifest entry; no command calls a live vendor API. |
| C1 | inventory | Write agent.yaml: surface, triggers, instructions, tools (credential names only), harness, the model actually used per call site, version history, the stable window, and the invocation command. | Stable window stated with evidence; model per call site observable; invoke.cwd exists; the agent runs once end to end from its command. |
| C2 | taxonomy | Build the coverage list from a source outside the suite: the product spec, API/tool surface, help-centre topics or job description. The owner edits it; the source document is recorded. | taxonomy.yaml names a source outside cases/ and a date, and its commit precedes the first case. |
| C3 | categorize | Sort traces into categories, one per trace. Give each a worth and a per-failure-kind cost matrix. Count observed share. | Every category has a definition, an example, worth, a simple flag, a cost row and a measured share. “Other” is under 10%. |
| C4 | decompose | Within each category: feature → scenario → failure mode. Mark each failure mode code-checkable, judge-needed or monitor-only, and set plane-B budgets. | Every failure mode names its category, check type and cost cell; budgets.yaml covers every category. |
| C5 | allocate | Size both case sets and print the arithmetic. Error-finding: ≥10 per category, 3 for simple ones. Quality-estimate: proportional to observed share, sized by the interval the owner needs (M14). | Both budgets printed with target intervals and the minimum detectable effect (M7); every cell filled, scheduled, or declared out of scope with a reason. |
| C6 | author | Write the labelling standard, then cases: frozen inputs, start state, trigger, scripted follow-ups, expected end state, and requirements walked rule by rule from the standard. Prefer self-verifying pairs where state allows. Scrub before writing. | The standard exists; every case has a set, a category, failure modes, provenance, requirements covering every rule that applies to it, and no secrets. |
| C7 | screen | Per case: a human read; the checks must pass on start → expected end (reference) and fail on start → start (do nothing); a third pass, do-nothing plus a success claim, catches checks that trust the agent’s report (G10). With a reference answer: other correct wordings must pass, the reference answer with nothing changed must fail, and an answer with every number wrong is reported if it passes. Drop cases that fail, and record the drop. | Screens recorded per case; drop rate reported. |
| C8 | checks | Implement plane-A checks: code state diffs first, graded against the goal rather than one path; binary judges only where code cannot decide; side-effect checks on every agent; a grounding check for every fact a prose answer states. A check never uses the agent’s own report of success. | Every failure mode has at least one binary check; side-effect checks present; every requirement names a check that runs on its case, or why it is unchecked. |
| C9 | label | Collect human labels per judge, stratified across categories: one item per case first, then other trials where the answers differ. Target 100–200; below 60 the interval is too wide to act on. | Label count and stratification printed per judge. |
| C10 | calibrate | Measure each judge against its labels: TPR and TNR separately (never raw agreement), κ, confusion matrix, cost per judgement. Choose a probabilistic backend’s threshold on the labels. Version the criteria. | Every judge has TPR/TNR on ≥60 labels at or above its floors, a named backend and model family, and a version. Uncalibrated judges may run but never count toward headline numbers. |
| C11 | reliability | Run m fresh sessions per item on two backends from different model families; report agreement at three levels (all / majority / none) next to C10’s TPR/TNR. | Self-consistency and cross-family agreement reported next to TPR/TNR, never in place of it. |
| C12 | baseline | Run the suite k = 3–5 times on an unchanged, committed build. Write per-trial outcomes and plane-B counters. Compute the variance band. | Run complete and finished, manifest complete, no eval-file exposure, variance band per category, plane B recorded against budgets.yaml. |
| C19 | agreement | Runs after C12 (numbered last so earlier numbers stay stable). A person labels whole baseline runs pass or fail with a note, seeing the request, the answer and what changed, never the checks. Every labelled run is re-graded from its raw record with the suite as it is now, so labels outlive suite versions. Report the catch rate and clean pass rate (M29), dev and holdout; give every dev disagreement a cause. | ≥60 labelled runs with ≥15 of each kind; a measurement current for the suite and the labels; dev catch and clean pass rates at or above their floors (85% by default); every dev disagreement triaged. Until then G14 fails and comparisons are provisional. |
| C13 | compare | Run a candidate on the same cases with one declared independent variable. Report gains, losses and a paired test, never a net delta alone, with plane-B deltas alongside. | Paired test present; gains and losses listed; manifests differ only on the declared variable. |
| C14 | coverage | Count cases per (taxonomy entry × failure mode), zeros included, with the source document named. | Coverage reported against an external list, or refused (G5). |
| C15 | redundancy | Over ≥2 agent versions, classify each case; report the discriminating fraction and a per-case index; propose prunes and cells to extend. | Discriminating fraction published; below the threshold (default 0.3), a prune/extend proposal. |
| C16 | driftGuidance | Re-run the frozen set on a schedule and on every prompt, skill, tool, model or harness change, against the baseline’s band. | Guidance only; not yet automated. |
| C17 | onlineGuidance | Set up the online regime: shadow → canary → online judging; behavioural signals; fingerprints. | Guidance only; not yet automated. |
| C18 | report | Write report.json (§8), then the HTML report (§9). | Every gate’s status is in the report. |
Gates: what it refuses to report
When a number can't be defended, it isn't printed; the report names the gate instead.
| Gate | Condition | Refuses |
|---|---|---|
| G1 | No condition record (model ids, commits, harness, k) | the whole report |
| G2 | k = 1 | any claim of improvement, regression or drift |
| G3 | Judge uncalibrated (under 60 labels, below its floors), or reported as raw agreement | that judge’s contribution to any headline; the judge is greyed |
| G4 | Generative judge from the same model family as the agent, for pairwise or preference grading | the pairwise result. A System One judge satisfies this |
| G5 | Coverage list with no external source, derived from the suite’s own cases, or committed after them | the word “coverage”; prints MISSING and the reason |
| G6 | Headline pass rate from the error-finding set | the word “quality”; offers the weighted estimate instead |
| G7 | Comparison without a paired test, or with undeclared manifest differences | the comparison |
| G8 | Cases that have not passed the reference / do-nothing screen | their inclusion in any gated number |
| G9 | Simulated user present, results not reported per simulator | the aggregate score |
| G10 | A check that relies on the agent’s own report of success | the check |
| G11 | A “better” verdict while plane B regressed beyond budget | the verdict; it becomes “trade”, with both numbers |
| G12 | A number whose data does not trace to an export manifest entry or a committed case | that number |
| G13 | Retried attempts or operational failures missing from outcomes.jsonl, or scored as correctness | the run’s plane-A and plane-B numbers |
| G14 | The suite’s verdict on whole runs not measured against a person’s, stale, below its floors, or with untriaged dev disagreements (C19) | the word “validated”; every comparison is marked provisional |
Situations every suite should cover
The conditions a request can put an agent in, which public benchmarks were built to test and an agent's own task types never name. In a discovery report they are the columns of the test case grid.
| Situation | Means | Catches |
|---|---|---|
| Unanswerable right answer is "none" | The data holds no answer, or the tools can't do what was asked. | Making up a record, number or outcome that isn't there |
| Ambiguous must ask or explore | The request matches more than one thing, or is too vague to act on without a question or a look at the data. | Acting on a guess when it should have asked, or asking when a look at the data would do |
| Forbidden or confirm-first refuse, or ask before acting | The policy forbids the request, or it needs the person's confirmation before the agent acts. | Doing what it was told not to do, or acting without the confirmation its rules require |
| Confidential don't disclose | The request asks for data the asker may not see, such as another person's private details or a record outside their role. | Reading out private or restricted data |
| Multi-turn pair same task over several turns | The same request arrives over several turns, with details given late or changed, and is scored against its single-turn twin. | Losing the thread between turns, or acting on a detail the person later changed |
| Tool error a call fails mid-task | A tool call fails, times out or returns an error partway through. | Stopping partway, or reporting success after a failed call |
| Different asker same job, another role | The same job asked by a different role, such as a manager wanting an aggregate where a rep wants one record. | Answering the wrong question for who is asking |
| Injected instruction untrusted content gives orders | Content the agent reads, such as a web page, email or record, contains instructions aimed at the agent. | Following an instruction that came from data rather than from the person |
| Person does a step the user must act on their side | Part of the job can only be done by the person, on their own device or account, with the agent guiding. | Doing the person's step itself, or giving the wrong step |
Go further
- The full reference: the measurement frame, the repository contract, judges, measurements M1-M29, experiments,
report.jsonand the report panels. - Practice findings: what the 2025-2026 literature says about evaluating production agents, with sources.
- The benchmark catalog: 364 public agent benchmarks, as a source of grading designs, case templates and sandboxes.