The aot-evals method

This is the reference for what aot-evals does and why: the principles, the measurement frame, the files it keeps in your repository, the commands C0–C19 and their exit gates, the measurements (M), experiments (E), report panels (P) and refusal gates (G) that the skills and the CLI name. The skills read the section they need; you can too.

Who it is for. The method is designed for production, user-facing agents on any stack: their work is specific to the product, so no public benchmark measures it; their volume makes a point of correctness, cost or latency worth measuring; and their output reaches users without an expert checking it first. Any agent that can be run from a command can use it. The agent is invoked as a command, so framework, language and model provider do not matter.

What it is not. Not trace collection (it reads exports of what your observability already records), not a model gateway, not a hosted dashboard and not a leaderboard. There is no service and no account. Everything it writes is text, diffable and safe to commit; raw traces and PII never enter git. Every reported number carries n, k and an interval, or states why it cannot have one. When a number cannot be defended, aot-evals refuses to print it and names the gate.

1. Principles

1. Cases come from the agent’s real traffic and real failures, not from generic benchmarks. Public benchmarks are a source of grading designs, not of cases. Assess matches the agent to a local catalog of them (aot-evals benchmarks, references/benchmarks.md) without searching; Build may search for newer ones. What the catalog does contribute is the situations (references/situations.yaml): the conditions a request can put an agent in (unanswerable, ambiguous, forbidden, confidential, multi-turn, tool error, …) that benchmarks were built to test and that an agent’s own task types never name. The discovery report lays the agent’s task types (rows, from its traffic) against the situations (columns) as a grid of cases that exist against targets; in Build each situation is a taxonomy entry, so coverage (C14) reports the grid’s empty cells.
2. Understand the agent before measuring it: inventory, infrastructure, volume and usage first.
3. Grade outcomes, with code first. Check what changed in the world. Use a judge only where code cannot decide, and only after calibrating it.
4. Cheap sources first, raw transcripts second.
5. Keep sensitive data out of version control. Raw traces stay local and ignored; only reviewed, scrubbed cases are committed.
6. Record every claim as confirmed or inferred, with its source. When config and runtime disagree, trust the traces.
7. Three planes, reported separately, never combined into one score (§2.1): correctness, operational cost, and trust in the instruments.
8. Two case sets and two regimes (§2.2, §2.3). A set built to find errors and a set built to estimate quality are different sets. A test that can be re-run and a signal that can only be observed in production are different instruments. Every reported number states which set and regime it came from.
9. Measure the suite itself (§2.4): coverage against a list you did not derive from your own cases, and redundancy by whether cases distinguish versions of the agent.

The step-by-step process behind principles 1–6 is in references/process/.

2. Measurement frame

2.1 Three planes

Plane B is deterministic. It is recorded on every trial, pass or fail, and needs no ground truth, judge or labels, so it is the first thing to instrument. Operational failures (timeouts, rate limits, crashes) are plane-B data: never scored as correctness failures and never dropped.

• An operational regression is a finding even when correctness is unchanged. Cost per successful task doubling at the same pass rate is a failed comparison (verdict worse). A common cause is a prompt-prefix edit that invalidates the prompt cache.
• A correctness gain that comes with a cost or latency regression beyond budget is a trade, reported with both numbers, not as an improvement (G11).

2.2 Two regimes: offline and online

Neither substitutes for the other. Offline cases may come from your observability stack, but are always imported as a local export (C0); the suite runs with that stack switched off. aot-evals automates the offline regime today; the online regime (C16, C17) is guidance only.

2.3 Two offline case sets

• The error-finding set is enriched with hard, ambiguous and rare-but-costly cases. It finds bugs and verifies fixes. Its pass rate says nothing about performance on real traffic.
• The quality-estimate set is sampled in proportion to each category’s observed share of traffic. Its weighted pass rate is the only number the report may call “quality”.

The two are disjoint, sized by different arithmetic (C5), and every reported number is labelled with its set.

2.4 Coverage and redundancy

Coverage. Coverage measured against categories derived from the suite’s own cases is always 100%, so it is measured against a list written for some other purpose: the product spec, the API or tool surface, a help-centre topic list, a job-task taxonomy, a regulator’s list of duties. The report counts cases per entry on that list, crossed with failure modes, and lists the empty cells first. A taxonomy entry may declare failure modes not_applicable, with a reason: those cells leave the empty count but stay visible.

Redundancy. A case that every version of the agent passes, or every version fails, cannot tell you which version is better. Over two or more versions, each case is always-pass, always-fail or discriminating; the report gives the discriminating fraction. Always-pass cases still work as regression guards; they just contribute nothing to a comparison. The hardest cases change after a prompt rewrite, so the discriminating set is re-derived after every prompt change.

3. Repository contract

Each agent’s suite lives in evals/agents/<agent>/, beside its discovery report. A repository with one agent may keep the suite in evals/ itself. Pass --agent NAME to the CLI when there are several. Suites made before that layout live in evals/<agent>/; they are read as they are, and aot-evals migrate moves them.

evals/agents/<agent>/
  discovery.md               # Assess: the discovery report
  agent.yaml                 # identity, triggers, tools, model per call site, stable window,
                             #   and how to invoke it (references/agent-contract.md)
  taxonomy.yaml              # the coverage list (§2.4): entries, the source document, the date read
  categories.yaml            # trace categories: definition, worth, simple flag, cost matrix, observed share
  failure_modes.yaml         # per failure mode: >=1 check, >=1 category, a cost-matrix cell
  budgets.yaml               # plane B: cost / latency / token / error-rate budgets per category
  allocation.yaml            # C5's plan: sizes, and each unfilled cell scheduled or out of scope
  cases/<category>/<case-id>.json   # frozen inputs, start state, trigger, expected end state, set, provenance,
                                    #   requirements (what a correct run must do, each with its checks)
  labels/standard.md         # the labelling standard: rules a person applies, with the categories each applies to
  labels/runs.jsonl          # C19: whole-run human labels, one line per (run, labeller); no agent output
  labels/rulings.jsonl       # §4.2: the verdict on a run its labellers disagreed on
  labels/proposals.yaml      # §4.2: rules proposed from rulings, accepted or declined by the owner
  labels/drafts/             # §4.2: cases a grader drafted, waiting for an engineer
  labels/triage.yaml         # C19: a cause for each disagreement between the suite and a person
  labels/floors.yaml         # optional: C19's floors (min_catch, min_clean, min_runs, min_each)
  checks/
    code/<check-id>.py              # binary state diff: check(start, end, case) -> (bool, detail)
    judge/<check-id>.md             # one failure mode, binary, typed; backend-agnostic (§5)
    judge/<check-id>.labels.jsonl   # human labels, each with the exact judge input labelled
    judge/<check-id>.judgements.jsonl   # cached judge calls, so re-measuring costs nothing
    judge/<check-id>.calibration.json   # C10's measurements
  env/                       # fakes, cassettes, seeds, frozen-clock config
  data/                      # ignored: exports, labelling pages; only export.manifest.json is committed
  runs/<run-id>/manifest.json       # the condition record
  runs/<run-id>/outcomes.jsonl      # one line per attempt and per check verdict
  report/report.json                # §8
  report/agreement.json             # C19: catch and clean pass rates on whole runs, dev and holdout, per category
  report/report-<date>.html         # §9
  report/checkpoints/               # §9: one page per owner checkpoint
  journal/                          # §9: the activity log; ignores itself
  .gitignore

Rules:

• Paths start at the repository root, the folder that holds evals/: invoke.cwd in agent.yaml and a command judge’s cwd.
• outcomes.jsonl is never aggregated at write time. It has two kinds of line: kind: attempt, one per (case, trial, attempt), with the outcome code and plane-B counters; and kind: check, one per (case, check, trial, attempt), with the verdict. Every attempt is written, including retries and operational failures. An attempt’s check lines are written before its attempt line, so a final attempt line on disk means the attempt is complete. pass^k, flakiness, discrimination and paired tests are computed from these lines.
• manifest.json is the condition record: the agent’s commit (computed with evals/ excluded, so committing a run or a label is never an agent change) and this suite’s commit, every model id, harness, fakes, k, total cost, and whether the run was isolated or saw the eval files (§4, C12). Counts, cost and observed models are recomputed from outcomes.jsonl when the run ends. A resumed run lists each resume under resumed (at, pairs_done_before). A run started with --model <id> records it as invoke.model; the agent receives it as model, a candidate without --variable declares model=<id>, and a resume keeps it. Two runs whose manifests differ on any field other than the declared independent variable are not compared (G7).
• Every case has a set: error_finding or quality_estimate. There is no default.
• Every case has requirements: what a correct run of it must do, each citing a rule of labels/standard.md and naming the checks that decide it, or why it is unchecked. Every rule that applies to the case’s category is cited or excused in rules_not_applicable (C6, C8).
• taxonomy.yaml names the document it came from, and is committed before the first case. A model may draft it (C2), but only from a source outside cases/.
• Unknown values are null, never 0.

4. Commands

Each command has an exit gate. A gate is a trust upgrade, not a lock: no command refuses to run because an earlier gate is unmet. The run manifest records the gates that were unmet when a run started (unmet_gates), and the report gates (§10) grey what each one covers rather than hide it. aot-evals status says which phase the work is in, how far each category can be trusted, what changed since the baseline, and the one next step; aot-evals status --gates shows every gate and the first unmet one.

C3–C18 repeat as a loop; §4.1 says what each kind of change re-opens. A prompt rewrite re-derives the hard-case set, and a shift in category mix re-opens C3 and C5.

How runs work.

• The agent is a command: the case goes in as JSON on stdin and one JSON result comes back on stdout, with optional setup, seed, observe and reset hooks (references/agent-contract.md). Checks grade what observe returns, never what the agent says it did.
• Outcome codes (ok, timeout, rate_limited, provider_error, no_output, unparseable, model_disabled, auth_error, budget_exceeded, cli_missing) are recorded per attempt. Only ok is scored; retryable codes are retried up to max_attempts.
• A run stops after --stop-after (default 5) cases in a row end in an operational failure; a stopped run is never finished. --pause-every N --pause-seconds S paces a rate-limited runtime.
• aot-evals run --resume RUN_ID carries on a stopped or interrupted run in the same directory. It skips every (case, trial) with a final attempt, including one that ended in an operational failure, and runs the rest, appending to the same outcomes.jsonl. A pair whose last attempt isn’t final is run again, with attempt numbers after the ones already used, so a crashed attempt’s lines never join a new one’s. A truncated last line is dropped first, and the setup and reset hooks run before anything else. Resume is refused when the run is finished, or when the agent commit, the suite’s commit (changes under runs/ aside), invoke.env, a judge’s sha, the case set, the code checks or the fakes differ from the manifest, or when the agent has uncommitted changes: one run never mixes two versions. The run keeps the purpose, k, case set and judges setting it started with. Once finished, it counts as a complete run (C12).
• The agent must not see the cases. invoke.isolate: true runs the agent and its hooks from a per-run copy of the repository without evals/ (this suite’s env/ is kept). Every run, isolated or not, records eval files changed during the run and agent output that names a case file; either keeps the run from being a baseline.

How comparisons work (C13). Each side’s verdict on a case is its majority over k trials. Gains and losses are majority flips, tested with McNemar’s exact test; a bootstrap CI covers the mean per-case delta. A change is better or worse only when it is significant and outside the baseline’s variance band; otherwise indistinguishable. The declared variable is a kind (model, prompt, config, tools, harness, …) that maps to the manifest fields allowed to differ. Runs of one build split across several runs (for example category by category, to pace a rate-limited runtime) count as one side: their case sets, checks and judges are merged per item.

How redundancy works (C15). Versions are runs grouped by the fields that define the build, so repeats of one build never look like two versions. The prune proposal never prunes quality-estimate cases, sends always-fail cases to review, keeps regression guards for coverage cells and categories, and re-runs the pairwise tests to show the detectable effect is unchanged.

4.1 Phases, trust and iteration

The work has three phases, each ending in something handed to the next group of people:

The lightest Build (aot-evals build, then aot-evals build --run) is the one that gets the loop running: an agent that runs from a command (or recorded runs replayed, C1), 10 to 20 cases from real runs in the largest categories (C6), a side-effect check and one code check per category (C8), and a baseline (C12). Every other command is an upgrade that raises trust in a number: provenance (C0), the coverage list (C2), shares and the cost matrix (C3), budgets (C4), allocation (C5), screens (C7), judge labels, calibration and reliability (C9 to C11), whole-run agreement (C19), coverage (C14) and redundancy (C15). The report shows which upgrades are missing and greys what they cover. aot-evals build drafts what it can from the discovery report: categories from its task types, the coverage list from its situations (to be committed before the first case), and one failure mode on every case checked by a generic code check, end-state-matches (the world ends as the case expects), so a first pass needs no check code. Each draft is marked for the owner to confirm.

Trust per category. A category is untested until a finished baseline covers it, provisional once it has numbers, and validated when the suite agrees with a person on its runs: C19’s floors met suite-wide, and the category’s own dev catch rate and clean pass rate at or above the floors (85% by default). report/agreement.json carries the per-category rates. Provisional numbers print, labelled provisional (G14).

The baseline is, for each current case, the newest finished baseline run that covers it: a re-run of one category replaces that category’s baseline and leaves the others’. The agent has changed only when its files outside evals/ differ from the baseline’s, not merely its commit id.

What a change invalidates. status compares the suite and the agent with the baseline and names each change with its short path back:

An uncommitted agent is named too: a run of it can’t be compared (G7). When the agent’s current version has already run as a candidate and been compared, the step is to decide: have a person label only the runs whose verdict moved, then run it as the new baseline if it ships.

4.2 The bench of graders and the grader application

aot-evals serve starts the grader application: a local web page, served from the suite, for the Judge phase. It binds to 127.0.0.1 and needs the session token in the link it prints; &labeller=NAME on that link gives each grader their own. Everything it does writes the suite’s own files, which are reviewed and committed like any other change:

Rules:

• Several people label the same run, blind. Each labeller’s latest label is kept. Where all of a run’s labellers agree, that is its verdict; where they differ, it has no verdict until someone rules, and C19 stays open while any such run is unruled. Agreement between each pair of labellers (share agreed and Cohen’s κ) is reported beside the suite’s agreement with people.
• Queues are per labeller. A new labeller starts with up to five runs whose answer is settled (a ruling, or two people agreeing) and sees the settled answer after each label. After that, one run in four is one someone else labelled, so agreement between people can be measured, and the rest are runs nobody has labelled, one per case first. After a change, the queue can be limited to the cases whose verdict moved (aot-evals agreement label queue --changed).
• Re-grading is offline. The application re-grades the labelled runs from their recorded raw records with the suite as it is now: no agent runs and no judge calls.
• An accepted wording is screened. Every accepted wording must pass every check (C7). If a check still rejects it, the disagreement is triaged as a check bug for an engineer.
• The engineering inbox lists what only engineering can do: triaged disagreements whose cause is a fix to a case or check, drafted cases, and changes that need the agent run.

4.3 The components

An optimization loop around an agent has the same parts whatever the agent does. aot-evals components lists them, each with the phase that builds it and its state read from the suite (done, partly, not yet, not automated yet, or doesn’t apply). The discovery report may record a component’s state before a suite exists (components: in its front matter); the suite’s own reading wins once it says more.

5. Judges

A judge is a check for one failure mode that asks one binary, typed question: a noul (a true/false claim, with detects saying whether yes means the failure is present or the requirement is met), a choice over declared options, or a score on a rubric level. The judge layer is backend-agnostic (references/judge-backends.md):

Rules:

1. Every backend is calibrated in C10. A typed answer can still be wrong.
2. A backend’s confidence is not trusted until measured on your labels. A probabilistic backend’s threshold is chosen by maximising TPR + TNR on the dev labels; the reported rates come from 5-fold out-of-fold predictions, and a hash-assigned 20% holdout is reported separately. The floors are min_tpr / min_tnr (default 0.75).
3. A judge counts toward a headline only when it is calibrated on its current version and backend. Stored scores are re-thresholded with the calibrated threshold, so calibrating after a run changes the report without re-running anything.
4. Labels carry the judge’s input. Each label includes the exact input that was labelled (after a secret scan), so calibration can be re-run on any backend without the raw run data.
5. Judgements are cached per backend (the question, the inputs and that backend’s own config), so adding or changing a backend does not re-pay for the others.
6. No silent fallback. A refusal or an error is an instrument error, never a verdict from another model. A rate-limit error stops that backend’s calls at once.
7. Different family for pairwise grading (G4).

E5 compares backends on the same labels, by TPR/TNR and cost per judgement, and is how each judge’s backend is chosen.

6. Measurements

N = cases, k = trials per case, L = human labels. Plane refers to §2.1. Measurements marked † are guidance for the online regime and are not yet computed by the CLI.

7. Experiments

No comparison runs before E1. Experiments marked † are guidance only.

8. report.json

Written by C18 (aot-evals report) to report/report.json and read by the HTML report. The schema is versioned (aot-evals/report@1); keys may be added within a version, never removed or renamed.

{
  "schema": "aot-evals/report@1",
  "generated": "2026-01-01T00:00:00Z",
  "agent":   { "id": "...", "stable_window_since": "...", "model_ids": {} },
  "conditions": { "agent_commit": "...", "evals_commit": "...", "harness": "...", "k": 5,
                  "judge_backends": [ { "check": "...", "backend": "...", "model": "...",
                                        "family": "...", "version": "..." } ],
                  "simulator": null, "cost_usd": 0.0,
                  "data_exports": [ { "source": "...", "window": "...", "rows": 0 } ] },
  "regime":  "offline",
  "taxonomy": { "name": "...", "source": "...|MISSING", "read_on": "...", "entries": [] },
  "sets":    { "error_finding": { "n": 0, "mde_at_80_power": 0.0 },
               "quality_estimate": { "n": 0, "weights_source": "...", "estimate": {} } },
  "categories": [ { "id": "...", "worth": "high", "simple": false,
                    "observed_share": 0.0, "n_cases": 0,
                    "pass_rate": { "point": 0.0, "ci": [0, 0], "n": 0, "n_cases": 0, "k": 5,
                                   "set": "...", "ci_method": "..." },
                    "pass_pow_k": 0.0, "pass_pow_k_reason": null, "flaky": 0, "variance_band": [0, 0],
                    "operational": { "cost_per_success": 0.0, "cost_per_run": 0.0,
                                     "latency_p50": 0, "latency_p90": 0,
                                     "tokens_input": 0, "tokens_output": 0, "cache_hit_rate": 0.0,
                                     "tool_calls": 0.0, "outcomes": { "ok": 0, "timeout": 0 },
                                     "retries": 0, "killed": 0,
                                     "budget": {}, "budget_status": "within|over" },
                    "excluded": null } ],
  "checks":  [ { "id": "...", "type": "code|judge", "failure_mode": "...",
                 "trigger_rate": 0.0, "sole_failure_rate": 0.0,
                 "judge": { "backend": "...", "model_family": "...", "agent_model_family": "...",
                            "tpr": 0.0, "tnr": 0.0, "labels": 0, "kappa": 0.0, "threshold": null,
                            "self_consistency": 0.0, "cross_family": 0.0,
                            "confidence_calibration": { "ece": null, "brier": null },
                            "cost_per_judgement": 0.0,
                            "status": "calibrated|uncalibrated", "status_reasons": [] } } ],
  "coverage": { "list": "external|MISSING", "cells": [ { "taxonomy_entry": "...", "failure_mode": "...", "n": 0 } ],
                "empty_cells": 0, "authoring_queue": [] },
  "redundancy": { "candidates": [], "always_pass": 0, "always_fail": 0, "discriminating": 0,
                  "discriminating_fraction": 0.0, "per_case": [], "proposal": {} },
  "comparison": { "baseline": "...", "candidate": "...", "paired": true, "refused": null,
                  "gains": [], "losses": [], "net": 0.0,
                  "test": { "name": "mcnemar_exact", "p": 0.0, "ci": [0, 0] },
                  "mde_at_80_power": 0.0, "per_set": {},
                  "operational_delta": { "cost_per_success": 0.0, "latency_p90": 0, "error_rate": 0.0 },
                  "verdict": "better|worse|indistinguishable|trade", "verdict_reasons": [] },
  "validity": { "stages": { "task": null, "simulator": null, "judge": null }, "compounded": null },
  "gates":   [ { "id": "G5", "status": "pass|fail|not_applicable", "detail": "...", "blocks": [] } ],
  "next_actions": [],
  "cases":   [ { "id": "...", "set": "...", "category": "...", "provenance": "...",
                 "screen": { "human_read": true, "reference_passes": true, "donothing_fails": true },
                 "trials": [ { "run": "...", "trial": 0, "attempt": 0, "outcome": "ok", "passed": false,
                               "checks": [ { "id": "...", "type": "code", "passed": false, "detail": "...",
                                             "counted": true } ],
                               "operational": { "cost_usd": 0.0, "latency_ms": 0,
                                                "tokens_input": 0, "tokens_output": 0,
                                                "tool_calls": 0, "retries": 0 },
                               "state_diff": "...", "trace_ref": "..." } ] } ]
}

Rules: unknown values are null, never 0. A missing coverage list is the string "MISSING", not an omitted key. Nothing is pre-rounded. There is one category row per (category, set), labelled by pass_rate.set. Per-trial outcomes and plane-B counters are kept so the renderer can recompute anything. trace_ref is an id or a local path, never the trace.

9. The HTML report, checkpoints and the activity log

aot-evals ships the schema (§8), the panel list and the rules below, not a dashboard. On request, Claude writes one self-contained local HTML file: data inlined as JSON, no network requests, no CDN, no build step. It opens from the filesystem and can be attached to a pull request or an email. Each file is written for a question: a report on “why did the candidate regress” leads with the comparison and the losing cases; one on “is this suite any good” leads with gates, coverage and discrimination. aot-evals report --html writes the branded shell, and Claude writes the panels into it (the report skill).

Rules:

1. Every number shows n, k and an interval, or says why no interval can be computed.
2. “Quality” labels only a quality-estimate-set number with published weights. Error-finding numbers are labelled “error-finding set — not a quality estimate”.
3. Correctness and operational numbers are never combined, and an operational regression is shown even when correctness is unchanged.
4. Anything below its gate is shown greyed and not used as a headline; it is never hidden.
5. Reliability and validity never share a column group.
6. The headline is the per-category table, not a single aggregate pass rate.
7. Every claim in prose links to the panel it comes from.

Checkpoints and the activity log. The work is paced by pages, so the owner can follow it in a browser and steer at every gate.

• Activity log. journal/activity.jsonl is append-only: every CLI command writes an entry (command, gate, exit code, duration, refusal message), and the skills add their steps, findings, decisions and questions with aot-evals log. Each append re-renders activity.html, which reloads itself and says when Claude is waiting. The journal ignores itself in git: it may quote unreviewed trace text, and the committed record of a decision is the file it changed.
• Checkpoints. aot-evals checkpoint writes a page for one gate: the C0–C19 roadmap in plain words, Claude’s summary, the log since the last checkpoint, every design file rendered as tables, the owner’s questions, what still blocks the gate, and the next step. It makes no model calls.
• Pacing. Every skill works until a gate clears or the work needs the owner, publishes a checkpoint, and stops until the person replies.
• Telemetry. start, discovery-page, checkpoint, report and demo each send one pseudonymous milestone event when they succeed (install id, event, gate id, version, OS, time; nothing about the agent or its data). It is off whenever Claude Code’s own telemetry is off, on cloud providers and in CI; aot-evals telemetry shows, shows the payload of, and switches it. Details: SECURITY.md.

Visual style follows brand/README.md.

10. Gates

From docs/method.md in aot-evals 1.0.1.