Glossary

The terms our research posts use, at more length than a post has room for, with references. Each post defines a term in a sentence where it first uses it and links here for the full version.

Accuracy

The share of a prompt’s answers that are right. In the gift micro-benchmark, each prompt gives 100 answers (20 shopper messages, five samples each), and accuracy is how many named the right gift. We sample each message five times because the same prompt on the same model doesn’t always give the same answer; one sample per message would mix that noise into the score.

On other people’s benchmarks the equivalent is usually called pass rate or score. We report differences in points.

References: our gift micro-benchmark, in What survives a model swap? (data and method to be published).

Adapter

A small set of extra weights trained on top of a fixed base model, most often with LoRA (low-rank adaptation). Instead of changing all of the model’s weights, LoRA learns a low-rank update to some of them; the “rank” (for example rank 64) sets how many parameters it learns. Adapters are cheap to train and store, which is why most fine-tuning services use them.

An adapter only means anything on the base it was trained against. When that base is retired, the adapter goes with it; you can retrain one on the new base from the same data, but you have to re-test it from scratch.

References: Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, 2021; Harvey, “Post-training update: Harvey Tenet”.

Deprecation

A lab’s notice that a model will be retired. It usually names a successor and two dates: when the model stops being offered to new users, and when calls to it stop working. Anthropic publishes a migration guide per model; OpenAI publishes a table of retired models and their recommended replacements.

The guides cover API changes (renamed parameters, new defaults) in detail. They say to re-test on your own evals but don’t say what a passing result looks like.

References: OpenAI, Deprecations; Anthropic, Model deprecations and migration guide.

Effort

A setting on recent Claude models (low, medium, high, xhigh, max) for how much the model reasons before it answers. Higher effort costs more tokens and time.

More effort isn’t always better. On FrontierCode, Sonnet 5.5 scored 52.1 at xhigh and 46.2 at max. The best level differs by model and by task, so Anthropic’s migration guide says to re-run the effort sweep when you change models. Effort can be changed partway through a conversation without losing the prompt cache, but only at a new user turn: inside an autonomous tool loop it stays fixed.

References: Anthropic, effort; Claude Haiku 5.5 system card, Table 8.1.A.

Fine-tune

A version of a base model trained further on your own examples, either by updating all its weights or, more often, by training an adapter. A fine-tune is tied to its base: it can’t be moved to a new model, only retrained on one. What does carry over is the data you trained it on and the evals you checked it with.

References: OpenAI, Deprecations, fine-tuned model shutdowns.

Graded environment

Three things together: a set of tasks, a rubric that says what a good answer to each looks like, and a grader (code, a model, or a person) that applies the rubric. It’s the one asset that answers “is this model better for our work?”, and it doesn’t depend on any particular model, so it survives every swap.

Its size decides what it can see. A 3-point difference between two models needs about 1,300 paired cases to tell apart from noise, and a 2-point difference about 2,900; a typical 20- to 500-case eval can only resolve differences of roughly 5 to 23 points.

References: Harvey LAB benchmark; our count of model releases and paired-eval sample sizing, in What survives a model swap? (method to be published).

Harness

The code around the model call: the loop that sends requests, retries failures, repairs malformed tool calls or JSON, caps steps and cost, and decides where outputs go. Also called the scaffold.

Changing only the harness can move results as much as a new model. Holding the weights fixed, harness changes took DeepSeek-V4-Pro from 63.4% to 80.1% on LAB. Deterministic harness code mostly carries to a new model; the prompts inside it and its fit often don’t.

References: Meta-Harness and its transfer notes; Legora, BAR.

Harness fit

How well one model works with one harness’s conventions: tool-call formats, memory layout, how much it searches, whether it calls tools in parallel. Labs post-train each model on their own harness, so a model can underperform in a harness built around another.

Fit has to be redone for each model. RELAI optimized harnesses for three open-weight models on the same 100 tasks and got three different harnesses, because each model failed differently.

References: Nicolas Bustamante, “Model-Harness-Fit”; RELAI, “Harness optimization brought open-weight agents to frontier performance at 3% of the inference cost”.

Model swap

Replacing the model an agent runs on with a different one, by choice or because the old one is retired, while keeping everything else the same. A swap within one lab (Haiku 4.5 to 5.5) isn’t reliably gentler than a swap across labs: across six micro-benchmarks, the same-lab advantage in how well a configuration ranking survived was zero within noise on five and small on one.

References: What survives a model swap?

Optimality gap

How far a configuration ends up behind the best configuration for a new model after a model swap: the new model’s best score minus the score of the configuration you carried over, in points. “Best” is measured by the best performance across the set of configurations you tried. It’s what you leave on the table by not re-tuning, and you close it by re-running your candidates on the new model and picking again. Across our six micro-benchmarks the gap for the configurations a team would have shipped ran from 0 (extraction, on GPT) to 12 points (tool routing, GPT-5.4 nano moved to Haiku 5.5). Earlier versions of the model-swap post called this the re-tuning gap.

References: our six micro-benchmarks, in Your model is being retired (data and method to be published).

Points

Percentage points of a quality measure: accuracy in our own tests, pass rate or score on other people’s benchmarks. Going from 80% to 85% is a gain of 5 points. This is different from a relative change: 80% to 85% is about 6% better in relative terms. We always say which one we mean, even when a source doesn’t.

Prompt

In our tests, a system prompt: the standing instructions the model gets before the user’s message. In the gift micro-benchmark each of the 20 prompts states a different subset of seven house rules in a different style, so we can see which differences come from what a prompt says and which from how it says it.

References: our gift micro-benchmark, in What survives a model swap? (data and method to be published).

Rank correlation

How well one ordering predicts another, from −1 to 1. We use Spearman’s rank correlation: rank the 20 configurations by accuracy on the old model, rank them again on the new one, and correlate the ranks. 1 means the same order on both; 0 means the old order tells you nothing about the new one.

It matters because prompt tuning is a ranking exercise: you keep the prompt that ranked first. A low rank correlation across a swap means the prompt you picked probably isn’t the best one any more. For a sense of scale, the same model run twice on split halves of our data gives 0.81 to 0.99, so that’s the ceiling noise allows. We call it “ranking kept” in the post.

References: our same-lab vs cross-lab test, in What survives a model swap? (data and method to be published).

Regression

A case that got worse after a change. In the gift micro-benchmark, a prompt-and-message case regressed if the new model got at least two fewer of its five samples right than the old one did. A change can raise the average while causing regressions; the net number hides them. In our re-analysis of a published prompt-optimizer benchmark, a change worth 4.2 points net flipped 12.4% of items, 166 to right and 82 to wrong.

References: our gift micro-benchmark and benchmark re-analysis, in What survives a model swap? (method to be published).

Routing

Sending different stages of one agent (planner, retriever, extractor, summarizer, subagent) to different models, or to one model at different effort levels. Each stage is its own prompt and model pair and needs its own eval. Switching models inside one long context loses the prompt cache, because caches are per model; whether that costs more than it saves depends on the two models’ prices.

References: Anthropic, prompt caching.

Trace

The full record of an agent run: inputs, model calls, tool calls, outputs and the outcome. Traces don’t depend on the model that produced them, so they carry over a swap. They can become training data for the next base (DeepSeek-R1 distilled a 32B model on 800K teacher traces) and are the only thing you can rebuild agent memory from on a new model.

References: DeepSeek-AI, “DeepSeek-R1”, arXiv v1; Goyal and Ray, “Does Your Agent’s Memory Survive a Model Upgrade?”.

Upgrade lift

How much a prompt’s score changes when you swap the model and change nothing else, in points. It’s the number a migration guide implies you’ll get for free. It can be negative: on the configurations a team would have shipped, the lab’s successor lowered the score in 3 of 12 upgrades across our six micro-benchmarks. Read it next to the optimality gap, because a positive lift can still leave you well short of the new model’s best configuration.

References: our gift micro-benchmark, in What survives a model swap? (data and method to be published).