Your model is being retired. Are you ready to upgrade?

TL;DR

We built six small agents, held each one fixed, and swapped the model underneath: Claude Haiku 4.5 to Haiku 5.5, and GPT-5.4 nano to GPT-6 Luna. Here’s what we found:

• A model “upgrade” doesn’t guarantee better performance. The lab’s successor lowered the agent’s score in 3 of 12 upgrades and left it flat in 1. Eleven of the 12 regressed some cases, even where the average rose.
• The lab’s successor isn’t automatically your best option. Switching to the other lab’s new model matched or beat the successor in 10 of 12 upgrades. Staying with your lab buys little continuity: on five of the six agents, the successor preserved your ranking of configurations no better than the other lab’s model did, and on the sixth only slightly better.
• Context, skills and prompts have to be tuned again. What you built in code mostly carries. Prompts, schema descriptions and tool descriptions lost their ranking across the swap. Retrieval settings, error handling and rules enforced in code kept theirs. The limit: code that depends on the model reading its inputs carried no better than the prompt.

And here’s what to do about it:

• Put your biggest investments into the components that persist. Your evals, your traces, and your harness code outlive Haiku 4.5, 5.5 and whatever comes after. Prompt text, tool descriptions and effort settings are tied to one set of weights.
• Treat the swap as a test, not an edit. Run the successor and at least one other lab’s model through the same evals, and re-tune the context, skills and prompts on whichever one you pick. This isn’t going away; make it a routine.

The reality behind the labs’ deprecation pages

OpenAI lists a “recommended replacement base model” for each fine-tune [1], and Anthropic publishes a migration guide from Haiku 4.5 to Haiku 5.5 [2]. The guides are thorough about the API changes: renamed parameters, new defaults, an effort setting to re-sweep.

On quality they’re nearly silent. Anthropic’s guide tells you four separate times to test on “your own evals”, but never says what passing looks like [2]. That’s surprising from businesses whose success rests on the eval sets and feedback loops they use to train their own models.

Does the silence mean the labs engineered their successors to behave the same way on existing workloads? That kind of backwards compatibility would mean the way you tuned your agent for one model carries, roughly, to the next.

Our benchmarks say otherwise.

What we tested: six micro-benchmarks

We tested the labs’ claim against our agent micro-benchmark suite [4]. The suite consists of six small agents built to exercise different kinds of agent work: recommendation, tool routing, text-to-SQL, retrieval, document extraction and multi-turn error recovery. It spans multiple domains: a gift shop, an online store, a subscription business, a product handbook, a stack of invoices and a clinic’s booking system. Each has verifiable ground truth, and every answer is graded by code.

The benchmarks are designed to exercise a wide set of agent components, as detailed in our agent optimization library. Each one optimizes a different component across 20 configurations and holds everything else fixed. Between them they cover 6 of the library’s 10 sections: Tools, Prompts, Context, Orchestration, Retrieval and Verification. Models, Memory, Topologies and Safety are not covered.

Every benchmark is tested across two deprecation paths: Claude Haiku 4.5 to its successor, Claude Haiku 5.5, and GPT-5.4 nano to its successor, GPT-6 Luna. Requests are identical apart from the model, with reasoning off. Each micro-benchmark is set up to have 20 configurations, and each configuration is run against 20 cases five times. A configuration’s score is the percentage of its 100 answers that were right, and differences between scores are in percentage points.

We assess upgrade performance with two metrics. The first measures the out-of-the-box value of the new model. The second measures how much tuning the new model needs once it’s in place.

• Upgrade lift: the change in score when only the model changes.
• Optimality gap: how far the carried-over configuration lands behind the optimal configuration on the new model. The gap is what you leave on the table by not re-tuning, and you close it by re-running your candidates on the new model and picking again.

Both metrics are computed on the old model’s best configuration, the one a team would have shipped.

Question 1: can a model upgrade make my agent worse?

Yes. Our six benchmarks were tested across two labs, which gives 12 same-lab upgrade tests. Three of the 12 lowered the agent’s score, one left it flat, and 11 got test cases wrong that the old model had gotten right.

• Three of the 12 upgrades made the agent worse, and one left it flat. Two of the three also lowered the average over all 20 configurations: Haiku 5.5 scored below Haiku 4.5 on tool routing (73.8% to 69.1%) and on SQL (71.3% to 67.0%). The third didn’t: GPT-6 Luna raised the SQL average and still lowered the score of the best configuration. A model that’s better on average is no guarantee for the configuration you shipped.
• Eleven of the 12 regressed individual cases, even where the average rose. A case regressed when the new model got it wrong more often than the old one did, under the same configuration. On the SQL benchmark, the Claude upgrade regressed 17 of the 20 test cases under at least one configuration. The one clean upgrade was extraction on GPT, which regressed none.
• Model upgrades sometimes fix their own bugs. Haiku 5.5’s largest gain, 25 points on extraction, is one of those. Haiku 4.5 wrapped every JSON reply in code fences and broke the strict parser; Haiku 5.5 doesn’t. Scored on the extracted values alone, the two models were close. A big lift can mean the old model was broken, not that the new one reads your configuration better.
• Tuned prompts did worst of all on their own lab’s successor. On MB1, MB3 and MB5 we also optimized prompts the way a team would: from a set of 15 candidate rewrites, written in three rounds of five by a stronger model that read the old model’s failures on ten separate development cases, keeping a rewrite only when it scored better than the current one. On Haiku 5.5, every SQL prompt tuned on Haiku 4.5 or nano lost 8 to 14 points. On GPT-6 Luna, the two recommender prompts tuned on nano were the only configurations of 24 that got worse, by 2 and 4 points, while the 20 untuned ones gained 14 on average. They ended 14 and 17 points behind Luna’s best.

Tuning bakes the old model’s habits into the prompt. One tuned extraction prompt never says that the total is the amount due, because Haiku 4.5 reported it anyway; Haiku 5.5 reports the invoice total instead. One tuned recommender prompt added a clause that tropical cities count as hot “even when the message never says hot”, which fixed a development case and then made Luna answer t-shirt for a Florida grandmother whose nights are getting cold.

Question 2: is the lab’s successor the best replacement for my agent?

Not really. If a lab builds its models to be upgraded into each other, the configurations you tuned on Haiku 4.5 should fare better on Haiku 5.5 than on GPT-6 Luna. So for each retiring model we also moved its best configuration to the other lab’s new model.

Each retiring model has one same-lab column and one switch-lab column. Bold marks the moves where one of the pair won by more than 2 points.

• Switching matched or beat the lab’s successor in 10 of 12 moves. Calling anything within 2 points a tie, switching won 5, tied 5 and lost 2. On average, switching gave the bigger lift from Haiku 4.5 and the smaller one from nano. Neither lab’s successor was the safe choice: from nano, Haiku 5.5 won on three benchmarks and tied on two; from Haiku 4.5, Luna won on two and tied on three.
• GPT-6 Luna was the stronger model on these tasks. On the 20-configuration average it beat Haiku 5.5 on four of six (recommender 88.2% to 84.6%, tool routing 81.0% to 69.1%, SQL 75.1% to 67.0%, error recovery 64.4% to 61.3%) and tied on the other two. So the claim isn’t “switch labs”. It’s that the successor is not automatically the best fit for your agent, and the only way to know is to test both.
• Staying bought no continuity. If the successor were a closer fit to what you built, the ranking of your 20 configurations would survive better within a lab than across one. We measure that with rank correlation, old model against new, where 1 is the same order and 0 is no relation. The same-lab advantage, across the six benchmarks, was +0.14, +0.21, −0.01, −0.03, +0.05 and −0.06. Only the recommender’s clears zero, and only on the Claude side; the published version of the same task went the other way, at −0.23. Across six tasks, a lab’s next model is not reliably a closer fit to your configuration than another lab’s.

There are limits. Both upgrades are generation jumps, so a point release might hold up better. Small models, reasoning off, 20 cases each: these numbers show the pattern across six agents, not whether any one 2-point difference is real. What staying with a lab does save you is harness work: the API shape, tool-calling format and structured outputs stay the same. Our test held that part constant.

So when the notice arrives, put at least one other lab’s model through the same test as the successor. You’ll be re-testing either way.

Question 3: what has to be re-tuned?

Context, skills and prompts: anything the model reads as text. What you built in code mostly carries.

On the recommender, the component a team optimizes is the prompt, and the ranking of prompts didn’t survive the swap. Each of the other benchmarks has its own version of that question: the component you’d optimize on this agent, and whether its ranking survived. “Ranking kept” is the rank correlation of the 20 configurations’ scores, old model against new, over the four moves; the models’ own run-to-run noise caps it at 0.81 to 0.99.

Four findings explain the split.

The same rule survived in code and died in text. MB2 states its six routing rules three ways: in the tool descriptions, in a guidance block in the system prompt, or enforced in the backend, where a refund request on an unshipped order is canceled server-side whatever the model called. On Haiku 4.5 to 5.5, the backend version held (81.5% to 83.0%). The description version fell 10 points (72.9% to 62.7%), and the guidance block fell 5. Haiku 5.5 read the rule and then did what the customer literally asked: told to escalate address changes on shipped orders, it changed the address, in 17 of the 29 regressed cells.

Retrieval carried because recall carried. Whether the passage holding the answer reached the model predicted each configuration’s score at a rank correlation of 0.95 to 0.99 on every model. That’s decided by the retriever, which is code, and k = 5 was the best setting on all four models. Our prediction that the new models would want a larger k was wrong.

Output formats drift between generations, in both directions. Haiku 4.5 wraps its JSON in code fences every time; Haiku 5.5 mostly doesn’t. Haiku 5.5 wraps 15% of its SQL in explanation despite “reply with only the SQL query”, so 19% of its queries fail to run, against 0.1% for Haiku 4.5. A lenient parser and an execution retry absorbed both: the retry was worth 3.5 points on Haiku 4.5 and 7.9 on Haiku 5.5. The output channel carried on the GPT pair, where the format didn’t move, and didn’t on the Claude pair, where it did. Where the model’s output changes shape, a code check on that output is what absorbs it.

Code carries only as far as the model’s reading of its inputs carries. This is MB1’s lesson, and it’s the limit on the rule above. Six of the recommender’s configurations move the rules into a rule engine; the model only fills in the facts (cold, renter, daily drinker). On Haiku, those were the configurations that moved most, not least: the only two configurations to lose more than 3 points were both code configurations, and their ranking held at 0.26 against 0.76 for the prompts. The new models filled in the facts more accurately overall, but different facts went wrong. Haiku 5.5 marks a Florida grandmother as living somewhere hot and stops marking “it’s getting colder at night” as cold, so the engine answers t-shirt five times out of five. A rule in code fires every time; when the model reads a fact differently, it fires on the wrong case.

How things broke

We read every regressed cell across the six benchmarks. Five mechanisms account for nearly all of them.

Three of these come from the model’s defaults and from how it writes its answer, not from anything the prompt says. A review of your prompt won’t find them. Only an eval that runs the real harness end to end does.

A note on the recommender itself, since its earlier version carried the first edition of this post. We rebuilt it the same way as the other five: the five cases we had edited after seeing the old models’ answers were replaced with fresh ones written before any model saw them, reasoning was switched off on all four models, and the configurations and our predictions were written down before any new model ran. The published result replicated: both successors better on average, and the ranking of prompts not surviving on the GPT pair (0.43) and well below the noise ceiling on the Claude pair (0.76 against 0.96). The published numbers stay in the data as a reference row [5].

Takeaways: where to invest?

Nearly every week brings a significant improvement in what models can do, and with it the chance to build more and build faster.

As you put those capabilities to work, the complexity of your agents rises quickly, and quality, cost and performance get harder to manage. Each component of your agent asks for its own investment. The table sorts the components by whether that investment survives a swap.

you keep the dividing line the model, and everything tied to it graded environment traces & labels harness code raw transcripts fine-tuned weights prompt wording routing assignments harness fit

Prompt optimization and model settings may offer the highest return on a day’s work, and they are also the most tightly coupled to your model: every swap resets them. Your ground truth and evals are coupled to the problem you’re solving. As long as your agent solves that problem, the work persists.

These three answers are also the first public test of a broader thesis. The labs present themselves as a one-stop shop, and our hypotheses run the other way. Models and providers are not one-size-fits-all, and only your own cases show which model fits your agent: here the successor was the worse choice as often as the better one. Most of what makes an agent good is built around the model, not inside it: here the harness code carried across the swap, and re-tuning what the model reads as text was worth up to 12 points. And your own benchmark, not a public leaderboard, is what picks a model and protects you against regressions: the regressed cases in 11 of 12 upgrades were visible only because we had a graded environment to run. The next post in this series tests the sequel: we fine-tune a small model on our own runs and move it to a second base, to measure what has to be re-earned.

An eval set has value well beyond model retirement. With the right one in place, you can apply every strategy in our Optimization Library and optimize each component systematically. The next retirement becomes a test you run, with the successor and at least one other lab’s model as candidates, rather than a project you staff.

If your evals aren’t great yet, start with AOT Evals. It’s free and runs a diagnosis in five minutes, or your money back.

Sources

1. OpenAI. “Update to OpenAI’s Self-Serve Fine-Tuning” and fine-tuned model shutdown table. Deprecations, OpenAI Platform, https://platform.openai.com/docs/deprecations.
2. Anthropic. “Model Deprecations” and “Migration Guide” (Haiku 4.5 to Haiku 5.5). Claude Docs, https://platform.claude.com/docs/en/about-claude/model-deprecations, https://platform.claude.com/docs/en/about-claude/models/migration-guide.
3. Mahalanobis Systems. Model Launch Count and Paired-Eval Sample Sizing. OpenAI and Anthropic model launches, 1 Jan. to 5 Oct. 2026. Method to be published with this post. Link TBD.
4. Mahalanobis Systems. Six Micro-Benchmarks for the Model-Upgrade Test. 25 Oct. 2026. Numbered MB1–MB6 in this post, MB0–MB5 in the repository. Recommender, tool routing, SQL, retrieval, extraction and error recovery; 20 configurations × 20 cases × 5 samples per model (3 on MB6) on Claude Haiku 4.5 and 5.5 and on GPT-5.4 nano and GPT-6 Luna, reasoning off, cases and predictions frozen before any new-model call; tuned arms on MB1, MB3 and MB5. Upgrade lift and optimality gap are averaged over the old model’s top three configurations, ties included, since with 20 cases the best score is often tied. Design, per-benchmark reports and per-call results to be published with this post. Link TBD.
5. Mahalanobis Systems. Gift Micro-Benchmark: Model-Upgrade and Provider-Continuity Runs (the published version, now the reference row for MB1). 29 Sept. and 9 Oct. 2026. 20 fixed prompts, 20 messages and 5 samples per pair on Claude Haiku 4.5 and 5.5 and on GPT-5.4 nano and GPT-6 Luna (8,000 calls), plus 19 plain-twin prompts (7,600 calls), zero reasoning tokens. Means 79.8% → 87.5% and 70.1% → 88.9%; ranking kept 0.51 and 0.31. Data and method to be published with this post. Link TBD.