The Model Migration Runbook: Swapping the LLM Under a Production System

The email arrives on a Tuesday. Your provider is deprecating the model version that has been answering clinician questions in production for the last year. You have a sunset date, a suggested successor, and no idea whether the successor will behave the same way against your prompts, your tools, or your data. The arxiv framework work on end-of-life model migration opens on almost exactly this scene — a routine deprecation notice with no migration path and replacements that did not perform.

That email is now a recurring event, not an incident. Flagship refreshes from Anthropic, OpenAI, and Google land months apart rather than years, release trackers log new entries almost weekly, and the open-weight wave has produced models teams genuinely drop into agentic pipelines as substitutes for a frontier API. The market moves faster than your release train. The only sane response is to make model choice reversible — to treat “how long would it take us to swap the model?” as a measurable architectural property, and to drive that number down to days.

AWS puts the realistic range for a disciplined migration at two days to two weeks depending on complexity. Teams that take a quarter are not migrating a model. They are paying down debt they took on the day they wired a provider SDK into their business logic.

The interface boundary that makes a swap cheap#

There is one design decision that determines whether a migration is a week or a quarter: whether provider types leak.

Your system should have exactly one file that knows a vendor exists. It takes an internal request object — messages, tool definitions, a generation policy — and returns an internal response object. Nothing outside it imports a provider SDK, catches a provider exception class, or branches on a provider-specific finish reason.

The leaks are always the same, and always small enough to feel harmless when they happen:

• A provider’s message or content-block type used as the internal conversation type, so the conversation history format is the vendor’s format.
• Tool schemas authored in one provider’s dialect, so the tool registry has to be rewritten to move.
• Retry and rate-limit handling that catches vendor exception classes in the service layer instead of at the adapter.
• Prompt templates that hardcode a vendor’s preferred formatting conventions — the XML-style tag structure one family responds well to, for instance — with no way to render the same prompt differently per target.
• Token counting done with one provider’s tokenizer, then used for budgeting, truncation, and billing everywhere.

Define tools once in a provider-agnostic schema — a JSON Schema description of name, parameters, and required fields — and translate at the boundary. This matters more than it sounds, because providers do not even agree on the shape of a tool call in the response. Anthropic returns tool calls as structured content blocks; OpenAI returns them as separate fields with arguments encoded as a JSON string. If that difference is visible to your orchestration code, every model swap becomes a parser rewrite.

Prompts belong in the same discipline. Keep a canonical prompt with a per-target rendering step, so migrating means writing a new renderer rather than editing forty template strings and hoping you caught them all.

The things that break quietly#

A clean adapter makes the swap mechanically cheap. It does nothing about behaviour. The failures that hurt in a model migration are not exceptions — they are subtle changes that pass every integration test and then degrade the product.

System-prompt sensitivity. The same instructions produce different obedience profiles across model families. A prompt tuned over months against one model encodes a hundred small corrections for that model’s specific tendencies. Point it at a new one and some of those corrections become active harm — you are still suppressing a verbosity habit the new model does not have while failing to suppress the one it does.

Tool-calling behaviour, not just format. Beyond serialization, models differ in how eagerly they call tools, whether they emit several calls in one turn or insist on one at a time, how they handle optional parameters, and what they do when no tool fits. A School ERP assistant that used to call a fee-lookup tool and answer in one turn may now call it three times with slightly different arguments, or answer from memory without calling it at all. Neither is an error. Both are bugs.

JSON and structured output. JSON mode, constrained decoding, and schema-enforced structured output are different features with different guarantees, and support varies by provider and even by model tier. Function calling guides a model toward a schema; strict structured output constrains it. If your downstream automation assumed strict and you migrate to a target where it is advisory, you will discover it through parse failures in a queue at 2am.

Token accounting. Different tokenizers mean the same prompt is a different number of tokens on the new model. Every truncation rule, context-budget check, and chunk-size heuristic calibrated against the old count is now slightly wrong — usually in the direction of silently dropping the tail of a long patient history.

Stop sequences, refusals, and latency shape. Stop-sequence semantics and finish reasons vary. Refusal patterns vary more: a target model may decline a legitimate clinical summarization request the incumbent handled, and a refusal that renders as an empty answer is a support ticket, not an alert. Compare latency distributions, not averages — a model with a better mean but a fatter tail will blow the p95 budget that actually governs whether the UI feels responsive.

You cannot migrate safely without your own eval set#

Public benchmarks tell you nothing useful here. They measure a generic population of tasks; you need to know how a model performs on your prompts against your data.

Build the eval set from production traffic, which you already have if you instrumented traces. The method is unglamorous:

• Sample real requests across the distribution you actually serve — not just the happy path. Stratify by workflow, by request length, by whether tools were involved, and deliberately over-sample the rare-and-costly cases.
• Have a domain expert label the correct output. For a Hospital Management System assistant, that is a clinician or a records lead, not an engineer guessing. These human-labelled goldens are the only real ground truth you will ever have.
• Include the failures. Every complaint, every escalation, every logged bad answer becomes a permanent regression case.
• Score with the right instrument per case: exact or structural match where the answer is deterministic, an LLM-as-judge with a written rubric for open-ended text, human review for the highest-stakes slice.

A few hundred well-labelled examples beat ten thousand scraped ones. The set is an asset that appreciates — it outlives every model you run through it, and it is what converts “the new model feels better” into a number you can defend in a change review.

Shadow first, then a canary, then a kill switch#

Offline evals are necessary and insufficient — the offline set always under-represents the live distribution. So promote in stages, and treat each stage as a gate.

Run the candidate in shadow first: duplicate live requests to the new model, log its output, serve only the incumbent’s. Shadow traffic costs money and zero user risk, and it surfaces the distribution problems your sample missed — the request shapes nobody thought to include. Diff the two outputs automatically and route disagreements to review.

Then canary: a small percentage of live traffic on the new model, segmented so you can watch cohorts separately, with automated rollback wired to your quality, latency, and error signals — the standard shadow-then-canary progression applied to model versions rather than service deploys.

The non-negotiable is the kill switch. Model selection must be runtime configuration, not a deploy artifact. Reverting should be a config flip that takes effect in seconds, executable by whoever is on call, with no build and no code review. If rollback requires a release, you do not have a rollback — you have an outage with extra steps.

Per-token price is not per-task price#

The most common miscalculation after a swap is comparing sticker prices. Per-token price is an input; per-task cost is what you pay. A cheaper model that reasons at greater length, calls tools twice as often, or fails validation and gets retried can cost more per completed task than the expensive one it replaced.

Re-baseline against tasks, not tokens. Measure tokens per completed task on the new model, count tool-call rounds, count retries after failed validation, and only then apply pricing. Do it per workflow — the same side-by-side cost comparison AWS builds into its migration framework, attributed to the specific automation that spends the money. A migration is also the natural moment to re-examine routing: much of an Operational Automation workload is classification and extraction that a small model handles at a fraction of the cost, leaving the frontier model for the work that needs it.

The checklist#

Run this in order. Most of it is preparation you do once and reuse forever.

1. Confirm the boundary: one adapter, no provider types in business logic, tools defined in a neutral schema, prompts rendered per target.
2. Freeze an eval set from production traffic with human-labelled goldens and every past failure included.
3. Baseline the incumbent on that set — quality, p50 and p95 latency, cost per task. You cannot detect regression without a before.
4. Port prompts deliberately, rendering for the target’s conventions rather than pasting the old string.
5. Re-verify the tool contract: call frequency, arity, parallel calls, argument fidelity, behaviour when no tool applies.
6. Re-verify structured output against your strictest downstream consumer, and recalculate every token budget with the new tokenizer.
7. Shadow for long enough to cover a full traffic cycle, including your worst day of the week.
8. Canary with cohort-level monitoring and automated rollback.
9. Re-baseline cost per task per workflow, then revisit routing.
10. Keep the old path live and configuration-switchable for at least one full incident cycle.

None of this is exotic. It is the same reversibility discipline that separates an ERP built as a data platform with thoughtful UI on top from the heavy, data-trapped incumbents where every integration is welded in place and nothing can be replaced without a project. Applied to models, it buys you the ability to take the next release on your schedule instead of the vendor’s — and to walk away from one that disappoints without a rewrite.

---

If swapping your model is a quarter-long project, the model is not the problem — the boundary is. Tell us what you are running at /#contact.