Write the Latency Budget First, Then Choose the Model

Most AI features are designed in the wrong order. Someone picks a model, builds the feature, measures it at the end, and discovers it takes four seconds. Then begins a painful retrofit where every option costs quality, and the team argues about whether four seconds is really that bad.

It is that bad, and the retrofit is avoidable. Write the latency budget before you write anything else.

The method#

Start from the human, not the machine. Every AI feature exists inside an interaction with a deadline set by what the user is doing, not by what your infrastructure can manage.

Rough deadlines that hold up in practice:

• Under 100 ms — feels instant. Required for anything inline in typing, live video overlay, or a control loop.
• 100 ms to 1 s — feels responsive. Fine for search-as-you-type, autocomplete, validation on blur.
• 1 to 3 s — noticeably a request. Acceptable if the user asked for something and sees progress.
• Over 3 s — the user context-switches. Only acceptable if you have restructured the interaction: streaming output, a progress indicator with meaning, or an asynchronous result they get notified about.

Pick the number. Write it in the design document as a hard requirement, at the ninety-fifth percentile, not the median. Then subtract everything that is not the model.

What eats the budget before the model runs#

Here is a realistic accounting for a retrieval-augmented feature in a web application, and the numbers that surprise people:

Component	Typical p95
Client to your edge	20–80 ms
Auth, session, request handling	5–30 ms
Embedding the query	20–60 ms
Vector search	10–100 ms
Fetching source documents	20–200 ms
Prompt assembly	1–5 ms
Model inference	200–3000 ms
Response streaming to first token	included above
Client render	10–50 ms

Against a 1-second budget, you have spent 100 to 400 ms before the model receives a single token. Against a 3-second budget you have room, but only if the document fetch is not the pathological case — and it frequently is, because it is the step nobody profiles.

The most common discovery when we instrument this properly is that the model is not the bottleneck. It is a synchronous call to a slow internal API buried in context assembly, adding 800 ms that nobody attributed to the AI feature because it was not in the AI code.

So: profile before you optimise, and profile the whole path, not the inference call.

Where the edge genuinely helps#

“Edge inference” gets deployed for the wrong reasons frequently enough that it is worth being specific about when it earns its complexity.

It helps when network round-trip dominates. If your users are 200 ms away from your only region, that is 400 ms of budget gone on physics. A small model running near the user beats a better model far away for any latency-sensitive task. This is the same reasoning behind embedded CDN points of presence serving traffic from inside ISP networks — the fastest packet is the one that never leaves the metro.

It helps when the input is large and the output is small. Streaming a video feed to a central service to get back a boolean is a terrible trade. Run the detector where the camera is, ship the event. Bandwidth saved, latency saved, privacy improved as a side effect.

It helps when connectivity is unreliable. A ward tablet, a rural school, a vehicle. If the feature must work when the link drops, the model has to be local. There is no architecture that makes a remote call work offline.

It does not help when your task genuinely needs a frontier model’s capability, when your update cadence is high (deploying model updates to a thousand edge devices is an operations problem you should not take on lightly), or when your actual bottleneck is the database call you have not profiled.

The techniques that buy the most time#

In rough order of leverage:

Stream, always. Time to first token is the number the user feels; total generation time is the number you measure. A response that starts appearing in 300 ms and completes in 3 s is subjectively fast. One that appears complete at 2 s is subjectively slow. This is nearly free and it is the single largest perceived-latency win available.

Do the fixed work ahead of time. Embeddings for stable content, retrieval for predictable queries, prompt prefixes that never change. Anything computed per-request that could have been computed yesterday is budget you gave away. With current pricing structures, a cached prompt prefix is also roughly ten times cheaper, so this pays twice.

Parallelise the independent steps. Retrieval, user context lookup, and permission checks usually have no dependency on each other. Serialising them is a habit, not a requirement.

Route by difficulty. Most requests are easy. A small fast model handles them in a fraction of the time; escalate only what needs it. This halves p50 and improves p95 by reducing queue contention on the expensive path.

Set a deadline and degrade. Every call in the chain needs a timeout derived from the budget, and a defined behaviour when it expires — return the retrieved documents without the summary, return a cached answer, return a clear message. A feature that reliably degrades beats one that occasionally hangs.

Two examples from client work#

In a Hospital Management System, we built a clinical documentation assistant with a 1.5-second p95 budget, set by watching clinicians work: longer and they abandon it mid-note. The budget forced three decisions immediately — retrieval restricted to the current encounter rather than the full record, a smaller model with a task-specific fine-tune instead of a frontier general model, and aggressive streaming. Designed capability-first, it would have been a four-second feature that clinicians tried twice.

In a School ERP, a live timetable-conflict checker runs entirely client-side because the deadline is 100 ms — it fires as the administrator drags a class. No network call fits in that budget, so the constraint chose the architecture before anyone discussed models. That is the right order.

The rule#

Latency is a product requirement, not an implementation detail. Features that miss their deadline are not slow features; they are features nobody uses, which is the same as features you did not build.

Fix the deadline, subtract the fixed costs, and let the remainder tell you which model you can afford. It is a more constrained design space, and it produces things that ship.

---

If nobody wrote down the deadline, the feature does not have one — and it will miss it. We start every AI build with the latency budget. Tell us your number.