Plumbing-First AI: Why Implementation Is Mostly Data Engineering

There is a recurring scene on AI projects. A team spends three weeks comparing models, runs a bake-off across a half-dozen providers, argues about benchmarks, and picks a winner by a few points on some leaderboard. Then they wire it up to their actual data and the thing falls over. The answers are stale, the retrieval pulls the wrong document, the latency is unacceptable, and nobody can say what any of it costs per request.

The uncomfortable truth of AI implementation is that the model is the smallest part of a working system. Everything that determines whether your AI ships — and keeps working after it ships — is data engineering. Retrieval quality, latency budgets, evals, observability, cost tracking per workflow. The model is a component you plug into infrastructure you have to build. Get the plumbing wrong and the best model on the planet produces confident nonsense.

The model is interchangeable; the context is not#

Start with the part everyone gets backwards. Frontier models from Anthropic, OpenAI, and Google are close enough on most production tasks that the choice rarely decides the outcome. They are also swappable — a well-built system lets you change the model behind an interface in an afternoon. What is not swappable is the pipeline that decides what the model sees.

An LLM answers based on the context you hand it. If that context is wrong, incomplete, or stale, the answer is wrong, and no amount of model capability rescues it. This is the single most expensive misunderstanding in the field: teams treat the model as the product and the data pipeline as glue code. It is the reverse. The pipeline is the product. The model is glue.

Consider a Hospital Management System with an AI assistant that answers clinician questions about a patient. The model is incidental. What matters is whether retrieval surfaces this patient’s current medication list and not last admission’s, whether the lab results are the latest draw, whether allergy flags are present in the context window at all. Those are data engineering questions — freshness, joins, access control, indexing. A wrong answer here is not a model failure. It is a plumbing failure wearing a model’s clothes.

Retrieval quality is a data problem, not a prompt problem#

Most “the AI is hallucinating” complaints are retrieval failures. The model was handed the wrong chunks and faithfully summarized them. When teams respond by tuning prompts, they are sanding a wall that is falling because the foundation is cracked.

Retrieval quality comes from boring, measurable work. Chunking strategy that respects document structure instead of splitting mid-sentence. Embeddings that match your domain. Metadata filtering so a query about one department cannot pull another department’s records. Hybrid search — keyword plus vector — because pure semantic search misses exact identifiers like an order number or a patient ID, and pure keyword search misses paraphrase. Reranking to push the genuinely relevant results to the top of a tight context window.

None of that is glamorous. All of it decides whether the system works. In a School ERP, a parent asking “when is my child’s fee due” needs the system to retrieve the correct student record, the correct term, and the current balance — not a similar-looking record from a sibling or a prior year. That is a join and a filter, expressed through retrieval. The model only formats the answer.

Latency budgets are data problems#

“The AI feels slow” almost never means the model is slow. Token generation is usually a small slice of the wall-clock time a user waits. The rest is plumbing: the embedding call, the vector lookup, the database queries that hydrate retrieved IDs into real records, the reranking pass, the serialization, the network hops between services you stitched together.

Treat latency the way a data engineer treats a slow query — as a budget you allocate across stages and then measure. Decide the request must return in under two seconds, then account for every stage against that budget. Cache embeddings for repeated queries. Precompute what can be precomputed. Run independent retrieval calls in parallel instead of in series. Keep the data the model needs close to the model instead of three microservice hops away.

This is where performance-first data infrastructure earns its place. A query layer built on something like ClickHouse that returns aggregates in milliseconds is the difference between an AI feature that feels instant and one that users abandon. The model did not get faster. The plumbing did.

Evals: you cannot improve what you cannot measure#

Shipping an AI feature without evals is shipping blind. You have no way to know whether a prompt change, a model swap, or a retrieval tweak made things better or worse. You are guessing, and your users are the test suite.

An eval is a dataset of representative inputs paired with a way to judge outputs — exact match where answers are deterministic, an LLM-as-judge for open-ended responses, or human review for the highest-stakes cases. Run it on every meaningful change. Track the score over time. The first version can be fifty hand-written examples in a spreadsheet; the discipline matters more than the scale.

Evals are also how you say no to a model upgrade safely. A new release drops, marketing promises it is better, and your eval suite tells you in an hour whether it actually is for your workload — not for a generic benchmark. Without evals, every upgrade is a coin flip you make in production.

Observability and cost tracking per workflow#

Once an AI system is live, you need to see inside it. Log the full trace of every request: the query, what was retrieved, the assembled context, the model call, the response, and the latency of each stage. When something goes wrong — and it will — this trace is the difference between a ten-minute fix and a week of speculation. This is the same observability discipline you would demand of any production data pipeline, applied to the AI path.

Cost tracking deserves its own line item because token costs hide in aggregate. A dashboard that says “we spent X on the LLM provider last month” is useless. The number you need is cost per workflow. Which feature burns tokens? Is one workflow stuffing the entire knowledge base into every prompt? Are you paying frontier-model rates for a classification task a small model would nail?

Attribute every token to a workflow and the optimizations become obvious. Route simple tasks to cheaper models. Cache responses to repeated questions. Trim bloated context. Teams that track cost per workflow routinely cut their bill by half without touching answer quality — because most of the spend was waste the aggregate number hid.

The cost of bad context#

Tie it together with the failure mode that sinks AI projects quietly: bad context. Not an outage, not an error — just answers that are subtly, plausibly wrong because the data feeding the model was wrong.

Bad context erodes trust faster than any visible bug. A clinician who catches the assistant citing a discontinued medication stops trusting it entirely, and rightly so. A parent given the wrong fee balance from a School ERP calls support, and now the AI created work instead of removing it. The model performed exactly as designed. The plumbing fed it garbage, and garbage in confident prose is worse than no answer, because it looks like an answer.

Where model choice actually matters#

To be fair to the other side: model choice is not irrelevant. It matters at the edges. If your task needs a context window large enough to hold an entire patient history, that constrains your options. If you need strong tool-use or reliable structured output for an automation that feeds downstream systems, models differ meaningfully. If latency is brutal and you are willing to trade some quality, a smaller, faster model can be the right call. And cost per token, multiplied across millions of requests, is a real number that should inform routing.

But notice the pattern in every one of those cases: the deciding factor is a property of your system — your context size, your output contract, your latency budget, your request volume. You cannot even evaluate which model fits until the plumbing exists to define those constraints. The model decision is downstream of the data decisions, not upstream of them. Teams that pick the model first are answering a question they have not yet earned the right to ask.

A working order of operations#

If you are starting an AI Implementation, invert the usual sequence. First, get the data right: identify the sources, fix freshness, sort out access control. Second, build retrieval and measure it against a small eval set before a single user sees it. Third, instrument everything — traces and cost attribution from day one, not bolted on after the incident. Fourth, set a latency budget and hold the system to it. Only then, with all of that in place, pick a model — and treat the choice as reversible, because a system built this way lets you swap models behind an interface without rewriting anything that matters.

This is why the call is unambiguous: spend less time choosing the model and more time building the pipeline that feeds it. Freshness, retrieval accuracy, access control, latency, evals, observability, cost. That is where working AI lives. The model is the last ten percent, and it is the part that takes care of itself.

---

AI implementation is data engineering with a model on top — build the plumbing first. Tell us what you’re shipping at /#contact.