The Bottleneck Isn't GPUs: Data Infrastructure and AI-Driven Science

A materials lab gets an allocation on a national compute cluster. Good news, celebrated in the group meeting. Six weeks later the AI project has produced nothing, and the reason is not the cluster. It is that the training data lives in 412 spreadsheets on a departmental file share, exported over nine years from four generations of instrument software, with column headers that drifted every time a new student took over the rig. Two of those years have no data at all because the person running the experiments left and nobody knew the export script was on their laptop. The GPUs sit idle while three postdocs argue about whether temp_C and Temp mean the same thing.

This is the normal case, not the pathological one. And it is why the most interesting infrastructure news of the month was not about chips.

The awards that went to plumbing#

On July 22, 2026, the U.S. National Science Foundation announced $83 million in awards through its Integrated Data Systems and Services (IDSS) program — money aimed squarely at expanding the data infrastructure researchers can use alongside computing and AI resources. The funded projects are national-scale data plumbing: Fabric for AI-Driven Science (FabAID) at the Morgridge Institute for Research and the National Data Platform (NDP) at UC San Diego in the first category, with the Interactive Discovery Laboratory at UCLA, the BRIDGE National Center at UC Irvine, the National Science Data Fabric at Tennessee, and MESA at Arizona following. The stated goal is to improve how scientific data is collected, stored, shared and analyzed across NSF facilities, and to make it easier to find, use and reuse data across projects and fields.

Read that as an admission. When a funding agency spends this much on connecting repositories and making workflows interoperable, it is because the compute has outrun the data. You can hand a lab a GPU allocation in an afternoon. You cannot hand them a clean, documented, queryable corpus of their own past ten years of work — that has to be built, and almost nobody budgeted for it.

Research data fails in ways enterprise data does not#

Enterprise data is messy. Research data is messy in different, worse ways, and the standard corporate playbook mis-diagnoses it.

• The format is the instrument’s, not yours. A mass spectrometer, a sequencer, a confocal microscope each emit a proprietary or semi-documented binary. The vendor changes it between firmware versions. There is no upstream team you can file a ticket with.
• The schema lives in one person’s head. In a company, three people can explain the orders table. In a lab, the only person who knows why there are two run_id columns is the graduate student who built the collection script, and they defended eighteen months ago.
• The gaps are years long. Instruments go down. Funding lapses. A pandemic empties the building. Enterprise pipelines have outages measured in hours; research datasets have holes measured in semesters, and they are not recoverable.
• Ownership expires by design. The whole institution is built around people leaving. When a student graduates, custody of their data transfers to nobody in particular. There is no offboarding checklist that reassigns a dataset the way a company reassigns an on-call rotation.
• Volume is not the problem; variety is. A single instrument run may be modest. The difficulty is that a lab has thirty of them, each with its own dialect, and the scientific question requires joining across all of them.

None of these are solved by a bigger warehouse. They are solved by conventions, ownership, and tooling that survive personnel turnover — which is an organizational problem wearing an engineering costume.

FAIR as an engineering spec, with a price tag per letter#

The FAIR Guiding Principles — Findable, Accessible, Interoperable, Reusable — have been on posters in research buildings since 2016. Treated as values, they change nothing. Treated as an engineering specification with a cost estimate per letter, they become tractable.

Findable means persistent identifiers and machine-readable metadata. The real cost is not the DOI; it is the metadata registration step that someone must perform for every dataset, forever. Budget it as an ongoing operational duty, not a one-time migration.

Accessible means a documented retrieval protocol with defined authentication — and metadata that stays reachable even when the data itself cannot be shared. For human-subjects work that is the difference between a compliant dataset and an unusable one. The cost is an access-control layer and someone empowered to make release decisions.

Interoperable is the expensive letter. It means a shared vocabulary and a common representation, which means somebody has to decide that temp_C wins and then rewrite nine years of ingest. This is where projects quietly die. The honest estimate is measured in months of engineering, not sprint points.

Reusable means licensing clarity plus provenance rich enough that a stranger can judge fitness for their purpose. The cost is documentation discipline enforced at write time, because retroactive documentation does not happen.

If you cannot name who pays for each of those four, you do not have a FAIR programme. You have a poster.

Provenance is not optional when the model’s output becomes a claim#

In commerce, a bad recommendation costs a conversion. In science, a model’s output becomes a sentence in a paper — and that sentence enters the literature, gets cited, and shapes what other people spend three years doing. The chain from raw instrument file to published claim has to be reconstructible on demand: by a reviewer, by a replication team, by the authors themselves when someone finds an anomaly.

That means provenance captured mechanically, not narrated afterwards. Every derived artifact should carry the identity of its inputs, the code version that produced it, the parameters, and the environment. When a preprocessing bug is discovered — and it will be — the question “which figures are affected?” must be answerable by query rather than by archaeology. A lab that cannot answer it has to either retract broadly or hope quietly, and both are bad science.

Provenance is also what separates a genuinely reusable dataset from a nominally public one. A stranger downloading your corpus needs to know whether the values were background-subtracted, which calibration file was current, and whether outliers were removed or retained. Without that, they are not reusing your data; they are guessing at it.

Reproducibility is versioned data plus versioned pipelines#

A README is a promise. Reproducibility is two artifacts under version control: the data and the transformation that produced it.

The tooling exists and it is not exotic. DVC puts datasets and pipeline stages under Git-adjacent version control, each stage declaring explicit inputs and outputs so the dependency graph is machine-checkable. For petabyte-scale storage, lakeFS applies branch-commit-rollback semantics to object stores without duplicating the bytes. Containers pin the environment; a lockfile pins the dependency tree.

The pattern that works is boring: raw data is immutable and append-only, every derived layer is regenerable from raw by a pipeline stored in the same repository as the analysis, and the paper cites a commit. When a reviewer asks how a number was produced, you hand over a hash, not a narrative. This is the same discipline good MLOps teams already run — the novelty in a research setting is only that the output is a claim rather than a prediction.

The infrastructure outlives the grant that built it#

Here is the failure mode that no architecture diagram shows. A grant funds three years of platform work. The platform gets built, gets used, becomes load-bearing for four downstream labs. Then the grant ends, the research software engineer takes an industry job, and the thing rots — dependencies drift, certificates expire, the storage bill lands on a department that never agreed to it.

This is a structural problem, not bad luck: funding is scoped to development while maintenance obligations run indefinitely, and the community has argued for explicit maintenance provisions and sustainable financing models for data reuse infrastructure for years. The O3 guidelines push the same direction — open data, open code, and open infrastructure specifically so a curated resource can outlive the team that started it.

The engineering response is to design for abandonment from day one. Prefer standard formats over clever ones. Prefer managed services with a predictable monthly line item over a bespoke cluster that requires an expert. Write the runbook assuming the author is gone. Make the exit path — full export to open formats — a tested feature, not a theoretical one. If the platform cannot be handed to a competent stranger with a week of ramp-up, it is a liability disguised as an asset.

What transfers from enterprise practice, and what doesn’t#

Plenty of hard-won Data Platforms practice moves across cleanly. Data contracts work: an agreed schema at the boundary between an instrument pipeline and downstream analysis catches drift at write time instead of at paper-writing time. Catalogs work, because discovery is the actual bottleneck when a new student joins. Orchestration works — a scheduled, monitored, retryable DAG beats a cron job on a workstation under a desk. Operational Automation of ingest and validation removes exactly the toil that nobody’s job description covers.

What does not transfer is the assumption of a permanent, funded platform team, and the assumption that schemas stabilize. In a company, the orders table settles down. In a lab, the experiment is supposed to change — that is the point of research — so a rigid canonical model becomes an obstacle within a year. The right shape is a strict, immutable raw layer with permissive schema, and opinionated curation only at the level where analysis actually happens.

The parallel to operational systems is closer than most researchers expect. A Hospital Management System holds the clinical records that become the substrate for outcomes research; a School ERP holds the attendance and assessment histories that education researchers spend years trying to reconstruct from exports. In both cases the data is research-adjacent, and in both cases legacy vendors trap it behind reporting modules and per-seat export licences. The alternative we argue for is the same in a lab and in a hospital: build the ERP as a data platform with a thoughtful UI on top, so the record is queryable by design instead of extractable under duress.

The lesson from the NSF’s $83 million is not that data infrastructure is finally solved. It is that the field has stopped pretending the constraint was ever the compute. Any serious AI implementation — in a national facility or a single lab — is a data engineering project first, and the sooner that is on the budget line, the sooner the GPUs stop idling.

---

If your AI project is blocked on data you cannot find, trust, or reproduce, that is an infrastructure problem with a known shape. Tell us what you’re building at /#contact.