Entity Resolution Is the Prerequisite Nobody Budgets, and It Kills More AI Projects Than Models Do
The same person exists three times in your systems with no reliable mapping. Every analysis downstream is wrong and nobody can tell you by how much.
Here is a diagnostic you can run this week. Pick your most important entity — customer, patient, student, supplier, asset. Count how many distinct identifiers it has across your systems. Then ask who owns the mapping between them.
In most organisations the answer to the first question is between three and nine, and the answer to the second is a colleague’s spreadsheet, or nobody.
That is entity resolution, and its absence is the most common root cause of AI implementation projects that produce technically correct outputs which domain experts refuse to trust.
Why it is invisible until it is expensive#
Broken entity resolution does not throw errors. Your pipeline runs. Row counts look plausible. Dashboards render. The model trains and reports a respectable metric on a holdout set drawn from the same corrupted data.
What actually happens is subtler and worse:
- Aggregates undercount. A customer who appears three times has their lifetime value split three ways. Your “top customers” list is wrong, and it is wrong in a direction that systematically favours entities with clean records.
- Features leak or vanish. A patient’s prior admission is in the record under a different identifier, so “number of previous admissions” is zero. The model learns that patients with no history are low risk, which is true of genuinely new patients and false of the ones your join failed on.
- Labels attach to the wrong entity. The outcome you are predicting was recorded against identifier B while the features came from identifier A. You have introduced label noise that no amount of regularisation addresses.
- Deduplication happens per-report, inconsistently. Two analysts write two slightly different dedupe rules for two dashboards, the numbers disagree, and the organisation concludes the data platform is unreliable. It is — but not for the reason anyone says out loud.
The reason nobody catches it is that there is no test that fails. The system’s behaviour under broken entity resolution is indistinguishable from the system’s behaviour under a genuinely harder prediction problem. Teams respond by trying better models. It does not help, because they are not model-limited.
The shape of a real solution#
Entity resolution is a well-understood engineering discipline. It is not research. It has four parts and each one has a decision you have to make deliberately.
Blocking: reduce the comparison space. Comparing every record to every other record is quadratic and infeasible past trivial sizes. Blocking groups records into candidate sets that could plausibly match — same postcode, same date of birth, same phonetic surname key — and you only compare within blocks. The design tension is that a tight blocking key is fast but misses matches whose block key itself is wrong (a typo in the postcode). The standard answer is several independent blocking passes with different keys, unioned.
Matching: score candidate pairs. Per-field comparison — exact, edit distance, phonetic, date proximity, address normalisation — combined into a score. Probabilistic record linkage (the Fellegi-Sunter framework) remains the right default because it weights each field by how discriminating it actually is: matching on a rare surname is strong evidence, matching on a common one is nearly none. A learned classifier over pair features works well too, if you have labelled pairs. You usually will not at first, which is why you start probabilistic.
Clustering: turn pairwise matches into entities. This is where naive implementations break. If A matches B and B matches C but A does not match C, what do you do? Transitive closure will happily merge your entire database into one giant cluster through a chain of weak links — we have seen it happen, and it happened because someone joined on a placeholder phone number. Use connected components with a threshold, or hierarchical clustering, and always cap cluster size with an alert, because a cluster of 4,000 “people” is a bug, not a discovery.
Survivorship: decide the golden record. Which address wins when the three source records disagree? This is a business rule, not a technical one, and it must be written down: most recent, most complete, highest-trust source, or per-field combinations of those. Undocumented survivorship rules are how “the system changed my address” tickets get created.
The two things that make it stick#
Getting a good match run is the easy half. Keeping it correct is where projects fail.
Persist the identity, do not recompute it. If you re-run resolution from scratch each night, identities are unstable — the same person can land in a different cluster tomorrow because one new record shifted a score. Downstream systems that stored yesterday’s identifier are now wrong. Assign a stable surrogate key, persist the crosswalk, and treat new records as incremental assignments against existing clusters rather than a full re-clustering.
Build a human review path for the middle. Every entity resolution system has three zones: confident match, confident non-match, and a band in between where the score is genuinely ambiguous. Automatically merging the middle band produces false merges, which are far more damaging than false splits — a false split costs you a joined record, a false merge puts one person’s data in front of someone else. In a hospital that is a patient safety incident and a notifiable breach.
Route the ambiguous band to a reviewer with the evidence assembled, and feed decisions back as training data. This is the same detect-present-decide pattern as semi-automated officiating, for exactly the same reason: asymmetric costs on a workload made entirely of edge cases.
What this looks like in our verticals#
In a Hospital Management System, duplicate patient records are a well-documented safety problem, not just an analytics annoyance. Allergies recorded against one identifier and a prescription written against another is a mechanism for real harm. Hospitals that have run a proper resolution exercise routinely find duplicate rates in the low single-digit percentages of the master index — which sounds small until you multiply it by admissions and note that the duplicates concentrate in exactly the patients who attend most often.
In a School ERP, the same student appears in admissions, the learning platform, the catering system and the transport roster, frequently with different name spellings and no shared key. Any attempt to build early-warning analytics across attendance, attainment and welfare requires the identity spine first. Build the model before the spine and you will produce a risk score that is confidently wrong for exactly the mobile, frequently-transferring students who need it most.
The budget argument#
Entity resolution is politically hard to fund because it produces no visible feature. Nobody demos a crosswalk table. It reliably gets cut in favour of the model, which demos beautifully.
The counter-argument that works in my experience is not technical. It is this: every downstream analysis is a claim about a specific entity, and if you cannot identify the entity reliably, you are not building an analytics capability — you are building a machine for producing plausible numbers.
Six weeks of resolution work is cheaper than a year of models nobody acts on. And unlike the model, it does not depreciate when someone releases a better one next quarter.
If the same person exists three times in your systems, no model above that is trustworthy. We build the identity spine first, because everything else depends on it. Ask us to audit yours.