3,000 Actions a Match, Tagged by Hand: The Labour Under Football's AI

There is a version of the World Cup technology story where everything is sensors and computer vision. Cameras track limbs, a ball reports its own acceleration, models infer the rest, and no human touches the data.

That version is wrong, and the correction is instructive. A great deal of the structured event data that fed analytics, broadcast graphics and betting markets during this tournament was produced by people watching video and tagging it by hand.

Reporting from Rest of World found annotators working in Manila, Rio de Janeiro, Cambodia, India, Cairo, Ternopil and across Eastern Europe — often footballers themselves or people with deep knowledge of the game, because you cannot tag a “progressive pass under pressure” without knowing what one is. A single annotator can capture up to 3,000 actions in a match, spending three to four hours per game. One Rio-based worker was reported at around 60 euros per match plus transport.

I am not raising this to moralise. I am raising it because the labelling layer is the accuracy ceiling of everything built above it, and most engineering organisations treat it as a procurement line item rather than an engineering system.

Why annotation quality caps your model#

This is arithmetic, not opinion. If your ground truth labels are 92% consistent with each other, no model trained on them will reliably exceed that on the same task, because the remaining 8% is noise the model cannot distinguish from signal. You can spend a quarter upgrading architectures and gain nothing, because you are not compute-limited. You are label-limited.

The tell is specific and easy to recognise once you have seen it: your model’s error rate plateaus, and the errors cluster on ambiguous cases that your own domain experts also disagree about. When two experienced reviewers look at the same input and produce different labels, you have found the ceiling. No amount of model work moves it.

We hit this constantly outside sport:

• In a Hospital Management System deployment, clinical coding for a diagnosis-prediction model came from three coders who applied one guideline differently on comorbidities. Inter-annotator agreement on the contested category was in the low seventies. The model’s ceiling was in the low seventies.
• In a School ERP project, “at risk” was labelled by pastoral staff across four campuses with no shared rubric. One campus flagged on attendance alone; another required a behavioural incident. The resulting model learned the campus, not the risk.
• In document processing for a finance client, invoice line-item extraction stalled at 89% until we discovered two annotator cohorts had opposite conventions for handling multi-line descriptions.

In every case the fix was the same, and it was not a model change.

What actually fixes it#

Measure inter-annotator agreement before you measure model accuracy. Pick a sample, have two or three annotators label it independently, compute agreement (Cohen’s or Fleiss’ kappa depending on how many raters). If you have never done this, do it this week. The number is usually worse than anyone expects, and it immediately reframes the conversation from “the model is underperforming” to “the task is underspecified.”

Treat the annotation guideline as source code. It should be versioned, it should have a changelog, and every change should trigger a re-review of a sample of prior labels. Guidelines drift silently. An annotator who joined in March is applying a different standard from one who joined in January unless you have made that impossible.

Adjudicate disagreement instead of averaging it. Majority vote on a three-way label hides the fact that the case was hard. Route disagreements to a senior reviewer and — this is the important part — feed the adjudication back into the guideline. A disagreement is a bug report against your task definition. Resolving the individual case without fixing the definition means you will see it again next week.

Keep a golden set and inject it continuously. A small, expert-labelled set mixed into the ongoing work at a low rate gives you a live quality signal per annotator, per cohort, per week. Without it, quality decay is invisible until a model retrain mysteriously underperforms.

Pay for the hard cases separately. If annotators are paid per match or per item, the incentive structure penalises exactly the careful attention that ambiguous cases require. Rate structures produce data distributions. This is not an HR point; it is a data engineering point.

The supply chain question is becoming a procurement question#

There is a second reason to care, and it is moving faster than most teams have noticed.

When a client asks where your training data came from, “we used a vendor” is no longer a sufficient answer in regulated verticals. Healthcare and public sector buyers increasingly ask about the provenance of labelled data, the conditions under which it was produced, and whether the annotation vendor’s terms permit the downstream use you are making of it. The EU AI Act’s data governance provisions for high-risk systems put documentation obligations on exactly this layer.

If you cannot answer, in a procurement questionnaire, who labelled your training data, against which guideline version, with what quality controls, you have an audit exposure sitting inside a system you thought was finished.

There is also the straightforward ethical dimension. One annotator quoted in the reporting observed plainly that the data being recorded was for betting. People with genuine expertise are producing the input to a multi-billion-dollar analytics and wagering industry at rates that do not reflect that. You do not have to solve global labour economics to at least know what is in your own supply chain, and increasingly you will be asked.

The pattern to take away#

The framing that changes how teams behave is this: annotation is not data collection, it is knowledge capture, and the guideline is the artefact.

A well-specified guideline that produces high agreement is a piece of institutional knowledge worth more than the labels it generates, because it lets you regenerate labels, onboard new annotators, expand to new domains, and defend your dataset in an audit. A vague guideline that produces 3,000 tags an hour is a machine for manufacturing noise at scale.

The World Cup version of this is glamorous — footballers in Manila tagging Spain’s build-up play at two in the morning. Your version is duller and structurally identical. Somebody, somewhere, is deciding what counts as a positive example in your dataset. Find out who, find out how, and measure whether they agree with each other.

---

Before you upgrade the model, measure whether your labels agree with each other. We have saved clients a quarter of model work with a two-day annotation audit. Ask us to look.