Building an AI Center of Excellence: Structure, Charter, Failures
AI CoEs are the most-named, least-understood enterprise function. The structures that actually ship and the ones that don't.
“Center of Excellence” is the most-named, least-understood enterprise function in AI. Every Fortune 500 has announced one; most haven’t shipped anything substantial. The structures that succeed look different from the structures that don’t. This post walks through what we’ve seen work and fail across many enterprise engagements.
What an AI CoE is supposed to do#
An AI CoE typically has four nominal functions:
Capability building — develop AI skills, expertise, and assets that business units can leverage.
Governance — set standards for responsible AI deployment, model risk management, compliance.
Strategy — identify high-value AI opportunities, prioritize investment, coordinate across business units.
Acceleration — help business units actually deploy AI in production faster than they could alone.
In practice, most CoEs do (1) and (2) but fail at (3) and (4). They build internal capability and write governance documents but don’t substantially accelerate business-unit AI deployment.
The structural patterns#
Several distinct structural patterns appear:
Pattern A: Embedded SMEs. The CoE is a small team (10-30 people) of senior AI specialists who embed into business units to support specific projects. The CoE doesn’t own deployments; business units do. The CoE is leverage.
This pattern tends to work well when the SMEs are genuinely senior and when the business units have AI capability they’re trying to extend.
Pattern B: Central delivery team. The CoE is a larger team (50-200 people) that builds and operates AI capabilities centrally. Business units consume from the CoE rather than building their own.
This pattern works when the use cases are sufficiently standardized that central delivery is more efficient than distributed. It fails when business units have substantially different requirements that don’t fit the centralized capability.
Pattern C: Platform team. The CoE builds the platform (MLOps infrastructure, model registries, deployment tools) that business unit AI teams use. The CoE doesn’t build models; it builds the substrate.
This pattern works when business units have their own AI capability that benefits from common infrastructure. It fails when business units don’t have their own AI capability — then they need delivery support, not just platform.
Pattern D: Governance-only. The CoE is a small policy and governance function — sets standards, reviews high-risk deployments, runs the model risk framework. It doesn’t deliver anything substantial.
This pattern is common in regulated industries where governance is the primary need. It fails when the organization mistakes governance for capability building.
Pattern E: Multi-purpose hybrid. Most real CoEs are some mix of the above patterns.
Why CoEs typically fail#
Several patterns consistently produce CoE failure.
Unclear charter. “Be the AI center of excellence” without specifics produces a function that does many things badly rather than a few things well.
No real authority. The CoE has the responsibility but not the authority — can’t actually decide what gets built, can’t actually direct business unit resources.
Talent gap. The CoE was staffed with available people rather than world-class AI talent. The capability is not actually excellent.
Ivory tower disconnect. The CoE is too separate from business units. It produces frameworks and documents that business units ignore.
Process-over-outcomes orientation. The CoE measures itself on artifacts produced (policies written, reviews conducted) rather than business outcomes (AI deployed, value created).
Inability to actually deliver. When push comes to shove, the CoE can’t actually ship AI in production. Business units lose confidence.
What successful CoEs look like#
Successful AI CoEs in enterprise have several common characteristics:
Specific charter with measurable outcomes. “Reduce time-to-production for AI projects from 12 months to 4 months” rather than “be excellent at AI.”
World-class talent. The CoE has the senior AI talent the rest of the organization doesn’t. Substantial competitive compensation; substantial standards for hire.
Demonstrated delivery capability. The CoE has actually shipped AI in production at the organization. Track record matters.
Strong executive sponsor. A C-suite sponsor (CTO, CDO, CIO, or sometimes CEO) who actually backs the CoE’s decisions.
Defined business unit relationship. Clear about how business units engage with the CoE, what the CoE provides, what business units own.
Outcome metrics. The CoE measures success on business outcomes — AI deployed, value created, capability built — not artifacts.
The capability mix#
A high-functioning enterprise CoE in 2026 typically has substantial:
ML engineering — the technical capability to build production AI systems.
Data engineering — frequently AI projects bottleneck on data quality and availability.
MLOps — the deployment, monitoring, retraining infrastructure expertise.
Product management — translating business needs into AI specifications.
Domain expertise — understanding the business well enough to identify high-value opportunities.
Risk and governance — model risk management, compliance, responsible AI practices.
The total team size depends on organization scale and ambition. Forty to seventy is common for a multi-business-unit enterprise.
The decision framework#
When evaluating whether to build a CoE and what structure to use:
Embedded SMEs when business units have AI capability and need senior augmentation.
Central delivery when use cases are standardized and business units don’t have AI capability.
Platform team when business units have AI capability and need infrastructure.
Governance-only when the primary need is governance, not delivery.
Don’t build a CoE when you don’t have the senior AI talent to staff one. A bad CoE is worse than no CoE.
Where pdpspectra fits#
Our AI strategy practice supports enterprise AI capability building including CoE design, talent strategy, and operational scaffolding.
Related reading: the AI procurement governance post, the enterprise AI deployment post, and the AI talent strategy post.
A CoE is leverage, not insurance. Talk to our team about your AI operating model.