How to Build an AI Center of Excellence: Structure, Roles, and 90-Day Roadmap

78% of organisations now use AI in at least one business function, up from 55% just a year prior, according to McKinsey’s 2024 State of AI survey. Yet only one in three has managed to scale those efforts beyond pilots. The gap between experimentation and systematic impact is not a technology problem. It is an organisational problem.

The organisations successfully scaling AI have something the majority do not: a coordinating function that owns AI strategy, ensures governance, prevents duplication, and drives adoption across business units. This function is the AI Center of Excellence.

An AI CoE is not about hiring more data scientists. It is about creating the organisational infrastructure that allows AI capability – once proven in a pilot – to be replicated, governed, and compounded across the enterprise, rather than rediscovered and rebuilt by each new team that wants to use it.

This guide covers when an AI CoE is the right investment, how to structure it for your organisation’s size and maturity, the roles it needs to function, the governance framework it must establish, and a concrete 90-day roadmap for launching one without disrupting the AI initiatives already underway.

For more context on the mid-market AI adoption gap and what separates scaling leaders from those stuck in pilots, see our analysis of how US mid-market companies are adopting AI in operations https://www.moweb.com/blog/how-us-mid-market-companies-adopting-ai-operations.

The Business Case for an AI CoE: What Problem It Solves

Before designing the structure, it is worth being precise about what problem an AI CoE is built to solve. Different organisations need a CoE for different reasons, and the diagnosis shapes the design

The duplication problem. Three different business units are separately building knowledge assistants for their teams, each engaging different vendors, using different vector databases, indexing overlapping document sets, and building separate governance documentation. The combined investment is three times what a coordinated approach would cost. The results are inconsistent across the three systems. And none of them have the quality that a single well-resourced implementation would have achieved. An AI CoE solves this through shared infrastructure, shared vendor relationships, and a reusable deployment playbook.

The governance gap. AI systems are being deployed across the organisation faster than the compliance and legal teams can assess them. Data handling practices vary across deployments. Some teams have implemented audit trails; others have not. When a compliance question arises about an AI system, there is no clear owner and no documentation trail. An AI CoE solves this through a governance framework that every new AI deployment passes through before production, with consistent standards and clear accountability.A structured AI readiness assessment https://www.moweb.com/blog/ai-readiness-assessment-checklist-mid-sized-enterprises typically surfaces the specific governance and infrastructure gaps that the CoE charter should address first.

The scaling problem. A team ran a successful AI pilot – a demand forecasting model that reduced planning error by 28%. Six months later, the pilot is still a pilot. Nobody has been designated to own the production deployment, the integration with the ERP system has stalled, and the data science team has moved on to the next project. An AI CoE solves this through a production pathway that takes successful pilots to deployment with defined ownership, integration resources, and ongoing operational support.

The knowledge retention problem. A data science team builds institutional knowledge of your specific data environment, integration patterns, vendor relationships, and deployment approaches. When two members leave, that knowledge walks out with them. An AI CoE solves this through documented standards, reusable components, and organisational processes that encode institutional knowledge in systems rather than in people.

An enterprise experiencing two or more of these problems simultaneously is ready for an AI CoE. An enterprise experiencing none of them either has not deployed enough AI to surface them yet, or has already built the coordinating function informally within a high-functioning central team.

Three Operating Models: Which Fits Your Organisation

The structure of an AI CoE depends on your organisation’s size, AI maturity, and the degree to which AI initiatives are centralised or distributed across business units. Three operating models cover most enterprise contexts.

Model 1: Centralised CoE (Best for Early-Stage Organisations)

In the centralised model, a single cross-functional team owns all AI strategy, delivery, governance, and enablement for the enterprise. Business units submit use cases to the CoE, the CoE evaluates and prioritises them, and the CoE team builds and deploys the approved use cases.

When it works: Organisations in the early stages of AI adoption – fewer than five AI systems in production – benefit from centralisation because it consolidates scarce AI expertise, enforces consistent governance from the start, and prevents the proliferation of fragmented approaches before standards are established.

When it fails: The centralised model becomes a bottleneck as AI demand grows. When business units have more ideas than the central team can deliver, the queue grows, adoption stalls, and business units start building independently anyway – defeating the purpose of centralisation.

The transition point: When the CoE’s delivery backlog consistently exceeds two quarters of work, it is time to evolve toward a federated model.

Model 2: Federated CoE (Hub and Spoke)

In the federated model, a central CoE team sets standards, governance, and shared infrastructure, while embedded AI leads within each business unit deliver AI applications for their function. The central team does not own delivery for every business unit – it enables business units to deliver within the guardrails the central team has established.

When it works: Organisations with AI adoption underway across multiple business units, where central delivery cannot scale to meet demand. The federated model allows AI capability to grow across the organisation while maintaining governance consistency and preventing duplication of infrastructure.

When it fails: The federated model requires strong standards documentation and active enablement from the central team. If the central team is too small or under-resourced to provide genuine guidance and support to the embedded AI leads, the federated model devolves into fragmented adoption with CoE branding but without CoE coordination.

The most common enterprise AI CoE structure in 2026 is a federated model: a small central team of 4 to 8 people setting strategy, governance, and infrastructure standards, with AI practitioners embedded in each major business unit who deliver within those standards and escalate governance questions to the central team. In 2026, as agentic AI systems, autonomous agents that take multi-step actions across enterprise systems become more prevalent, CoEs are increasingly responsible for setting governance standards specifically for agent deployment, including access boundaries, human-in-the-loop requirements, and audit trail standards.

Model 3: Centre-of-Enablement (Advisory)

In the advisory model, the AI CoE functions as an internal consulting and enablement resource rather than a delivery team. Business units own their AI programmes entirely; the CoE provides guidance, reviews architectures for governance compliance, runs enablement programmes, and maintains the shared component library.

When it works: Mature AI organisations where business units have sufficient in-house AI capability to deliver independently, but benefit from a central function that prevents divergence in governance standards, maintains shared infrastructure, and provides a credible escalation path for complex questions.

This model requires the most AI maturity across the organisation and is typically the evolution of a federated model rather than a starting point.

Core Roles: Who Needs to Be in the AI CoE

The specific roles depend on the operating model and organisation size, but five role categories are present in every effective AI CoE regardless of structure.

1. Executive Sponsor and AI Leadership

Every AI CoE requires a named executive sponsor with the authority to approve the budget, set strategic priorities, and resolve cross-functional conflicts. The ownership question is one of the most politically charged CoE decisions organisations face.

Four ownership patterns are common in 2026:

• CIO-owned: Strong IT discipline and infrastructure alignment. Risk: perceived as a technology initiative rather than a business one.
• CTO-owned: Good for technically sophisticated organisations. Risk: strategy stays too close to engineering and too far from business value.
• CDO-owned: Logical when data is the primary bottleneck. Risk: CDOs often lack enterprise-wide authority to drive adoption across business lines.
• CAIO-owned: The Chief AI Officer role is increasingly common in regulated industries and large enterprises in 2025-2026. It provides dedicated executive authority for AI without conflating it with broader IT or data responsibilities.

For most mid-market enterprises without a CAIO, CDO-ownership with a strong cross-functional steering committee is the most practical approach.

2. AI Strategy and Use Case Management

The strategy function translates business priorities into an AI investment roadmap. It is responsible for: evaluating and prioritising incoming use case requests from business units; maintaining the AI portfolio (what is in discovery, development, production, and retired); communicating AI strategy to business leaders; and tracking the business value delivered by the AI programme against the investment made.

This role requires someone who can hold two conversations simultaneously: the business value conversation with executives and business unit leaders, and the technical feasibility conversation with data scientists and engineers. It is often the AI programme manager or AI strategy lead.

3. Data and Platform Engineering

The data and platform function owns the shared infrastructure that every AI use case in the organisation benefits from: the data warehouse and feature store, the model serving infrastructure, the RAG indexing and retrieval pipelines, the monitoring and observability stack, and the development environment standards.

This function prevents every business unit from building its own version of the same infrastructure. By maintaining shared platforms, it also ensures that AI systems across the organisation operate on consistent data foundations and can be monitored and governed centrally. For the technical architecture of the shared data infrastructure, the CoE’s platform engineering function maintains, see our guide to data engineering for AI: building the foundations your models actually need https://www.moweb.com/blog/data-engineering-for-ai-foundations-models-enterprise.

4. AI Governance and Risk

The governance function owns the policies, processes, and accountability structures that ensure every AI system deployed in the organisation meets the required standards: data privacy compliance, bias testing, access control, audit trail implementation, and regulatory alignment.

This role is not a gate that slows down AI deployment. It is a review and enablement function that helps delivery teams build governance from the start rather than retrofit it after deployment. A governance reviewer embedded in the CoE who reviews AI designs at the architecture stage (not the deployment stage) adds significantly less friction than one who reviews at the deployment stage.

For the governance framework that the CoE’s AI governance function implements, our guide to AI governance for LLMs and enterprise agents covers the eight controls that matter most.

5. Enablement and Adoption

The enablement function owns the organisational change management work that determines whether AI systems are actually used by the people they are built for. Technical capability without adoption generates no business value. The enablement function is responsible for: training business users and embedded AI leads; building and maintaining internal AI literacy programmes; documenting AI tools and standards in an accessible knowledge base; collecting and acting on user feedback from deployed AI systems; and celebrating and communicating AI successes across the organisation to build momentum.

This function is frequently the most underfunded part of an AI CoE relative to its impact. According to McKinsey, Bosch spends 30% of its AI programme budget on shop-floor training and change management. Most enterprises spend less than 10%. The gap shows up in adoption rates.

The Governance Framework the CoE Must Establish

Before the AI CoE can enable AI deployment at scale, it must establish the governance framework within which all deployments occur. This framework has five components:

Use case intake and evaluation process. A defined process for how new AI use case requests are submitted, assessed for strategic fit and technical feasibility, prioritised, and approved. Without a structured intake process, the CoE will be overwhelmed with requests and unable to maintain quality standards across what it approves.

AI development standards. Documented standards for how AI systems should be built: data quality requirements, model evaluation methodology, access control requirements, audit trail specifications, and documentation requirements. These standards are the reusable institutional knowledge that prevents each new project from reinventing the same wheels.

Pre-production governance review. A defined checkpoint before any AI system reaches production: a review by the governance function that confirms the system meets the required standards. This review is most efficient when it happens at the architecture design stage, before development is complete.

Production monitoring requirements. Defined standards for what every production AI system must have: performance monitoring, drift detection, access logging, incident response procedures, and a named operational owner. Organisations in regulated industries should also align their monitoring standards with ISO/IEC 42001, the AI management system standard that is increasingly required for enterprise AI compliance in 2025-2026. For the technical MLOps framework that underpins the CoE’s production monitoring standards, see our guide to MLOps best practices for regulated industries https://www.moweb.com/blog/mlops-best-practices-regulated-industries.

AI system inventory and lifecycle management. A maintained record of every AI system in production, including its current performance, its governance status, its operational owner, and its planned retirement or upgrade path. An AI system inventory is the foundation of every governance audit and every CoE maturity conversation.

The 90-Day Launch Roadmap

A practical 90-day plan for launching an AI CoE from scratch, suitable for a mid-market enterprise with existing AI pilots but no formal coordinating function.

Days 1 to 30: Foundation and Mandate

Week 1 to 2: Secure executive sponsorship and define the mandate. Identify the executive sponsor. Define the CoE’s mandate in a written charter: scope (which AI initiatives does it cover?), authority (what decisions can it make vs. recommend?), budget, and success metrics for the first 12 months. Present the charter to senior leadership for approval before any team building or process design begins.

Week 3 to 4: Conduct the AI programme audit. Inventory of every AI initiative currently underway across the organisation – pilots, production systems, vendor tools, and shadow AI deployments. For each document: what it is, who owns it, what data it accesses, whether it has governance documentation, and what its current performance status is. This audit is the baseline from which the CoE operates and the source of the first governance gaps to address. For a framework for interpreting the gaps identified in the AI programme audit, our guide to what AI readiness means for enterprises (https://www.moweb.com/blog/what-is-ai-readiness-enterprises-definition-framework-examples) covers the five readiness dimensions in detail.

Days 31 to 60: Team and Infrastructure

Week 5 to 6: Appoint or hire the founding team. The minimum viable AI CoE team for a mid-market enterprise is four people: an AI programme lead (strategy and portfolio), a data and platform engineer (shared infrastructure), a governance lead (policies and review), and an enablement lead (training and change management). These roles can be filled from existing staff (redeployed from AI-related roles) or supplemented with external expertise. The AI strategy lead is typically the hardest to source internally and the most important to get right.

Week 7 to 8: Establish shared infrastructure and standards. Using the audit findings, identify the shared infrastructure investments that will benefit multiple current and planned AI use cases. Begin the platform engineering work for the highest-priority components. Draft the first version of AI development standards based on the governance gaps identified in the audit.

Days 61 to 90: First Governance and Deployment Wins

Week 9 to 10: Conduct first governance reviews. Apply the pre-production governance review process to the highest-risk existing deployments identified in the audit – those with significant data access and no documented governance. Address the most critical gaps. Document the governance review process so it can be applied systematically to future deployments.

Week 11 to 12: Launch the first use case under the CoE process. Select one new AI use case to take through the full CoE process from intake to production. This serves as the live test of the intake, development standards, governance review, and deployment process. Document what works and what needs refinement. Communicate the success of this first structured deployment to senior leadership as evidence of CoE value.

Day 90 deliverable: Present the inaugural CoE progress report to the executive sponsor – including the AI programme audit findings, the governance gaps addressed, the shared infrastructure established, and the first use case deployed under the CoE process. This report sets the cadence for ongoing measurement and justifies the continued CoE investment.

Measuring AI CoE Success: The Metrics That Matter

An AI CoE that cannot demonstrate its own value will not survive a budget review. Four categories of metrics capture the full picture of CoE impact.

Portfolio metrics track the health of the AI programme the CoE manages: number of AI systems in production, number of use cases in the pipeline, time from intake to production deployment, and the proportion of AI investments that reach production versus stalling in pilot. The last metric is the most revealing indicator of CoE effectiveness.

Governance metrics track the compliance health of the AI portfolio: percentage of production AI systems with complete governance documentation, audit completion rate, number of governance incidents (AI systems producing non-compliant outputs), and time to resolve governance incidents when they occur.

Value delivery metrics track the business outcomes the AI programme is generating: productivity improvements by function, cost reduction from AI automation, revenue impact from AI-enabled decisions, and the aggregate ROI across the portfolio. McKinsey’s one-in-three scaling success rate implies that organisations without a CoE have poor portfolio-level visibility into these numbers. An AI CoE creates the measurement discipline that makes this visibility possible.

Adoption metrics track whether AI systems are actually being used: active user rates for deployed systems, feature adoption within AI tools, employee satisfaction scores for AI-augmented workflows, and the proportion of employees who have received AI literacy training.

Frequently Asked Questions About AI Centers of Excellence

What is the difference between an AI CoE and a data science team? A data science team builds AI. An AI CoE governs, coordinates, enables, and measures how the entire organisation builds and uses AI. In practice, data scientists and ML engineers are often members of the AI CoE, but the CoE’s purpose is broader than their technical work. The CoE owns the organisational operating model for AI – strategy, governance, enablement, and measurement – while the data science team owns specific AI system development.

How large should an AI CoE be? For a mid-market enterprise (200 to 2,000 employees), a minimum viable AI CoE team is 4 to 6 people: one strategy lead, one data and platform engineer, one governance lead, and one enablement lead. For a large enterprise (5,000+ employees), a federated model with a central team of 8 to 15 people plus embedded AI leads in major business units is more appropriate. The CoE should be as small as it can be while still fulfilling its coordinating function – overstaffed CoEs become bureaucratic bottlenecks.

How long does it take to set up an AI CoE? A minimum viable CoE – executive charter, founding team, AI programme audit, initial governance framework, and first CoE-process deployment – can be established in 90 days with the right resourcing and executive commitment. A fully mature CoE with complete governance documentation, established shared infrastructure, an operating federated model, and comprehensive metrics reporting takes 12 to 18 months. Most organisations see meaningful governance and efficiency benefits within the first quarter.

Who should own the AI CoE – CIO, CTO, or CDO? The ownership decision depends on your organisation’s specific context, but the trend in 2025-2026 is toward dedicated Chief AI Officer (CAIO) ownership in larger enterprises, and CDO ownership with a strong cross-functional steering committee in mid-market organisations. CIO ownership works well when AI is primarily an infrastructure and operational efficiency initiative. CTO ownership works well when AI is embedded in product development. CDO ownership works well when the data strategy is the primary bottleneck. The most important factor is that whoever owns the CoE has genuine cross-functional authority – not just technology authority.

What is shadow AI, and how does an AI CoE address it? Shadow AI refers to AI tools and applications adopted by business teams without central IT or governance involvement – individual employees using ChatGPT for work tasks, departments subscribing to AI-powered SaaS tools, or teams building small AI applications without formal approval. Shadow AI creates data governance risks (sensitive data sent to unapproved external services), inconsistent quality, and hidden costs. An AI CoE addresses shadow AI through an AI system inventory process that actively discovers unapproved deployments, a light-weight approval process that makes formal adoption faster and easier than shadow adoption, and an internal AI tool catalogue that gives employees approved alternatives to external tools.

How does an AI CoE relate to an existing digital transformation or innovation function? An AI CoE may be built within an existing digital transformation or innovation function, or it may be established alongside it. The key distinction is scope: digital transformation covers the full enterprise technology modernisation agenda, of which AI is one component. An AI CoE is specifically focused on the AI programme. In organisations where AI is a top-three strategic priority, a dedicated AI CoE is typically more effective than housing AI within a broader innovation function, because the specific governance requirements and operating model demands of AI require dedicated focus and expertise.

Conclusion: An AI CoE Is How You Scale From Pilots to Competitive Advantage

The enterprises achieving systematic AI ROI in 2026 are not doing it by running more pilots. They are doing it by building the organisational infrastructure that converts successful pilots into enterprise-wide capability – and that is exactly what an AI CoE provides.

Without a CoE, every AI project is a one-off: bespoke governance, bespoke infrastructure, bespoke delivery. With a CoE, every subsequent project inherits the governance framework, the shared infrastructure, the deployment playbook, and the institutional knowledge of everything that preceded it. The compounding effect is why enterprises with established CoEs are pulling ahead on AI ROI while those without them are rediscovering the same problems with each new initiative.

The 90-day roadmap in this guide is designed to get an AI CoE operational within a quarter – not fully mature, but functional enough to resolve the immediate governance and duplication problems and establish the foundation for systematic scaling.

Moweb’s AI Strategy & Consulting team designs and helps launch AI Centers of Excellence for enterprises across the UK, USA, India, and East Africa – covering charter design, operating model selection, governance framework development, and the first 90-day operational period. Talk to us about setting up your AI CoE.