AI Safety Compendium

AI Governance

6 min read

AI Governance

Definition

AI governance is the field concerned with the policies, regulations, institutions, and international coordination mechanisms needed to ensure that the development and deployment of advanced AI systems benefits humanity rather than causing catastrophic harm. It complements technical alignment research by addressing the structural, economic, and political dimensions of ai-safety (Atlas Ch.4 — Governance Architectures; see atlas-ch4-governance-04-governance-architectures).

The Atlas’s standard three-layer architecture (Atlas Ch.4):

  • Corporate-level — lab-internal policies and frameworks (RSPs, OpenAI Preparedness Framework, DeepMind FSF).
  • National-level — domestic regulation (EU AI Act, US executive orders, UK AISI, China’s Generative AI rules).
  • International-level — cross-border coordination (Bletchley Summit, AI Safety Institute network, International AI Safety Report 2025).

Why it matters

Technical alignment alone cannot prevent AI catastrophe — even a perfectly aligned AI system could be dangerous if controlled by a small group with unchecked power, and even robust safety techniques are useless if competitive pressures drive labs to skip them (Atlas Ch.4 — Governance Problems; see atlas-ch4-governance-01-governance-problems).

Three structural problems governance addresses:

  • Race-to-the-bottom dynamics. Without coordination, AI developers face competitive pressure to skip safety investment. Governance frameworks (binding commitments, evaluation requirements, audit access) try to make safety investment incentive-compatible across competing labs (Atlas Ch.4 — Systemic Challenges).

  • Power concentration. Advanced AI could enable unprecedented concentration of power — corporate, national, or in a small group. The mechanisms driving AI-enabled power concentration are the same whether the controllers are aligned or not (80,000 Hours, Extreme Power Concentration; see stable-totalitarianism).

  • International externalities. AI development is global, but regulation is national. Bridging this gap is the central challenge of AI governance and the explicit motivation for the post-Bletchley AISI network (International AI Safety Report 2025).

Key results

  • The EU AI Act (artificialintelligenceact.eu). The first major comprehensive AI regulation. Introduces a risk-tiered structure (unacceptable / high / limited / minimal risk), specific GPAI (general-purpose AI) obligations, transparency requirements, and the EU AI Office. Effective August 2024, with phased application through 2026-2027. The EU AI Act has had a significant Brussels effect — its provisions are influencing AI regulation worldwide. See brussels-effect.

  • The Bletchley + AISI institutional layer (post-2023). The November 2023 UK AI Safety Summit triggered a structural shift from research-and-advocacy to government-backed institutions: AI Safety Institutes in UK, US, Japan, Singapore, France, Canada; recurring summit cadence (Seoul 2024, Paris 2025); first global government-commissioned scientific review chaired by yoshua-bengio (International AI Safety Report 2025).

  • The US Executive Order on AI (Biden EO 14110, October 2023). Required reporting on dual-use foundation models, established US AISI, mandated agency-by-agency safety planning. Largely rescinded under the Trump administration’s “National Policy Framework for AI” (December 2025), but the AISI persists. The episode illustrates how fragile US-side AI governance can be without legislative grounding.

  • Compute governance (Atlas Ch.4 — Compute Governance; see atlas-ch4-governance-02-compute-governance). Compute is identifiable, expensive, and concentrated — making it the most tractable governance lever. Concrete proposals: chip-supply-chain monitoring, on-chip controls, KYC for compute providers, export restrictions (US chip export controls to China). This is the most operationally-developed strand of AI governance.

  • Bostrom’s vector-field framework (Bostrom 2020, Public Policy and Superintelligent AI: A Vector Field Approach). Rather than proposing specific regulations, identifies policy directions that are robust across different AI-emergence scenarios. An influential framework for thinking about robust governance under uncertainty.

  • The “AGI as government project” thesis (Aschenbrenner 2024, Situational Awareness). Argues superintelligence will inevitably become a government project, with private labs and state resources converging by 2027–28 — analogous to DoD relationships with Boeing/Lockheed. The thesis is contested but influential in policy debate, including via the AI 2027 scenario which dramatizes the trajectory.

Open questions

  • Can voluntary commitments survive competitive pressure? Frontier-lab RSPs and frontier-safety frameworks are voluntary. The Superalignment dissolution at OpenAI is a data point that they may not. Whether governance must move to mandatory frameworks to be effective is a central debate (International AI Safety Report 2025).

  • How tight should compute governance be? Tight controls slow capability research (good for safety) but concentrate compute access (bad for distributed control). The Pareto frontier between these is contested (Atlas Ch.4 — Compute Governance).

  • Can international coordination hold under geopolitical competition? AISIs and the International AI Safety Report are the first attempts at globally-coordinated safety. Whether they hold under US/China competition, especially as AI is increasingly framed as a national-security issue, is empirically open.

  • Does regulation lag capability fatally? EU AI Act’s GPAI rules apply to systems whose capabilities have already moved on. The pace mismatch between capability research (months) and regulatory cycles (years) may make some governance approaches structurally unable to keep up (Atlas Ch.4 — Systemic Challenges).

  • What governance regimes work for extreme power concentration specifically? Most existing AI-governance work focuses on misuse and accidents. The “even-aligned-AI-can-cause-power-concentration” failure mode receives much less governance attention; whether existing tools can address it is largely open (80,000 Hours).

Sources cited

Primary URLs harvested from this page’s summary references. Auto-generated by scripts/backfill_citations.py; edit by re-running, not by hand.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.

Backlinks