Governance Problems

AI governance differs fundamentally from traditional technology regulation because AI is simultaneously a general-purpose technology, an information processor, and potentially an intelligent system — three properties that together create challenges without precedent. The AI Safety Atlas (Ch.4.1) identifies three fundamental problems and three criteria for effective governance targets.

Why Traditional Frameworks Fail

Pharmaceutical clinical trials, nuclear treaties, facility monitoring all assume:

• Predictable development paths
• Clear, narrow applications
• Controllable physical infrastructure

AI breaks all three:

• General-purpose — one system spans healthcare, finance, transportation, education simultaneously
• Information — generates and manipulates content at unprecedented scale
• Intelligence — sufficiently capable systems may evade controls or pursue unintended objectives

Three Fundamental Problems

1. Unexpected Capabilities

Foundation models exhibit emergent abilities appearing suddenly with scale. Examples:

• GPT-3 unexpectedly performed arithmetic
• Later models showed unanticipated reasoning

Current evaluations cannot guarantee absence of unknown threats, forecast new emergent abilities, or reliably assess autonomous-system risks. Testing best practices remain nascent. See capability-evaluations for the technical-evaluation side.

2. Deployment Safety

Once released, AI gets repurposed beyond intended uses:

• Same dialogue model → misinformation generator → cyberattack assistant
• Jailbreaks bypass safety measures
• Autonomous AI agents amplify risks by chaining capabilities over extended periods

3. Proliferation

“AI models are patterns of numbers instantly copied and transmitted globally.”

• Cyberattacks, insider leaks, rapid replication
• Open-source ChatGPT clones strip safety features
• Model distillation extracts capabilities without weight access
• Containment is fundamentally impossible — unlike nuclear materials

Three Criteria for Effective Governance Targets

Any successful intervention point must be:

Measurable

Can it be quantified for compliance monitoring?

• ✅ Compute (FLOPs)
• ⚠️ Data (volumes yes, content harder)
• ❌ “Algorithm quality” without operationalization

Controllable

Are there practical mechanisms to influence the target?

• ✅ Compute (NVIDIA/TSMC/ASML chokepoints)
• ⚠️ Data (non-rival, easy to copy)
• ❌ Capability (proliferates instantly)

Meaningful

Does the target shape the fundamental capability/risk profile?

• ✅ Compute (directly constrains what can be built)
• ✅ Data (directly shapes capabilities/behaviors)
• ❌ User interfaces (don’t prevent emergent capability)

Intervention Points

The development pipeline offers different intervention windows:

• Early: compute infrastructure, training data
• During: safety frameworks, monitoring systems
• Late: deployment controls, post-deployment monitoring

Most leverage is upstream (compute, data) — by deployment, proliferation makes containment hard.

Connection to Wiki

This concept page is the analytical foundation for the rest of Ch.4 governance work:

• compute-governance — meets all three criteria most cleanly
• data-governance — partial: meaningful but harder to control
• ai-governance — parent concept
• risk-amplifiers — proliferation is one of the Ch.2 amplifiers
• atlas-ch4-governance-01-governance-problems — primary source

Related Pages

• ai-governance
• compute-governance
• data-governance
• governance-architectures
• capability-evaluations
• risk-amplifiers
• ai-safety-atlas-textbook
• atlas-ch4-governance-01-governance-problems

Sources cited

Primary URLs harvested from this page’s summary references. Auto-generated by scripts/backfill_citations.py; edit by re-running, not by hand.

• AI Safety Atlas Ch.4 — Governance Problems — referenced as [[atlas-ch4-governance-01-governance-problems]]