AI Safety Compendium

AI Safety

7 min read

AI Safety

Definition

AI safety is the field dedicated to ensuring that AI systems — especially advanced and increasingly autonomous AI — do not cause catastrophic harm to humanity. It spans technical research (alignment, interpretability, evaluations), governance and policy (ai-governance, RSPs), and strategic analysis of how to navigate the transition to transformative AI (Wikipedia: AI safety; Atlas Ch.3 — Definitions).

The field’s standard technical taxonomy comes from Hendrycks et al. 2021, Unsolved Problems in ML Safety, which decomposes AI safety into four interlocking research areas:

  • Robustness — adversarial robustness, fault tolerance.
  • Monitoring — uncertainty estimation, OOD detection, transparency, backdoor detection.
  • Alignment — proxy goals, reward-hacking, instrumental strategies, emergent goals, deception.
  • Systemic safety — cyber defense, institutional decision-making, race-to-the-bottom dynamics.

The field is often dated from Amodei et al. 2016, Concrete Problems in AI Safety — the canonical first technical-research agenda — though it has earlier intellectual roots at MIRI and the Future of Humanity Institute.

Why it matters

AI safety has been ranked the world’s most pressing problem by 80,000 Hours since 2016, on the basis of a multi-claim argument:

  • AI could replace human cognitive labor in the most economically valuable tasks.
  • This could trigger a transition comparable to but faster than past industrial revolutions.
  • The transition could be extremely rapid, driven by AI-powered feedback loops in research and development.
  • A rapid AI-driven transition raises catastrophic and existential risks.
  • Work on these problems is tractable but neglected — the window for action is narrow.

This argument has now been corroborated by an international scientific body: the International AI Safety Report 2025, commissioned post-Bletchley and chaired by yoshua-bengio with 96 experts from 30 nations + UN, treats catastrophic risk from advanced AI as a serious empirical concern requiring policy response.

The field matters because the failure modes are not contingent on a single weak premise. goodharts-law guarantees specification problems; mesa-optimization makes inner-alignment failure a structural possibility; scheming is now empirically demonstrated in frontier LLMs (Greenblatt et al. 2024 — see alignment-faking-in-large-language-models). Each piece is independently load-bearing; together they argue for the field’s existence.

Key results

  • The four canonical technical sub-fields (Hendrycks et al. 2021). Robustness / monitoring / alignment / systemic — adopted as the standard taxonomy across labs and AISIs.

  • The risk-class decomposition (80,000 Hours):

    • Power-seeking AI — instrumental-convergence-driven catastrophic risk; ranked #1 by 80k.
    • Catastrophic misuse — humans deliberately using AI for mass-harm (CBRN, cyber).
    • Extreme power concentration — AI enabling unprecedented power concentration even when “aligned” to its controllers.
    • Economic disempowerment — humans losing economic bargaining power as AI replaces cognitive labor.

  • Bletchley + the AISI institutional layer (UK AI Safety Summit 2023; International AI Safety Report 2025). The November 2023 Bletchley summit triggered a structural shift from research-and-advocacy to government-backed institutions: AI Safety Institutes in the UK, US, Japan, Singapore, France, Canada, and others; first global government-commissioned scientific review (chaired by Bengio); recurring summit cadence (Seoul 2024, Paris 2025).

  • The historical timeline of mainstream AI safety (Wikipedia: AI safety; see ai-safety) shows the field’s transition from rationalist-community origins (MIRI, FHI, LessWrong) to mainstream technical research (Concrete Problems, DeepMind/Anthropic/OpenAI safety teams) to government infrastructure (AISIs, International AI Safety Report). Each phase marks a roughly order-of-magnitude expansion in field resources.

  • Defense-in-depth is the field’s organizing pattern (Atlas Ch.3 — Definitions; Hendrycks et al. 2021). No single approach is expected to suffice. Layered defenses — alignment + control + interpretability + governance + societal preparedness — are the rule, not the exception.

  • Capability and safety scale unevenly. Capabilities follow simple, robust scaling patterns; safety properties do not. This asymmetry — capabilities improve “for free” with scale, safety must be specifically engineered — is the structural reason AI safety needs to be a deliberate research investment rather than an emergent property of capability research (Hendrycks et al. 2021; Atlas Ch.3).

Open questions

  • What’s the right balance between alignment and control? Alignment aims to make AI want what we want; control assumes alignment may fail and bounds consequences operationally. The field has not converged on the right resource allocation between the two (Atlas Ch.3).

  • Can technical safety carry the weight currently placed on it? RSPs and frontier safety frameworks bet that technical evaluations + voluntary commitments are enough. Independent analyses argue this is insufficient and binding regulation is needed. Resolution is a governance question, not a technical one (International AI Safety Report 2025).

  • Near-term harms vs. existential risk. The field debates the relative priority of present-day harms (bias, surveillance, job displacement) and long-term catastrophic risk. Whether these are complementary or competing is contested. See near-term-harms-vs-x-risk.

  • How does AI safety scale internationally? AISIs and the International AI Safety Report are the first attempts at globally-coordinated safety. Whether they hold under geopolitical competition (US/China dynamics, EU AI Act vs. US executive orders, race dynamics) is an open governance question.

  • Is the field’s current research portfolio adequate? The Hendrycks et al. 2021 taxonomy is widely accepted but the resource allocation across the four areas is contested. Robustness and monitoring receive less safety-community focus than alignment despite arguably equal importance.

The field’s research agendas are tracked in detail at the SR2025 taxonomy. Major buckets:

Sources cited

Primary URLs harvested from this page’s summary references. Auto-generated by scripts/backfill_citations.py; edit by re-running, not by hand.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.

Backlinks