AI Safety Compendium

Existential Risk

6 min read

Existential Risk

Definition

Existential risk (often “x-risk”) is any threat that could either cause human extinction or permanently and drastically curtail humanity’s long-term potential. The concept was formalized by Bostrom 2002, Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards and refined in Bostrom 2013, Existential Risk Prevention as Global Priority.

The definition explicitly covers two categories (Bostrom 2002, §1):

  • Extinction risks — scenarios that would annihilate Earth-originating intelligent life entirely.
  • Permanent curtailment risks — scenarios that would not kill everyone but would permanently prevent humanity from reaching its potential, e.g., stable totalitarian regimes or value lock-in of a narrow set of values.

The second category matters because many catastrophic outcomes do not literally end the species but lock in a permanently diminished future (Atlas Ch.2 — Risk Decomposition).

Why it matters

Existential risk is the risk class that cannot be recovered from. Ordinary catastrophes — pandemics, wars, economic collapses — leave room for civilizational recovery on the historical timescale. Existential catastrophes, by definition, do not (Bostrom 2002, §3; Bostrom 2013, §4).

The argument for prioritizing existential-risk reduction rests on a multi-step structure:

  • The future is vast. If humanity survives, trillions of people could live fulfilling lives across millions or billions of years. An extinction event destroys not just the current population but all of that potential (Bostrom 2013; Ord 2020, The Precipice).

  • Even small probability reductions matter enormously. Multiplying tiny probability changes by enormous future-population stakes produces large expected-value differences. This is the formal argument for treating x-risk reduction as a top global priority (Bostrom 2013, §5).

  • AI now dominates the risk landscape. Ord’s The Precipice assigned ~10% to “unaligned AI” as the single highest probable cause of existential catastrophe in the next century — higher than engineered pandemics, nuclear war, or climate change (Ord 2020). The 80,000 Hours problem-profile ranking has reinforced this; three of its four top-priority problems are AI-related (80,000 Hours).

Key results

  • Bostrom’s original taxonomy (Bostrom 2002). Establishes the four categories of existential risks: bangs (sudden), crunches (recovery prevented), shrieks (narrow undesirable future), whimpers (gradual loss). The taxonomy remains the field’s standard structural decomposition.

  • The Precipice’s per-century probability estimates (Ord 2020). Ord’s qualitative ranges remain the most-cited probability estimates for x-risk:

    • Unaligned AI: ~10% — single largest factor.
    • Engineered pandemics: ~3%.
    • Other anthropogenic: ~3% (nuclear war, climate change as risk amplifier).
    • Natural risks (asteroid, supervolcano): <0.1% combined. Ord’s 2024 Precipice Revisited update notes that AI capability progress has reinforced rather than weakened the original estimates.

  • Carlsmith’s argument-by-stages for power-seeking AI x-risk (Carlsmith 2022). Multi-conjunct decomposition of the AI-x-risk pathway: capability + agentic planning + misaligned goals + strategic awareness + deployment + decisive strategic advantage. Each conjunct has illustrative probability; the aggregate produces non-trivial catastrophic-risk probability without requiring any single conjunct to be near-certain.

  • Existential security (Ord 2020). Ord’s positive counterpart concept: a state where humanity has durably reduced x-risk to acceptable levels across all known threat categories. Frames x-risk reduction as not just defensive but as a constructive civilizational project — reaching a robust long-term equilibrium.

  • The risk landscape is dominated by anthropogenic threats (Bostrom 2013, §3; Ord 2020). Natural risks (asteroids, supervolcanoes) are individually low per century. The risks that aggregate to most x-risk are made by humans — and on multi-century timescales, the rate has been increasing. AI is the largest contemporary new entry.

  • The 80k risk-class structure (80,000 Hours, Risks from artificial intelligence). Three of 80k’s top-priority problems are direct AI x-risk pathways: power-seeking AI, extreme power concentration, and catastrophic AI misuse. The fourth (engineered pandemics) has significant AI-enablement concerns. See 80k-ai-risk.

Open questions

  • How accurate are the probability estimates? Ord’s qualitative ranges are widely cited but rest on subjective extrapolation. Whether they are over- or under-stated, and whether they should be updated significantly given recent capability progress, is contested (Ord 2020; Carlsmith 2022).

  • Where is the boundary between “permanent curtailment” and “very bad but recoverable”? Some scenarios (stable totalitarianism, value lock-in) have a temporal-irreversibility component that’s hard to make precise. The category boundaries are useful but the operational distinction can be fuzzy (Bostrom 2002, §1).

  • Does the “vast future” argument require longtermism? The argument for prioritizing x-risk over near-term harms typically routes through longtermist ethics. Whether the priority survives without that ethical framework is contested — see near-term-harms-vs-x-risk (Bostrom 2013, §5).

  • How do x-risks interact? Most existing analyses treat each risk independently. In practice, AI-enablement of bioweapons, AI-enabled stable authoritarianism, and AI-driven climate inaction interact non-additively. Whether the joint risk distribution is well-characterized is open (Ord 2020).

  • What’s the right portfolio across risk categories? EA-community debates regularly question whether AI x-risk should receive the dominant share of resources, or whether a more diversified portfolio across risk classes is appropriate (Atlas Ch.2 — Risk Decomposition).

Sources cited

Primary URLs harvested from this page’s summary references. Auto-generated by scripts/backfill_citations.py; edit by re-running, not by hand.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.

Backlinks