AI Safety Compendium

Summary: Existential Risks — Analyzing Human Extinction Scenarios and Related Hazards

4 min read

Summary: Existential Risks — Analyzing Human Extinction Scenarios and Related Hazards

Author: Nick Bostrom Year: 2002 Source: bostrom-existential-risks.pdf Published in: Journal of Evolution and Technology, Vol. 9, March 2002

Main Argument

This is the foundational paper that introduced existential-risk as a formal analytical category. Bostrom argues that humanity is rapidly approaching a critical phase due to accelerating technological progress. While we have long experience managing personal, local, and endurable risks, existential risks — those that could cause our extinction or permanently and drastically curtail our potential — are a fundamentally new kind of threat for which our evolved biological and cultural coping mechanisms are inadequate.

The paper’s core thesis is that existential risks require a proactive rather than reactive approach, because there is no opportunity to learn from errors. A trial-and-error strategy is inherently unworkable when the first failure is terminal.

Key Concepts

The Risk Taxonomy

Bostrom classifies risks along three dimensions: scope (personal, local, global), intensity (endurable vs. terminal), and probability. Existential risks occupy the intersection of global scope and terminal intensity. He then subdivides existential risks into four categories, inspired by T.S. Eliot’s “The Hollow Men”:

  • Bangs — Sudden extinction events (e.g., nuclear war, nanotech weapons, superintelligent AI gone wrong, engineered pandemics, asteroid impact, physics disasters)
  • Crunches — Humanity survives but is permanently thwarted from reaching its potential (e.g., resource depletion preventing civilizational recovery, repressive world government, dysgenics)
  • Shrieks — Posthumanity is attained but in an extremely narrow, undesirable form (e.g., a single upload achieves intelligence-explosion and imposes its values; a flawed ai-alignment locks in bad goals)
  • Whimpers — Posthuman civilization arises but gradually and irrevocably loses what we value (e.g., evolutionary drift toward entities we would not consider valuable; hostile alien encounter)

Specific Risks Analyzed

Among the “bangs,” Bostrom ranks the following (roughly by estimated probability):

  1. Deliberate misuse of nanotechnology — Self-replicating nanobots could destroy the biosphere; offense is easier than defense
  2. Nuclear holocaust — Existing arsenals, future arms races, and nuclear winter remain genuine threats
  3. Simulation shutdown — If the simulation argument is taken seriously, our reality could be terminated
  4. Badly programmed superintelligence — An AI given flawed goals could convert all matter to serve its objective (an early articulation of instrumental-convergence)
  5. Engineered pandemic — Genetic engineering enabling creation of doomsday viruses
  6. Natural pandemic, asteroid impact, runaway greenhouse effect

Indirect Methods for Estimating Risk

Bostrom discusses three indirect approaches to assessing existential risk probabilities:

  • The Fermi Paradox — The absence of observable alien civilizations implies at least one “Great Filter” in the evolution of intelligent life. If that filter lies ahead of us, most civilizations destroy themselves.
  • Observation selection effects — The Doomsday argument and anthropic reasoning constrain our estimates, though foundational questions remain unresolved.
  • The Simulation Argument — This redistributes probability in ways that narrow the range of positive future scenarios.
  • Psychology of risk perception — A “Good-story bias” leads us to overestimate dramatic, narratively satisfying risks and underestimate boring extinction scenarios.

Policy Implications

Bostrom derives several actionable recommendations:

  1. More research is urgently needed — There is more scholarly work on the dung fly than on existential risks.
  2. International cooperation is essential — Existential risk reduction is a global public good that will be undersupplied by markets.
  3. Preemptive action may sometimes be justified — In extreme cases where a nation’s reckless development of dangerous technology threatens all of humanity, sovereignty concerns may be overridden.
  4. differential-development — Rather than trying to ban technologies (which is often infeasible), we should accelerate defensive technologies and delay offensive ones. Superintelligence itself is identified as broadly risk-reducing because it could advise on policy and shorten vulnerability windows.
  5. Maxipok rule — “Maximize the probability of an okay outcome” — a satisficing heuristic that prioritizes avoiding existential catastrophe over optimizing for any particular good outcome.

Significance

This paper is the intellectual origin point for the entire field of existential risk studies. It provided the conceptual vocabulary (existential risk, bangs/crunches/shrieks/whimpers, Maxipok), the analytical framework (scope/intensity/probability), and the moral argument (the astronomical value of the future) that would later be developed more rigorously in Bostrom’s 2013 paper “Existential Risk Prevention as Global Priority” and in toby-ord’s The Precipice. The paper also contains early articulations of ideas central to ai-safety — including the risk of badly programmed superintelligence and the concept of differential-development — that Bostrom would later elaborate in Superintelligence (2014).

Bostrom’s subjective estimate that the combined probability of existential catastrophe should not be set lower than 25% was striking at the time and helped catalyze the effective-altruism movement’s focus on longtermism and x-risk reduction.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.