AI Safety Compendium

Nick Bostrom

3 min read

Nick Bostrom

Nick Bostrom is a Swedish-born philosopher at the University of Oxford whose work has been foundational to existential-risk research, ai-safety, and the intellectual infrastructure of effective-altruism. He coined the formal concept of “existential risk” and wrote Superintelligence, one of the most influential books in the AI safety canon.

Key Works

Academic Papers

Bostrom has published five papers covered in this wiki’s academic papers collection:

  1. Existential Risks: Analyzing Human Extinction Scenarios (2002) — The paper that introduced existential-risk as a formal category, defining it as threats that could annihilate Earth-originating intelligent life or permanently curtail its potential. This paper laid the groundwork for EA’s focus on x-risk.
  2. Existential Risk Prevention as Global Priority (2013) — Makes the economic and moral case that even small probability reductions in existential risk have enormous expected value, central to the longtermism argument.
  3. Public Policy and Superintelligent AI (2018) — Introduces a “vector field” approach to ai-governance, identifying policy directions robust across different superintelligence scenarios.
  4. Future Progress in Artificial Intelligence: A Survey of Expert Opinions — Co-authored with Vincent Muller. 170 AI experts surveyed in 2012-2013; median prediction of 50% chance of HLMI by 2040; one-in-three chance of bad or catastrophic outcomes.
  5. Optimal Timing for Superintelligence (2026) — Argues the baseline is not safe (170,000 die daily), models optimal AGI deployment timing, and concludes “swift to harbor, slow to berth” — move quickly to AGI capability, then pause briefly before full deployment.

Superintelligence: Paths, Dangers, Strategies (2014)

Bostrom’s most widely known work examines what happens when machines surpass human intelligence. The book introduced or popularized several concepts now central to AI safety:

  • Intelligence explosion — A superintelligent AI could rapidly improve itself
  • Orthogonality thesis — Intelligence and goals are independent; a superintelligent AI need not share human values
  • Instrumental convergence — Most goals lead to similar sub-goals (self-preservation, resource acquisition)
  • Treacherous turn — An AI might behave cooperatively until powerful enough not to
  • The control problem — How to maintain meaningful human oversight of a superintelligent system

Institutional Connections

Bostrom’s work connects to the global-priorities-institute at Oxford, where research on existential risk and global priorities is conducted with academic rigor. His papers collectively form much of the philosophical infrastructure of the EA movement’s cause prioritization framework.

Significance for This Wiki

Bostrom is arguably the single most important intellectual figure in the lineage from academic philosophy to practical AI safety concern. His 2002 existential risk paper created the conceptual category, his 2014 book brought AI risk to mainstream attention, and his academic work continues to provide the philosophical grounding for why ai-safety is treated as a priority cause area within effective-altruism. Understanding Bostrom’s arguments is essential for understanding why this wiki’s topics are considered important.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.

Backlinks