AI Safety Compendium

Summary: Public Policy and Superintelligent AI -- A Vector Field Approach

6 min read

Summary: Public Policy and Superintelligent AI — A Vector Field Approach

Authors: Nick Bostrom, Allan Dafoe, Carrick Flynn Year: 2018 (version 4.3; first version 2016) Source: bostrom-ai-policy.pdf Published in: Liao, S. M. (ed.), Ethics of Artificial Intelligence (Oxford University Press, 2020)

Main Argument

This paper analyzes the governance challenges that would arise from the development of machine superintelligence and derives a set of policy desiderata — properties that any good governance framework should have. Rather than arguing for one specific set of policies or one particular ethical framework, the authors introduce a novel “vector field” approach: they identify special circumstances unique to the superintelligence context and show how those circumstances should push policy in certain directions regardless of one’s starting normative position.

The metaphor is a vector field in physics: at each point in the space of possible policy positions, the special circumstances of superintelligent AI create a directional force. Whether you start from a libertarian baseline, a social-democratic baseline, or a utilitarian baseline, the same circumstances push your preferred policy in identifiable directions. This makes the analysis relevant to a wide range of actors with different values and ideologies.

The Special Circumstances

The paper identifies several features that make the superintelligence governance context distinctive:

  1. Enormous technological opportunity — Superintelligent AI could expand the production-possibility frontier far more rapidly and extensively than any previous technology, enabling both “outward” growth (space colonization) and “inward” growth (health, wellbeing, cognitive enhancement). This ceiling for growth makes it especially important that potential is not squandered.

  2. Existential-level AI risk — The probability of AI-induced destruction, including human extinction, is plausibly non-negligible. This makes ai-safety research and cautious development especially important as policy objectives.

  3. Catastrophic coordination failures — Superintelligence could enable accessible “doomsday devices,” trigger a risk-increasing AI race dynamic, or create “races to the bottom” in deployment standards. Without effective global coordination, these dynamics could prove fatal.

  4. Extreme turbulence — The speed and magnitude of a machine intelligence revolution would stress existing institutions, potentially causing ill-conceived regulation, fraying international agreements, and increased risk of conflict.

The Desiderata

The authors organize their policy desiderata under four headings:

Efficiency Desiderata

  • Expeditious progress — Policies should (a) lead with high probability to the eventual development of safe superintelligence and (b) enable speedy AI progress so benefits arrive in a timely fashion.
  • AI safety — Techniques must be developed to ensure superintelligent AI behaves as intended (ai-alignment), and conditions during emergence should encourage cautious deployment.
  • Conditional stabilization — If catastrophic global coordination failure would otherwise result, drastic stabilizing measures (potentially including a singleton, intensive surveillance, or suppression of dangerous technology) should be feasible and undertaken in time.
  • Non-turbulence — The transition should minimize efficiency losses from chaos and conflict; political systems should maintain stability and adapt to rapid change.

Allocation Desiderata

  • Universal benefit — Everyone alive at the time of the transition should receive some share of benefits, as compensation for the existential-risk externality to which they are exposed. The paper articulates a general Risk Compensation Principle: those exposed to risk from another’s activities should be compensated.
  • Epsilon-magnanimity — In a post-superintelligence cornucopia, a wide range of resource-satiable values should be realized using a minute fraction of total resources. The key is ensuring the floor of generosity stays above zero. This amounts to a weak but crucial form of practical value pluralism.
  • Continuity — The transition should maintain sufficient order and institutional stability to prevent unnecessary concentration of wealth and power and to honor existing social contracts.

Population Desiderata

  • Mind crime prevention — Advanced AI governance must prevent or minimize maltreatment of sentient digital minds. The paper notes that mind crime could occur inadvertently (e.g., during machine learning training procedures) and at massive scale, since digital minds can be copied cheaply and run at high speed.
  • Population policy — Procreative choices about new digital beings must be coordinated to avoid Malthusian outcomes and political erosion (e.g., copying digital voters to swamp electoral outcomes).

Process Desiderata

  • Informed decision-making — The novelty, depth, and technical complexity of the context demands governance processes with genuine understanding of AI technology, not just reliance on opaque expert edicts. The paper calls for wisdom — the ability to reliably get the most important things at least approximately right.
  • Speed and decisiveness — Events may unfold at an unusually fast pace, requiring governance processes that can move quickly and make binding decisions in advance of crises.
  • Legitimacy and representation — Despite the need for speed and technical expertise, governance processes should maintain legitimacy and represent the interests of all affected parties, including future generations.

Methodological Contribution

The vector field approach itself is a significant methodological innovation for normative policy analysis. By focusing on directional change rather than absolute positions, the paper avoids the usual impasse where competing ethical theories recommend different policies. Instead, it identifies a set of considerations on which diverse perspectives converge: nearly everyone, regardless of their starting ideology, should want more AI safety, more global coordination, more benefit-sharing, and more protection for digital minds than they would want in a non-superintelligence context.

Key Insights on Distribution and Power

The paper’s analysis of allocation is especially prescient. It identifies two salient dynamics:

  • Concentration — Income and power could become dramatically more concentrated if capital (including AI capital) earns most of the returns and labor’s share collapses. In the limit, one entity controlling the first superintelligent AI could gain a decisive strategic advantage.
  • Permutation — The ranking of actors by wealth and power could be radically reshuffled by unpredictable technological churn, creating winners and losers in ways that current diversification strategies cannot hedge against.

These dynamics create a veil-of-ignorance situation in which insurance-like redistribution schemes become prudentially attractive to most actors, even self-interested ones.

Significance

This paper is a cornerstone of the ai-governance literature. It moves beyond abstract philosophical arguments about whether AI risk matters (which Bostrom had already established in earlier work) to the practical question of what governance arrangements should look like. The vector field approach has been influential because it offers actionable guidance without requiring consensus on foundational ethical questions.

The paper’s desiderata — particularly AI safety, conditional stabilization, universal benefit, and mind crime prevention — have shaped the agenda of organizations working on AI governance, including the future-of-humanity-institute and governance programs at openai, anthropic, and deepmind. Its analysis of race dynamics and coordination failures anticipated much of the real-world policy debate that has emerged around frontier AI development.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.