AI Safety Compendium

AI Red Lines

3 min read

AI Red Lines

AI Red Lines are internationally agreed prohibitions on specific dangerous AI capabilities or deployments — capability classes that no actor should be permitted to develop or deploy regardless of national interest or commercial advantage. They are the international-level analog of if-then-commitments and a flagship initiative of Charbel-Raphaël Ségerie / CeSIA.

The Initiative

The Global Call for AI Red Lines was initiated by Ségerie and endorsed by 12 Nobel laureates, presented at the UN General Assembly and Security Council in September 2025. The call argues that certain AI capabilities are too dangerous to develop regardless of commercial or national-security framing — analogous to existing prohibitions on biological weapons, human cloning, blinding laser weapons.

Examples of Proposed Red Lines

While the specific list varies across proposals, recurring candidate red lines include:

  • Autonomous AI replication — systems making copies of themselves outside intended environments (autonomous-replication)
  • Bioweapon uplift — AI capability sufficient to enable novice bioweapon design
  • Mass-scale persuasion / disinformation — capabilities enabling civilizational-scale manipulation
  • Autonomous cyber-warfare — fully autonomous offensive cyber operations
  • Critical-infrastructure control — AI with autonomous access to power grids, financial systems, weapons
  • AI with personhood-deception capabilities — systems pretending to be human at scale

Why Red Lines Differ from Other Governance

Vs. if-then commitments

  • If-then = “if you build X, you must do Y”
  • Red lines = “you must not build X, full stop”

Red lines are harder commitments — pre-emptive prohibition rather than capability-conditional safeguards.

Vs. comprehensive regulation

EU AI Act regulates broadly across risk tiers. Red lines target a narrow set of catastrophic capabilities with international consensus. Complementary, not competing.

Vs. moratorium

Moratorium halts a development class for a fixed duration. Red lines are permanent prohibitions on specific capabilities.

Governance Function

Red lines serve several functions in the governance architecture:

  • Coordination focal point — gives international governance a concrete agenda beyond general principles
  • Norm-setting — shapes what’s considered acceptable in industry/academia
  • Pre-emptive risk reduction — prohibits capabilities before they become widely available
  • Verification target — easier to monitor “did anyone build X” than “is everyone safe”

Challenges

Definition Difficulty

What exactly does “bioweapon uplift” mean? “Autonomous replication”? Definitions need to be precise enough to be actionable but broad enough to capture relevant variants.

Verification

Detecting violations requires intrusive monitoring — invasive access to model evaluations, training data, capability assessments. Conflicts with corporate proprietary interests and national security claims.

Defection Incentives

Race dynamics (risk-amplifiers) create pressure to defect. Without enforcement, red lines become aspirational.

Definitional Drift

Once a “red line” is crossed by one actor, others have incentive to argue it should be reinterpreted. Each crossing weakens the norm.

Strategic Role

The Atlas treats red lines as one of several international governance instruments alongside Global AI Summits, International Network of AI Safety Institutes, the Hiroshima AI Process, and OECD guidelines. Their distinct contribution: identifying capability classes where consensus prohibition is feasible and bringing political weight to that consensus.

The September 2025 UN presentation marks the initiative’s emergence into formal international politics. Whether it produces binding agreements remains to be seen.

Connection to Wiki

Sources cited

Primary URLs harvested from this page’s summary references. Auto-generated by scripts/backfill_citations.py; edit by re-running, not by hand.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.

Backlinks