AI Safety Compendium

AI Safety Atlas Ch.2 — Systemic Risks

5 min read

AI Safety Atlas Ch.2 — Systemic Risks

Source: Systemic Risks

Systemic risks emerge from interactions between AI systems and society, not from individual AI failures. Even AI systems working exactly as designed can collectively produce harmful outcomes when integrated with markets, democratic institutions, and social networks. Parallel to financial crises — collective behavior of many institutions creates systemic threats no single entity decided to produce. See the systemic-risks concept page for the consolidated treatment.

Properties of Complex Systems

Five properties of complex systems that produce systemic AI risks:

  • Emergence — behaviors unpredictable from analyzing components in isolation
  • Feedback loops — amplify changes into self-reinforcing cycles (engagement-optimizing AI gradually pushing users toward extreme content)
  • Non-linearity — small changes produce disproportionately large effects
  • Self-organization — multiple AI systems optimizing independently can spontaneously organize into unintended patterns (algorithmic traders creating dynamics no single actor controls)
  • Agent-agnosticism — “even perfectly aligned AI systems could collectively produce harmful outcomes.” This is the critical distinction from misalignment risk.

Two Pathways: Decisive vs. Accumulative

Decisive Systemic Risks

Interconnected systems reach critical thresholds → rapid collapse with cascading effects faster than humans can respond. Reference point: 2010 financial flash crash — algorithmic traders’ self-reinforcing reactions caused a trillion-dollar market drop in minutes before human intervention restored stability.

Accumulative Systemic Risks

Gradual disempowerment through several mechanisms — the bulk of this subchapter. The Atlas details five.

Five Accumulative Mechanisms

1. Epistemic Erosion

Society’s ability to distinguish fact from fiction deteriorates as AI-generated content floods information ecosystems. Volume overwhelms verification; authenticity degradation undermines verification mechanisms; epistemic learned helplessness sets in as truth-seeking appears futile.

Concrete: “what fraction of new images indexed by Google… are generated by humans? Nobody knows.” Verification mechanisms (fact-checking, peer review, credentialing) operate at speeds mismatched to AI generation.

Reality fragmentation through AI personalization: “in widespread persuasive AI environments, people’s beliefs become determined by which systems they interact with most.” Erodes consensus reality, cooperation, civic participation, and collective problem-solving — including the conversation about AI risks themselves. See epistemic-erosion.

2. Power Concentration

“Five companies currently control foundation models increasingly mediating human experiences.” As of 2025, only US and China host foundation-model-training-capable companies.

Two pathways:

  • Corporate concentration — self-reinforcing data and compute advantages; cloud computing consolidated; foundation-model development centralized.
  • State concentration — AI surveillance and automated governance. Social credit systems exemplify; democratic governments increasingly invest in AI surveillance; administrative automation removes human discretion.

Empirical: facial-recognition firms receiving government surveillance contracts increase total software production 48.6% within two years and become 3× more likely to export internationally — “politically-motivated procurement drives technological advancement.”

Why this matters for moral progress: historical moral improvements (slavery abolition, women’s suffrage, environmental protection) all required shifting existing power structures. AI-enabled power concentration threatens systems resistant to these change mechanisms. Connects to stable-totalitarianism and value-lock-in.

3. Mass Unemployment

Once AI performs 30–40% of economically valuable tasks, annual growth could exceed 20% — primarily benefiting capital owners rather than workers. Wages could crash below subsistence: roughly 33% chance within 20 years, 67% within a century. Even partial automation of remote work (~34% of current job tasks) could double or 10× economies while leaving most humans economically marginalized.

Economic disempowerment becomes the pathway to broader human disempowerment. As humans lose economic leverage, they lose political and social influence. “AI owner economic power concentration could translate into concentrated political power, creating feedback loops where human interests become progressively irrelevant.” See mass-unemployment.

4. Value Lock-in

If AIs become deeply embedded into society and highly persuasive, current values may become entrenched and resist change. “AI systems trained in the 1960s would perpetuate moral defects people then accepted.” Future generations will reject moral views held today, just as we reject many past widely-held views. Locking in current values would curtail moral progress permanently. Substantial overlap with the wiki’s existing value-lock-in page.

5. Enfeeblement

Gradual human capability and agency erosion through AI overdependence. Unlike sudden control loss, enfeeblement unfolds through countless small individually-rational delegation choices.

Mechanism cluster:

  • Overreliance — humans trust AI beyond actual capabilities (chatbot therapy in mental health crises)
  • Trust miscalibration — emotional attachment to AI undermines normal skepticism, becomes manipulation vector
  • Cognitive atrophy — GPS analogy: spatial reasoning declined when navigation became delegated; broader analogous decline likely as AI handles more
  • Social isolation — AI-mediated relationships optimize satisfaction without genuine reciprocity, undermining social competence
  • Organizational automation — companies delegate hiring, lending, medical, legal decisions to AI; individuals lose advocates exercising human judgment

Self-reinforcing path dependence: each delegation choice makes independent action slightly more difficult. Society may reach points where reversal is practically impossible even when risks become apparent. See enfeeblement.

Connection to Wiki

This subchapter is the most novel for the wiki — the systemic-risk frame partially exists in ai-population-explosion and stable-totalitarianism but the Atlas adds:

  • Agent-agnosticism — explicit decoupling from misalignment
  • Decisive vs. accumulative as a structural distinction
  • Five-mechanism taxonomy for accumulative risks
  • Wage-collapse probability estimates (33%/67%)
  • Enfeeblement as a distinct mechanism — not previously a wiki page

The new concept pages systemic-risks, epistemic-erosion, mass-unemployment, enfeeblement all derive from this subchapter.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.

Backlinks