AI Safety Compendium

AI Control

6 min read

AI Control

Definition

AI control is a safety paradigm — formalized by buck-shlegeris and colleagues at redwood-research in Greenblatt et al. 2023, AI Control: Improving Safety Despite Intentional Subversion — that asks: how do we safely deploy AI systems even if they are actively misaligned and trying to subvert our oversight? Control deliberately assumes the worst case (the model is scheming) and develops protocols that bound the probability of catastrophic outcomes regardless (Greenblatt et al. 2023; 80,000 Hours podcast: Buck Shlegeris on AI control).

It is a complement to ai-alignment, not a replacement. Alignment asks “how do we make the model want what we want?”; control asks “given that we may not know whether it does, how do we still get useful work out of it without enabling catastrophe?” (Greenblatt et al. 2023, §1; Atlas Ch.3 — AGI Safety Strategies).

Why it matters

If deceptive-alignment is real and our interpretability tools cannot reliably rule it out, then any deployment decision based on “the model seems aligned” is fundamentally fragile. Control provides the missing layer: a deployment can be safe by protocol design even when the model’s internal goals are unknown (Greenblatt et al. 2023, §2).

The framing matters most for the rapidly-approaching regime where AI systems are doing their own alignment research at scale — Shlegeris frames this as the “automated AI R&D” period in which “thousands of AI systems running in parallel, each operating roughly 15× faster than a human” do nearly all frontier safety work (80,000 Hours podcast). A single coordinated escape attempt during that window could be catastrophic, but the same scale also creates exploitable structure: cross-checking, redundancy, and protocol-enforced compartmentalization.

Key results

  • The control evaluation methodology (Greenblatt et al. 2023, §3-5). Frame deployment as a red-team / blue-team game: a “scheming” model attempts to subvert the protocol; a “trusted” weaker model + human raters do the monitoring. The protocol is safe if the scheming model cannot reliably succeed at sabotage even with adversarial play. Reduces “is this deployment safe?” to a measurable empirical question.

  • Untrusted-model monitoring works under a coordination assumption (Greenblatt et al. 2023, §6). One untrusted model can monitor another. If the two cannot reliably communicate to coordinate sabotage attempts, the monitor catches misbehavior — even though neither model is trusted. This breaks the apparent paradox of using an unaligned model as a safety mechanism.

  • Compound-probability defense for multi-step attacks (Greenblatt et al. 2023, §6; 80,000 Hours podcast). Sabotage that requires N sequential undetected steps faces a probability of detection that compounds rapidly. Even a per-step detection rate of 5% pushes a 6-step attack toward ~26% total detection probability, dramatically tilting the game toward the defender.

  • Resampling protocols (Ctrl-Z) extend control to interactive agents (Bushnaq et al. 2025, Ctrl-Z: Controlling AI Agents via Resampling). When a monitor flags a suspect action, the protocol resamples from the agent — running multiple draws and aggregating — rather than committing to a single trajectory. Maintains control guarantees while keeping productivity high.

  • Disjunctive monitoring removes the trustedness-threshold problem (Roger 2026, Control protocols don’t always need to know which models are scheming). Standard control assumes you know which models are trusted. Roger shows that ensembling monitors of varying capability and blocking on any positive flag matches the safety of the oracle protocol — without needing to identify the trustedness threshold in advance.

  • Many basic control techniques are cheap to deploy (80,000 Hours podcast: Buck Shlegeris). Shlegeris argues that even “lazy” labs could implement basic monitoring, sandboxing, and red-teaming — the cost is not the bottleneck. This is strategically important because it makes broad adoption realistic on short timescales.

Open questions

  • How do control guarantees degrade when the scheming model is much more capable than the monitor? The compound-probability and untrusted-monitoring arguments rely on the scheming model being roughly comparable to (or weaker than) the trusted-monitor stack. Capability gaps break the assumption (Greenblatt et al. 2023, §7).

  • Can scheming models coordinate via shared training data? The “no coordination” assumption is doing real work. If two untrusted instances share enough pretraining data and structure, they might converge on attack strategies without explicit communication (Roger 2026).

  • What is the right control budget under realistic productivity pressure? Disjunctive monitoring maximizes recall but at a precision cost; productivity loss compounds across the deployment (Roger 2026). The Pareto frontier of safety vs. productivity for realistic deployments is largely uncharacterized.

  • Are control techniques ethically appropriate if AI systems have moral status? Permanently constraining a potentially-conscious system raises real moral questions; Shlegeris frames control as a transitional measure during the alignment-research bootstrap, but the boundary conditions are unsettled (80,000 Hours podcast).

  • How do we audit control protocols at scale? A protocol that works in a lab setting may have failure modes only visible across millions of deployment hours. The “control safety case” methodology is still being developed (takeaways-from-sketching-a-control-safety-case, see takeaways-from-sketching-a-control-safety-case).

  • ai-alignment — control’s complement: alignment makes the model want the right thing, control prevents disaster if it doesn’t.
  • deceptive-alignment — the failure mode that makes control necessary; control is the response to “we may not be able to detect scheming.”
  • scheming — the model behavior control assumes as worst case.
  • scalable-oversight — overlapping but distinct: oversight makes evaluation possible, control makes deployment safe.
  • capability-evaluations — the evidence base control protocols depend on.
  • responsible-scaling-policy — RSPs increasingly specify control thresholds at higher safety levels.
  • interpretability — interpretability could weaken or strengthen the assumptions control protocols rely on.
  • ai-takeover-scenarios — control is specifically designed to prevent the catastrophic-takeover branch of these scenarios.

Sources cited

Primary URLs harvested from this page’s summary references. Auto-generated by scripts/backfill_citations.py; edit by re-running, not by hand.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.

Backlinks