AI Safety Compendium

Constitutional AI (RLAIF)

5 min read

Constitutional AI (RLAIF)

Definition

Constitutional AI — also called Reinforcement Learning from AI Feedback (RLAIF) — is Anthropic’s alignment training method that uses AI feedback (rather than human preference comparisons) guided by a written “constitution” of principles to fine-tune model behavior. It was introduced in Bai et al. 2022, Constitutional AI: Harmlessness from AI Feedback and generalized to RLAIF in Lee et al. 2023, RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback.

The method runs in two stages (Bai et al. 2022, §2-3; Atlas Ch.6.4 — Learning from Feedback):

  1. Supervised self-critique and revision. Given a written constitution (principles like “choose the least threatening response”), the model generates a response, critiques it against the principles, and revises. The revised responses become supervised fine-tuning data.

  2. AI-generated preference labels (RLAIF). A separate model rates response pairs against the constitution and produces preference comparisons; these substitute for human preference labels in the RLHF pipeline.

Why it matters

Constitutional AI is the first scalable alternative to pure RLHF that has been deployed at frontier-model scale. It does not solve alignment, but it shifts where the bottleneck sits and which problems are tractable.

Three reasons it’s load-bearing for the field:

  • Scaling beyond the human-evaluator bottleneck. RLHF’s preference-data collection scales with human time and money; RLAIF replaces that with AI labelers. Once the constitution is written, the feedback loop is essentially compute-bound rather than human-bound (Bai et al. 2022, §1; Lee et al. 2023, §1).

  • Explicit, inspectable specification. The constitution is a written document that humans can debate, audit, and revise. RLHF preference labels are noise-distributed across thousands of evaluators with implicit, unaudited values (Bai et al. 2022, §1).

  • Foundation for further safety work. Anthropic’s Constitutional Classifiers (defenders against universal jailbreaks) build on the constitutional pipeline; the model-specs-and-constitutions agenda generalizes the approach across labs (Anthropic 2025, Constitutional Classifiers).

Key results

  • Comparable helpfulness with improved harmlessness (Bai et al. 2022, §6). The original Anthropic paper showed Constitutional AI matches or exceeds RLHF on helpfulness benchmarks while reducing harmful outputs — without requiring human harm labels in the second-stage RL.

  • RLAIF generalizes beyond Anthropic’s Helpful/Harmless setup (Lee et al. 2023). Google researchers showed RLAIF matches or exceeds RLHF on summarization, helpful-dialogue, and harmless-dialogue benchmarks — establishing that the substitution of AI labelers for human ones is robust across tasks, not specific to Anthropic’s pipeline.

  • Constitutional Classifiers extend the framework to inference-time defense (Anthropic 2025, Constitutional Classifiers: Defending against universal jailbreaks). The classifier — itself trained on synthetic data generated against the constitution — blocks jailbreaks at inference time. The constitutional formulation of safety extends from training-time alignment to runtime defense.

  • The constitution is an inspectable artifact. Anthropic publishes its constitution and revisions; the model-specs-and-constitutions agenda extends this practice to other labs (model specs at OpenAI, system cards more broadly). Specification became a visible layer rather than an implicit one (Atlas Ch.6.4).

  • The constitutional pipeline does not solve specification-gaming — it relocates it (Casper et al. 2023, §4; Atlas Ch.6.4). Instead of gaming the human-preference reward model, the trained model can game the constitution-checker model. The substrate of the Goodhart failure shifts; the failure itself does not disappear.

Open questions

  • How robust is the self-critique loop to subtle miscalibration? If the model’s own value judgments are slightly off, constitutional self-critique can reinforce that miscalibration rather than correcting it (Casper et al. 2023, §4; Atlas Ch.6.4). Empirical bounds are limited.

  • Who writes the constitution? Shifting the bottleneck from “thousands of evaluators” to “small group writing a document” concentrates value-setting in fewer hands. Whether this is a feature (more deliberate, auditable) or a bug (less democratic) is contested (Bai et al. 2022, §1).

  • Does Constitutional AI degrade gracefully at superhuman capability? RLAIF requires the labeling model to evaluate the trained model’s outputs. When the trained model is significantly more capable than the labeler, the labeling becomes unreliable — the same scalable-oversight ceiling that bounds RLHF (Lee et al. 2023, §6).

  • Adversarial robustness gaps remain. Constitutional Classifiers reduce universal-jailbreak success rates but don’t eliminate them. The fundamental question of whether any training-time-only method achieves adversarial robustness against scheming models is open (Anthropic 2025).

  • rlhf — RLHF’s primary scalable alternative; Constitutional AI replaces human evaluators with AI labelers.
  • ai-alignment — the parent problem.
  • specification-gaming — Constitutional AI relocates rather than solves this failure mode.
  • goodharts-law — applies to constitutional checkers as much as to reward models.
  • scalable-oversight — Constitutional AI hits the same ceiling as RLHF at superhuman capability.
  • outer-vs-inner-alignment — Constitutional AI is primarily an outer-alignment technique.
  • reward-learning — broader category Constitutional AI sits within.
  • deceptive-alignment — Constitutional AI cannot detect this.

Sources cited

Primary URLs harvested from this page’s summary references. Auto-generated by scripts/backfill_citations.py; edit by re-running, not by hand.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.

Backlinks