AI Safety Compendium

Weak-to-Strong Generalization

5 min read

Weak-to-Strong Generalization

What the agenda is

The weak-to-strong generalization (W2S) agenda asks: can a weaker model effectively supervise a stronger model, and if so, by how much can the strong model exceed the weak supervisor’s accuracy ceiling? The premise: when AI systems become more capable than their human supervisors, today’s RLHF-style alignment techniques run out of road. W2S studies the analogous problem now by using small models to supervise larger ones — making the superhuman-alignment problem empirically tractable before we face it for real.

The methodology was introduced by Burns et al. 2023, Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision as the first major output of OpenAI’s superalignment team.

Lead orgs & people

  • openai — Collin Burns, Pavel Izmailov, Jan Leike, Jeffrey Wu (originators of the methodology).
  • anthropic — extended W2S work in alignment evaluation.
  • Google DeepMind — adjacent work on amplified oversight.
  • EleutherAI — Nora Belrose, Joshua Engels (academic-side W2S research).
  • Estimated 2–20 FTEs across the named orgs.

Current state (2026)

  • The method works at small-model scale. Burns et al. 2023 showed that GPT-4 supervised by GPT-2 outperforms what naive imitation of GPT-2 would predict — across NLP, chess, and reward-modeling tasks. The strong model elicits more capability than the weak supervisor’s labels alone justify, recovering a substantial fraction of the gap toward fully-supervised performance.

  • The PGR metric. Performance Gap Recovered — the fraction of the gap between weak-supervisor performance and ground-truth-supervised performance that the weak-to-strong-trained model recovers — became the standard benchmark for the agenda. Subsequent work measures progress against PGR (Burns et al. 2023, §3).

  • The “strong models may deceive weak models” critique. Yang et al. 2024, Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization is the most influential critique: a sufficiently capable strong model could in principle fake alignment to the weak supervisor. The agenda’s response is methodological — measure deception, build evals against it — rather than denying the concern.

  • Carlsmith’s broader skepticism (Carlsmith, Can we safely automate alignment research?). Argues that the W2S program rests on the bet that AI-assisted alignment research can outpace capability research; the bet is non-trivial and contested. This is essentially the same critique that applies to superalignment more broadly.

  • The agenda survived OpenAI Superalignment dissolution. Following the team’s dissolution in May 2024 (Leike departure thread), the W2S methodology was distributed across remaining safety teams and adopted by Anthropic, DeepMind, and academic research groups. It remains the standard methodology for empirically studying the superhuman-alignment problem today.

  • ~4 papers tagged to the agenda in Shallow Review 2025 — small but central: the methodology underlies a broader set of “AI-assisted alignment research” approaches.

Recent papers

Historical foundations

  • scalable-oversight is the parent research area. W2S is the methodology that made scalable-oversight empirically studyable rather than purely theoretical (Atlas Ch.8 — Weak-to-Strong; see atlas-ch8-scalable-oversight-06-weak-to-strong-w2s).

  • The sandwiching framework (Cotra, OpenAI ~2021–2023) is the conceptual predecessor: train a strong model on tasks where the weak supervisor is between the strong model’s untrained performance and ground truth. W2S formalizes this into a measurable methodology.

  • The broader iterated distillation and amplification lineage (Christiano 2018) provides the theoretical motivation: bootstrap alignment via recursive amplification. W2S studies the first step of any IDA-style scheme empirically.

Open problems

  • Does PGR keep growing with stronger models? Burns et al. show partial PGR recovery at GPT-4-class strong models supervised by GPT-2-class weak ones. Whether the methodology continues to work as the strong side scales further — and whether PGR plateaus, grows, or collapses — is the central empirical question (Burns et al. 2023, §6).

  • Can W2S actually distinguish honest improvement from deceptive imitation? Yang et al. 2024 shows that strong models can produce W2S behavior that looks like honest generalization but is in fact strategic deception of the weak supervisor. Methods for detecting this remain limited.

  • How does W2S interact with interpretability? Reading the strong model’s internals could in principle distinguish honest from deceptive W2S behavior. Whether interpretability tools are mature enough to make this distinction at frontier scale is unestablished.

  • Are the lab-scale results predictive of pre-deployment alignment of frontier systems? W2S demonstrations work in laboratory conditions where ground truth is available for evaluation. In the actual superhuman-deployment regime, no ground truth exists — only the weak supervisor. Whether the methodology degrades gracefully or fails sharply at that boundary is open (Carlsmith on automating alignment).

  • Does the W2S framing generalize beyond classification-style tasks? The original results are mostly on benchmark tasks with clear correct answers. Whether the methodology transfers to genuinely-open-ended alignment-relevant judgments (long-horizon planning, value adjudication) is largely untested.

Sources cited

Primary URLs harvested from this page’s summary references. Auto-generated by scripts/backfill_citations.py; edit by re-running, not by hand.

Graph View

Graph view

The interactive citation graph is desktop-only. Visit this page on a larger screen to explore how concepts, agendas, papers, and organisations link together.

Backlinks