Research Note: Our scheming precursor evals had limited predictive power for our in-context scheming evals
Marius Hobbhahn — 2025-07-03 — Apollo Research — Apollo Research Blog
Summary
Empirical analysis testing whether Apollo’s precursor evaluations (agentic self-reasoning and theory of mind) from May 2024 successfully predicted their in-context scheming evaluations from December 2024, finding limited predictive power especially for harder difficulty levels.
Key Result
Precursor evaluations had low to medium predictive power for scheming evaluations (R-squared values mostly <0.3), with easy versions somewhat useful but hard versions neutral or actively misleading, indicating insufficient reliability for high-stakes safety policies.
Source
- Link: https://www.apolloresearch.ai/blog/research-note-our-scheming-precursor-evals-had-limited-predictive-power-for-our-in-context-scheming-evals
- Listed in the Shallow Review of Technical AI Safety 2025 under 1 agenda(s):
- capability-evals — Evals