Scientist AI — SR2025 Agenda Snapshot
One-sentence summary: Develop powerful, nonagentic, uncertain world models that accelerate scientific progress while avoiding the risks of agent AIs
Theory of Change
Developing non-agentic ‘Scientist AI’ allows us to: (i) reap the benefits of AI progress while (ii) avoiding the inherent risks of agentic systems. These systems can also (iii) provide a useful guardrail to protect us from unsafe agentic AIs by double-checking actions they propose, and (iv) help us more safely build agentic superintelligent systems.
Broad Approach
cognitivist science
Target Case
pessimistic
Orthodox Problems Addressed
Pivotal processes require dangerous capabilities, Goals misgeneralize out of distribution, Instrumental convergence
Key People
Yoshua Bengio, Younesse Kaddar, and the Scientist-AI team at LawZero (Damiano Fornasiere, Oliver Richardson, Gaël Gendron, Iulian Serban) — LawZero is the institutional home for operationalizing this agenda since June 2025.
Funding
ARIA, Gates Foundation, Future of Life Institute, Coefficient Giving, Jaan Tallinn, Schmidt Sciences
Estimated FTEs: 1-10
Critiques
Hard to find, but see Raymond Douglas’ comment, Karnofsky-Soares discussion. Perhaps also Predict-O-Matic.
See Also
Outputs in 2025
2 item(s) in the review. See the wiki/summaries/ entries with frontmatter agenda: scientist-ai (these were generated alongside this file from the same export).
Source
- Row in
shallow-review-2025/agendas.csv(name = Scientist AI) — Shallow Review of Technical AI Safety 2025.