AI research capability exceeds the strongest human researchers

Last updated

Tracker summary: An individual Agent-4 copy operating at human speed is qualitatively better at AI research than any human.

No qualifying broad, independently replicated complete-project comparison has been identified.

At a glance

  • Assessment: Not Yet Testable
  • Confidence in assessment: 85%
  • Outcome: unresolved
  • Timing: pending
  • Evidence: insufficient
  • Predicted timing: September 2027
  • Primary source: AI 2027 archived scenario: September 2027: Agent-4, the Superhuman AI Researcher, p. 19; Appendix J - How we’re forecasting the capability progression beyond superhuman coders, p. 54

What AI 2027 Predicted

Tracker summary: An individual Agent-4 copy operating at human speed is qualitatively better at AI research than any human. The source is September 2027: Agent-4, the Superhuman AI Researcher; Appendix J in the AI 2027 scenario.

Scenario source: archived version and page-number reference: September 2027: Agent-4, the Superhuman AI Researcher, p. 19; Appendix J - How we’re forecasting the capability progression beyond superhuman coders, p. 54.

How We Track This

For a strong test, preregister at least six complete research tasks across at least three ML subfields and recruit at least three independently credentialed leading human researchers. Match experimental compute, access and wall-clock budgets; report inference resources separately. Use blinded external judges for validity, novelty and importance, include failed tasks, and require the system to outperform the best participating human on at least four of six tasks without a systematic deficit in one subfield.

These operational thresholds specify this tracker’s test. Where the scenario uses qualitative language, they are interpretations rather than numbers quoted from the original.

Current Evidence

  • August 28, 2026 (first submission; revised September 2): Automated Researchers Can Mitigate Well-Characterized Alignment Failures (dated paper): The best automated methods outperform a 28-researcher human-idea baseline on well-characterized alignment failures, not on broad independently replicated research programs.
  • August 2026: TASTE: Top tested models agree with a constructed expert proposal-ranking reference about 60% of the time versus about 77% for humans, measuring preference agreement rather than objective research success.
  • August 13, 2026: Beyond Final Scores: Across seven frontier models and 36 long-horizon tasks, the authors find practical engineering optimization but unstable performance and rare methodological novelty, rather than broad autonomous research mastery.
  • October 7, 2026: Epoch AI, InnovationEval: After adjusting scope and comparing similar wall-clock times, GPT-5.6 Sol’s in-scope method achieves 15% of the human-developed SDPO method’s improvement over the GRPO baseline; its self-imitation technique resembles prior work rather than independently matching the target innovation.

InnovationEval is direct evidence about method discovery against a human reference on its evaluated task, not merely a generic coding benchmark; a single target cannot establish or exclude superiority to the strongest humans across research fields. The separate AI teams conduct complete research projects with little human direction dossier tracks organizational autonomy, not an individual human-speed copy’s capability.

Counterevidence & Limitations

A finite contest cannot prove “better than any human.” Treat success as strong operational evidence, keeping that limitation in the headline assessment. To mark the broad claim confirmed requires convergent independently replicated evidence across complete research programs, not this threshold alone. Narrative deadline is September; alternate supplement milestones are tracked separately.

Missing qualifying public evidence does not establish that the event has not happened privately. The assessment concerns the documented test and its observable evidence.

What Would Change Our Assessment

Remains Not Yet Testable or moves to Behind after September 2027 if: top humans remain superior on research selection, novel insights, error detection or reproducible complete projects, or apparent AI wins depend on unmatched compute; comparable evidence of such a shortfall supports Behind after the deadline, whereas missing complete-project comparisons retain Not Yet Testable with measurement as the limitation.

Evidence satisfying the full test could strengthen or confirm the claim, including early qualifying results before the scenario window; partial proxies are recorded with their scope. These are conditional triggers, not changes to the current assessment.

Update History

DateUpdate
2026-11-07Assessment reviewed. Status: Not Yet Testable; confidence: 85%. InnovationEval finds limited independent novelty in its tested tasks, and the long-horizon study finds that agents mainly adapt established techniques. Neither provides the matched, complete research-program comparison against leading humans required for this claim. Separated narrowly scoped research-task results from a matched comparison of complete programs against leading humans.
2026-09-06No qualifying broad, independently replicated complete-project comparison has been identified. Relevant primary evidence and limitations are described above.