Leading humans still add substantial value to AI-led research

Last updated

Tracker summary: Most humans contribute little, but the strongest retain research taste and planning value; the appendix estimates removing humans would slow research by roughly half.

The marginal contribution of humans to whole-project research speed remains unmeasured under this test.

At a glance

  • Assessment: Not Yet Testable
  • Confidence in assessment: 80%
  • Outcome: unresolved
  • Timing: pending
  • Evidence: insufficient
  • Predicted timing: June 2027
  • Primary source: AI 2027 archived scenario: June 2027: Self-improving AI, pp. 14, 15; Appendix I - Managing a corporation of AIs, p. 53

What AI 2027 Predicted

Tracker summary: Most humans contribute little, but the strongest retain research taste and planning value; the appendix estimates removing humans would slow research by roughly half. The source is June 2027: Self-improving AI; Appendix I in the AI 2027 scenario.

Scenario source: archived version and page-number reference: June 2027: Self-improving AI, pp. 14, 15; Appendix I - Managing a corporation of AIs, p. 53.

How We Track This

Compare matched AI-led complete projects with versus without expert human input over at least three projects, holding model, tools and experiment compute comparable. Primary measure: accepted progress per calendar time. A 40-60% reduction on removal operationalizes the “roughly half” clause; report the estimate and interval, intervention logs and the distribution across experts. Separately assess whether most participating humans add little value.

These operational thresholds specify this tracker’s test. Where the scenario uses qualitative language, they are interpretations rather than numbers quoted from the original.

Current Evidence

  • August 2026: TASTE: The strongest tested models’ proposal rankings agree with the constructed expert-preference reference only about 60% of the time, versus about 77% for the human baseline; this tests proposal judgment, not marginal whole-project speed.
  • August 28, 2026 (paper first submitted; revised September 2): Automated Researchers Can Mitigate Well-Characterized Alignment Failures (dated paper): The best automated methods outperform ideas from 28 experienced researchers on well-characterized alignment failures, and human seed ideas do not improve the automated researchers’ performance, a constrained result rather than evidence that humans add no value across research.
  • Data as of August 2026: Anthropic’s internal AI-development measurements: Claude leads 26% of measured R&D work and no measured subset is fully autonomous, indicating continued human direction rather than a measured expert-removal speed effect.
  • August 2026: Anthropic’s redacted risk report: The lab judges its Mythos-class models not close to replacing its full research-scientist and research-engineer staff; this is a first-party assessment, not a controlled ablation.

All four observations precede June 2027 and none estimates the change in total accepted research progress per calendar time when experts are removed from matched AI-led projects. TASTE’s reference is human preference, not objective research success; the lab reports do not measure what fraction of individual humans adds value.

Counterevidence & Limitations

The 40-60% band is a proposed interpretation. Evidence that one human helps is not evidence that most do not. Keep marginal contribution separate from the total R&D multiplier and from fully autonomous project completion.

Missing qualifying public evidence does not establish that the event has not happened privately. The assessment concerns the documented test and its observable evidence.

What Would Change Our Assessment

  • Behind after June 2027: humans remain indispensable across the workflow, rather than adding mainly narrow research-taste or planning value, in comparable full-project evidence.
  • Not supported: removing experts has little effect on matched AI-led projects, with enough precision to rule out the proposed 40-60% speed reduction.
  • Strengthen or confirm: Replicated matched-project comparisons satisfy the specified expert-removal effect and separately show that most participating humans add little marginal value.

Missing comparable ablations retain measurement as the limitation; these conditional triggers do not change the current status or confidence.

Update History

DateUpdate
2026-11-07Assessment reviewed. Status: Not Yet Testable; confidence: 80%. Anthropic’s development study reports Claude leading 26% of measured R&D work with no fully autonomous measured subset, while TASTE finds an expert-judgment gap. Automated alignment experiments also show bounded gains without benefit from human seed ideas. None measures removal of experts across matched complete AI-led projects. The 40-60% whole-project speed effect and how many humans add value remain unresolved.
2026-11-07Correction: Labeled the scenario account as a summary and supplied its June/Appendix I archived locators. Dated the TASTE and automated-research studies and added Anthropic’s measured automation and staff-substitution assessments. Clarified the expert-removal triggers: 40-60% is the tracker’s operational band for the scenario’s roughly-half claim, not a quoted source range. The whole-project human-removal speed effect remains unmeasured.
2026-09-06The marginal contribution of humans to whole-project research speed remains unmeasured under this test. Relevant primary evidence and limitations are described above.