RE-Bench score reaches 1.3

Last updated

Tracker summary: 1.3 on RE-Bench matching top expert humans given 8 hours at well-defined AI research engineering tasks.

At a glance

  • Assessment: Confirmed
  • Confidence in assessment: 70%
  • Outcome: observed
  • Timing: early
  • Evidence: direct
  • Predicted timing: Early 2026
  • Primary source: AI 2027 archived scenario: Early 2026: Coding Automation, p. 6

What AI 2027 Predicted

Tracker summary: By early 2026, AI agents would reach a normalized score of 1.3 on METR’s RE-Bench, a suite of seven research-engineering tasks. The scenario describes an eight-hour human-expert reference; it does not impose an eight-hour cap on the agent’s run. A METR time horizon is a different measurement, not a substitute for this score.

The scenario places this prediction in “Coding Automation”. Archived PDF, p. 6.

How We Track This

Compare reported RE-Bench normalized scores with the 1.3 threshold. Identify the task coverage, scoring method, human reference and agent budget, and distinguish developer reports from independent replication. Use the date by which a result was reported separately from its exact evaluation date or the first-ever crossing.

The original RE-Bench paper supplies the benchmark reference. Subset evaluations and time-horizon estimates can add context but cannot be substituted for the specified aggregate score.

Current Evidence

Confirmed (70%), timing early, on developer-reported evidence: Intology’s November 19, 2025 Locus report reports RE-Bench 1.30 from a continuous 64-hour agent run. This supports the threshold being reached by that publication date, before early 2026. It is not independently replicated, and neither the exact evaluation date nor the first-ever achievement date is disclosed.

Intology says it follows the original benchmark methodology, checks top solutions for evaluation hacking and prevents changes to the scoring code. Its chart reports Locus at 0.70 after eight hours and 1.30 after 64 hours. The longer run counts against the numeric target: the scenario’s eight hours describes the human reference, not an agent-time limit. Intology’s own 64-hour human comparison is 1.27; do not conflate that comparison with the scenario’s reference description.

The report presents an aggregate RE-Bench result without identifying a reduced task subset, but does not publish a complete per-task manifest, run-level logs or an explicit normalization formula. These limitations hold confidence at 70%; they do not turn the reported early result into a late or missing result. The page’s August 3, 2026 update points to later post-training work, not a new date for the original RE-Bench score.

METR’s April 4, 2025 Claude 3.7 evaluation covers five of seven tasks. The TH1.1 time-horizon data measure a different quantity. Neither independently verifies the Locus score, nor negates it.

Counterevidence & Limitations

  • The reported 1.30 is direct evidence about the numeric target, but remains a developer report rather than independent METR validation.
  • Intology’s external-verification statement concerns its kernel results, not the RE-Bench run.
  • Seven structured research-engineering tasks do not establish broad autonomous research ability or generic long-horizon reliability.
  • A complete task manifest, per-task scores and reproducible scoring would improve confidence in comparability. No exact first-crossing date is established.

What Would Change Our Assessment

  • Strengthen confidence: Independent replication or full task-level records reproduce a comparable score of at least 1.3.
  • Reconsider confirmation: Evidence of incompatible scoring, omitted tasks, contamination or invalid evaluation undermines the reported threshold result.
  • Refine timing: A dated evaluation record could establish when the reported run occurred; discovery of the report in 2026 does not make a result reported in 2025 late.

Update History

DateUpdate
2026-11-092026-11-09 · Correction: status Behind (70%) → Confirmed (70%). Intology reported Locus at RE-Bench 1.30 on November 19, 2025, before the early-2026 target. The 64-hour agent run is not excluded by the scenario’s eight-hour human reference. Outcome observed, timing early, on developer-reported evidence rather than independent replication.
2026-11-07Evidence reviewed: METR’s April 4, 2025 evaluation covers five of seven RE-Bench tasks with 32-hour agent and 8-hour human budgets. It cannot establish a current, full-suite 1.3 score or a measured shortfall against that threshold. Correction: the subset result matches rather than surpasses the human reference. The later productivity study did not invalidate the earlier 19% slowdown. Time horizons were published in March 2025 and are not normalized RE-Bench scores; the dated historical corrections remain with those entries.
2026-09-06Current evidence and assessment explanation reviewed. Clarified measurement scope, source interpretation and limitations.
2026-06-06Corrected the Opus 4.6 METR time-horizon reference to current TH1.1 raw data: roughly 11h59m at 50% reliability and roughly 1h10m at 80% reliability. This weakens earlier wording that implied the 8-hour RE-Bench reference had been approximately vindicated by time-horizon data. Status changed from on-track to behind and confidence raised from 0.60 to 0.70 because no published 1.3+ RE-Bench score has been verified.
2026-03The improvement from Opus 4.5 (~4h49m) to Opus 4.6 on METR time horizons suggested the 1.3 score might come within reach. Key uncertainty: whether time-horizon metrics and RE-Bench V1 numeric scores map cleanly onto each other. Later correction: the current Opus 4.6 50% horizon is roughly 11h59m, while the 80% horizon is much shorter.
2026-02Claude Opus 4.6 initially appeared to surpass the 8-hour human-expert reference level on the 50% time-horizon metric, suggesting possible progress toward RE-Bench-style research engineering. Later correction: this did not establish reliable 8-hour autonomous performance, and no published 1.3+ RE-Bench score was available. METR announces redesign of developer productivity study (Feb 24), noting developers now refuse to work without AI, invalidating earlier “19% slower” finding. Status revised from Behind to On Track. Confidence 0.60. 2026-11-07 · Correction: METR’s February 24, 2026 update did not invalidate the earlier 19% slowdown result. Selection effects make the newer experiment only weak evidence about the size of improvement; the earlier controlled result remains specific to its original population and period. The historical Behind-to-On Track transition is unchanged by this correction.
2026-01METR publishes Time Horizon 1.1 (Jan 29). Claude Opus 4.5 measured at ~4 hours 49 minutes — a substantial jump suggesting rapid progress toward the 8-hour RE-Bench reference level. External forecasters had predicted RE-Bench 1.0 in 2027; this pace suggests that was too conservative.
2025-12METR begins publishing time-horizon metric — measuring how long a task an AI can autonomously complete at human-expert level. Reframes the RE-Bench question from a single score to “how many hours of autonomous work can an AI sustain?” 2026-11-07 · Correction: METR had already published the time-horizon metric in March 2025; the original paper was submitted March 18, 2025, not December. Time horizons remain distinct from RE-Bench normalized scores.
2025-09METR evaluates Claude 3.7 Sonnet on RE-Bench (preliminary). Results described as “impressive” on a subset of tasks. Exact numeric score not widely published, but qualitative signal is positive.
2025-07METR publishes study on early-2025 AI developer productivity (July 10). Finding: experienced developers were 19% slower with AI tools in a controlled trial — contradicting self-reported 20% speedups. Raises questions about AI’s ability to accelerate sustained research work.
2025-05RE-Bench V1 baseline established (published late 2024 by METR). Claude 3.5 Sonnet and o1-preview score ~0.5-0.8 at 8-hour human-expert reference. Models outperform humans ~4x at 2-hour budget but plateau or regress on longer tasks. Target of 1.3 appears distant.