Agents struggle with long-horizon tasks

Last updated

Tracker summary: Agent-1 is bad at even simple long-horizon tasks (page 6, Early 2026 section). Also: agents in Mid 2025 are 'impressive in theory but in practice unreliable.'

At a glance

  • Assessment: Confirmed
  • Confidence in assessment: 85%
  • Predicted timing: 2025 and early 2026
  • Primary source: AI 2027 archived scenario: Mid 2025: Stumbling Agents, p. 3; Early 2026: Coding Automation, p. 6

Assessment scope: Confirmed (85%) is retained for the historical pattern of long-horizon struggle. This page covers both 2025 unreliability and the early-2026 Agent-1 weakness, not only 2025. Early-2026 evidence is partial and task-specific; the unfamiliar-video-game example is not established by the cited evidence.

What AI 2027 Predicted

Tracker summary: The scenario predicts that while agents become useful for short tasks, they struggle with sustained, long-horizon work: tasks that require hours or days of coherent effort. This is framed as a temporary limitation that improves rapidly.

Scenario source: archived version and page-number reference: Mid 2025: Stumbling Agents, p. 3; Early 2026: Coding Automation, p. 6.

How We Track This

We monitor:

  • METR time horizon benchmarks (50% and 80% thresholds)
  • Reports on agent task completion rates for multi-hour workflows
  • Enterprise adoption patterns for long-running agent tasks
  • Academic research on agent planning and coherence over time

Current Evidence

Two historical windows: assessment on November 9, 2026

Retained Confirmed (85%) for historical long-horizon struggle, with an explicit partial early-2026 result. The 2025 unreliability observation remains supported. For early 2026, METR’s dated primary results for Claude Opus 4.6 show roughly 12 hours at 50% success but about 70 minutes at 80%. This supports continuing reliability limits on the evaluated tasks alongside major progress, not blanket inability on simple tasks. It does not test the scenario’s example of beating video games the model has not previously played. No qualifying early-2026 unfamiliar-game result is established by the evidence assembled here. The October 14 SpireClimb results and the later supervised engineering run below are relevant counterevidence to a timeless struggle claim, but do not resolve that earlier unfamiliar-game component. Both historical windows stay visible; neither a 2025-only freeze nor later success is a substitute for the early-2026 assessment.

The 2025 unreliability component is supported; the early-2026 component requires its own assessment. METR’s dated Time Horizon 1.1 results give Claude Opus 4.6 a 50% time horizon of roughly 11h59m and an 80% time horizon of roughly 1h10m. These results leave reliable full-work-day autonomy unproven. They show that structured software/ML/cybersecurity tasks requiring many human-hours are increasingly within reach at lower reliability. Published evaluations include failures on multi-day open-ended projects, without establishing a universal failure rate, and METR cautions that time horizon is a task-difficulty measure rather than the amount of time an AI can act autonomously.

Sources:

The later Claude Mythos Preview (early), April 7, 2026 entry has a 50% horizon of about 17.4 hours and an 80% horizon of about 3.1 hours in METR’s primary results. METR’s page cautions that estimates above 16 hours are unreliable and says the page is no longer updated. These dated measurements are not a claim to cover the newest deployed models.

Counterevidence & Limitations

  • NVIDIA’s AVO system ran a supervised seven-day attention-kernel search, explored more than 500 directions, and committed 40 versions. NVIDIA reported kernels up to 10.5% faster than FlashAttention-4 on evaluated DGX B200 configurations. This shows that persistent memory, supervision, and tool use can support productive week-long execution in a narrow engineering domain. It does not establish broad reliability across open-ended tasks or unsupervised deployment.
  • Epoch’s October 14 SpireClimb evaluation finds frontier models win Slay the Spire 1 at least 90% of the time at base difficulty through visual computer use. At Ascension 20 Heart, GPT-6 Astra wins 17% and GPT-6.1 Sol 20%. The single-agent harness permits notes and wiki access, with fixed arbitrarily chosen seeds and 15 single-shot attempts per character. Prior game knowledge and possible game-specific training limit generalization. This adds a strategic-task success case alongside persistent maximum-difficulty failures; it does not measure METR task horizons or reliable multi-day engineering.
  • METR tasks are unusually well-specified and evaluable compared with much real work; the suite is not entirely synthetic.
  • Improvements in time horizon don’t necessarily translate to reliability on diverse, open-ended tasks: agents may improve on structured tasks while still failing on ambiguous real-world problems
  • Some companies report agent failures on production workloads even when benchmarks improve, suggesting a gap between controlled and deployed performance
  • Rapid improvement makes the date of each observation essential: evidence of 2025 struggle, early-2026 reliability and later progress must not be pooled into a timeless claim.
  • There is no standardized definition of “long-horizon”: different benchmarks use different task lengths, making cross-study comparison difficult

What Would Change Our Assessment

  • Historical scope: Retain the supported 2025 judgment while separately assessing early-2026 long-horizon weakness. The current confirmation is not proof of every early-2026 example. Later improvement does not erase historical struggle, and later failures cannot establish an earlier state.
  • Resolve the early-2026 component: Seek dated results on unfamiliar video games and comparable long-horizon tasks from that window, with prior exposure, assistance and reliability documented.
  • Watch for: Reliable measurements on longer and messier tasks, alongside complete-project outcomes and documented assistance. Beyond-range horizon estimates need additional validation.

Update History

DateUpdate
2026-11-09Assessment reviewed. Status: Confirmed. Confidence: 85%. Retained Confirmed (85%) for historical long-horizon struggle. Expanded the displayed window to 2025 and early 2026. Kept 2025 unreliability distinct from the partial early-2026 reliability assessment; dated METR results do not establish the unfamiliar-video-game example. Later progress neither erases the supported history nor resolves that earlier component.
2026-11-07Assessment reviewed. Status: Confirmed; confidence: 85%. This is a historical judgment about 2025, not a ceiling on current agents. METR’s dated primary results put Opus 4.5’s 80% horizon at about 49 minutes on November 24, 2025, consistent with difficulty reliably completing longer tasks in that period. Opus 4.6’s February 5, 2026 estimate is about 70 minutes at 80%, despite a much longer 50% horizon. Later gains remain relevant current context without undoing the historical assessment. These horizons measure human-expert task difficulty at specified success rates, not uninterrupted autonomous runtime. Correction: Dated the Opus 4.6 and April 7, 2026 Mythos estimates and retained METR’s reliability and discontinued-update warnings. Time horizon measures human-expert task difficulty; it is not autonomous runtime.
2026-10-19Epoch documented stronger strategic visual-agent performance alongside substantial maximum-difficulty failures. These game-specific results update current capability context without changing the historical 2025 judgment. Status and confidence unchanged.
2026-09-06Current evidence and assessment explanation reviewed. Clarified measurement scope, source interpretation and limitations.
2026-08-24NVIDIA reported a supervised AVO agent completing a productive seven-day kernel-optimization run with more than 500 explored directions and 40 committed versions. This adds a concrete long-horizon success case, while the narrow task and system-level supervision limit generalization. Status and confidence remain unchanged.
2026-06-06Corrected the Opus 4.6 METR framing: current TH1.1 raw data estimates roughly 11h59m at the 50% horizon and roughly 1h10m at the 80% horizon, not 14.5h at 80% reliability. Status unchanged because the 2025-era “long-horizon struggle” claim remains historically confirmed, while newer capability gains are tracked as a rapidly changing limitation.
2026-03Prediction confirmed for 2025 timeframe. Agents still struggle with multi-day autonomous work, though rapid improvement visible in early 2026.
2026-01METR Time Horizon 1.1: Claude Opus 4.5 at ~4h49m (Jan 2026). Agents now sustain coherent work for nearly 5 hours — well beyond the “simple long-horizon tasks” the prediction describes. The “struggle” characterization from mid-2025 is increasingly outdated for frontier models, though it remains accurate for most publicly available agents.
2025-09Scale AI SWE-Bench Pro (Sep 19): models scoring 70%+ on standard SWE-bench drop to ~23% on long-horizon tasks involving multi-file refactors and cross-repository changes. Long-horizon capability gap now formally measured.
2025-07METR domain time-horizon analysis (July 14): 50% success horizon for frontier models at ~50 minutes on human expert tasks. Tasks requiring hours of sustained autonomous work remain beyond current capability.
2025-06SWE-bench and GAIA benchmarks confirm agents fail on tasks requiring sustained multi-hour work. Error accumulation over long horizons remains the primary failure mode.