Coding fully automated but research taste still requires humans
Tracker summary: Research taste remains difficult to train even when coding is fully automated; the two separate scenario excerpts appear below in source order.
Autonomous implementation is advancing while difficult research judgments remain a bottleneck in tested settings. The full March 2027 combination remains unestablished.
At a glance
- Assessment: Emerging
- Confidence in assessment: 80%
- Outcome: unresolved
- Timing: pending
- Evidence: proxy
- Predicted timing: March 2027
- Primary source: March 2027: Algorithmic Breakthroughs, archived PDF p. 12
What AI 2027 Predicted
For example, research taste has proven difficult to train due to longer feedback loops and less data availability.
Source: March 2027: Algorithmic Breakthroughs: archived PDF, p. 12.
Later in the same section (intervening material omitted):
Now that coding has been fully automated, OpenBrain can quickly churn out high-quality training environments to teach Agent-3’s weak skills like research taste and large-scale coordination.
Source: March 2027: Algorithmic Breakthroughs: archived PDF, p. 12.
Tracker summary: By March 2027, coding is fully automated while research taste and strategic judgment remain important human contributions. The scenario does not require AI to have no useful research judgment. AI 2027
Scenario source locator: March 2027: Algorithmic Breakthroughs: archived PDF, p. 12.
How We Track This
Assess implementation autonomy and research judgment separately. A coding benchmark cannot by itself establish sound problem selection, and success on a well-scored research task need not transfer to open-ended inquiry.
Shared full-coding-automation test: Require convergent evidence that a system can perform any AI-research coding task done by the best AGI-company engineer, while being much faster and cheaper. Record task coverage, human interventions, quality, elapsed time and cost against comparable human baselines. Narrow implementation results are precursor evidence; without the full comparison, the coding component remains unestablished on both pages. Research taste is a separate component, and confirmation of the conjunction requires both components.
Apply the same test as Superhuman coder emerges. The coding component remains unestablished under that test: the evidence below lacks the full best-engineer comparison, speed and cost demonstration. Emerging here describes precursor evidence for a developing implementation/judgment split, not full coding automation or the complete conjunction. The companion’s Not Yet Testable assessment concerns the full coding milestone. Neither displayed confidence is a probability that full automation has occurred.
Current Evidence
Anthropic and Theorem’s October 22 formal-verification report describes Rocq-checked no-crash proofs for 239 entry points across 123 libraries. The guarantees apply to specified patched, compiled RISC-V/Linux binaries with bounded inputs and fixed memory in a single-threaded setting, conditional on trusted semantics, theorem statements and checker correctness. They do not establish complete software correctness or coding automation. Patches regenerate specifications and can invite reward hacking; the authors penalized and audited patches, while entry-point and sanitizer coverage remain incomplete. This is a first-party result, not an independent rerun of the proofs.
OpenAI and collaborators’ biological-data study, whose preprint was posted October 16, 2026, reports blinded experts preferring distributed-agent over single-agent reports 75% of the time with model and resources held fixed. That supports useful research assistance, not superiority to timed human scientists: the months-to-year human comparison is an author estimate rather than a timed control or an AI-for-AI multiplier. Proposed mechanisms from the existing mouse-brain perturbation data still require experiments. These results do not establish that research taste must remain exclusively human.
Anthropic’s August 28 TASTE study compared judgments on 92 pairs of AI-safety proposals. The best model agreed with researcher preferences about 60% of the time, versus an estimated 77% for researchers. Model uncertainty was approximately ±10 percentage points. This identifies a judgment gap in one curated setting. TASTE study
An independent research study, first posted July 29, 2026 and revised August 7, 2026, ran two six-day shadow evaluations. Agents completed substantial engineering but failed to answer the central research questions; the original researchers rejected the outputs. That supports a distinction between implementation and research success, with a very small sample. Kirgis et al.
Epoch’s October 7 InnovationEval tested GPT-5.6 Sol and Claude Fable 5 on one post-training-method target. After scope and wall-clock corrections, Sol recovered about 15% of the reference method’s gains; Fable did not improve the in-scope result. One main run per model and a single target limit generalization. Epoch’s October 8 Automation Reports tested six models on 11 organizational tasks. They performed better on well-defined implementation than on experimental design and interpretation; the small suite used one human grader per output.
Google’s October safety report reports that fully delegated production pull requests rose from 7.5% in April to 11.9% in September 2026, while about 88% still required iterative human steering or human authorship. Its survey of 12 senior leaders and researchers describes greater automation of well-specified execution than of open-ended scientific judgment. The production analysis excludes experimental code. These first-party internal observations support a developing division of labor without establishing fully automated coding.
Counterevidence & Limitations
Automated researchers also produce useful methods. Anthropic tested ten well-characterized alignment failures and found automated method search could outperform one-shot proposals from 28 researchers, including on held-out evaluations. Unequal iteration budgets and measurable objectives limit broader comparisons. Automated alignment researchers
Preference agreement is not an objective measure of scientific value. Two difficult research projects cannot characterize every field. Current evidence supports an emerging division of labor, but neither comprehensive coding automation nor a durable human advantage in taste is established.
What Would Change Our Assessment
- Strengthen: Independent longitudinal evaluations show reliable autonomous implementation alongside persistent strategic research failures.
- Confirm: Both components are demonstrated at the specified March 2027 scope.
- Weaken: Agents reliably choose and execute valuable open-ended research, or coding itself remains a substantial human bottleneck.
Update History
| Date | Update |
|---|---|
| 2026-11-07 | Assessment reviewed. Status: Emerging; confidence: 80%. TASTE finds about 60% agreement with its expert-preference reference for the best models versus 77% for researchers, supporting an emerging judgment gap. Automated alignment research nevertheless improves held-out robustness in bounded settings. Reported coding migrations supply separate implementation-gain evidence; together these results support an emerging split. Preference-based testing, small research samples and successful automated method search leave complete coding automation and a durable best-human taste advantage unproved. |
| 2026-11-07 | Correction: Replaced the reverse-order assembled header with a labeled summary and two separately quoted sentences in source order on archived page 12; corrected predicted timing to March 2027. Added the biological-data preprint’s October 16 posting date and Kirgis et al.’s July 29 posting/August 7 revision dates. Synchronized the shared best-engineer, faster-and-cheaper coding test with the companion dossier; precursor implementation and research-taste evidence do not establish full coding automation or the conjunction. |
| 2026-10-26 | Added conditional software-proof results and useful distributed biological analysis. Full coding automation and a durable human research-taste advantage remain unestablished. Status and confidence unchanged. |
| 2026-10-19 | Google’s internal evidence separates increasing code generation from complete delegation and continuing human research judgment. Full coding automation and a durable research-taste gap remain unverified. Status and confidence unchanged. |
| 2026-10-12 | Epoch’s single-target method test found limited gains, while its 11-task workplace suite showed stronger implementation than experimental judgment. These bounded results add to the research-judgment evidence; full coding automation and a persistent taste gap remain unverified. Status and confidence unchanged. |
| 2026-09-07 | Added proposal-judgment and open-ended research evaluations, balanced against successful automated alignment-method search. The evidence supports a developing implementation/judgment split, not the full 2027 conjunction. |
| 2026-09-06 | Assessment revised from not-yet-testable (0.55) to emerging (0.80). Coding automation and research-judgment limitations have supporting evidence in particular settings; full coding automation and a universal judgment barrier remain unestablished. |
| 2026-09-06 | Removed the inference from developers’ reluctance to work without AI to full coding automation. METR’s study update does not establish that milestone. Status and confidence were unchanged. |
| 2026-08-31 | Two long-horizon AI-R&D evaluations found strong implementation and experiment execution but persistent limits in strategy revision, novelty, stability, and experience reuse. The larger trajectory analysis covered 1,338 post-training runs. This strengthens early evidence for a research-judgment bottleneck, while full coding automation remains unproven. Status and confidence remain unchanged. |
| 2026-08-03 | Two six-day shadow evaluations found frontier agents completed all engineering without human help but made no substantial progress on the central research questions. This supports the predicted research-taste bottleneck, while the two-case sample and absence of full coding automation keep the prediction not yet testable. |
| 2026-03 | Prediction timeframe not yet reached. Coding automation progressing rapidly but far from complete. Early signals support the predicted asymmetry — routine coding tasks automate faster than research taste and architectural judgment. |
| 2026-02 | METR notes developers refuse to work without AI (coding automated). METR time horizons showed Opus 4.6 making a major jump on general software tasks, but RE-Bench-specific research engineering scores still lag. Gap narrows but persists. |
| 2025-11 | Claude Code at $1B ARR confirms massive coding automation adoption. SWE-bench crosses 80%. But RE-Bench (research engineering) scores remain 0.5-0.8 — coding ahead, research taste behind. |
| 2025-08 | Anthropic internal study: engineers can only “fully delegate” 0-20% of work. Task complexity rising (3.2→3.8). Coding increasingly automated; judgment and design remain human. |