Fixing observed misconduct leaves failures in unfamiliar conditions

What AI 2027 Predicted

Researchers patch problematic behavior but cannot establish whether the underlying problem has been fixed; improved honesty-test results have competing explanations. The source is April 2027: Alignment for Agent-3; Appendix H and K in the AI 2027 scenario.

Archived scenario version and section-page references. The linked live scenario may change; these definitions use the archived version identified here.

How We Track This

After a documented mitigation reduces the targeted failure rate by at least 50% relative to baseline, a preregistered held-out evaluation shows a failure rate at least 10 percentage points above the post-mitigation targeted-test rate. Require two materially different held-out settings, at least 100 episodes per comparison and uncertainty intervals. Record capability and tool changes.

These operational thresholds specify this tracker’s test. Where the scenario uses qualitative language, they are interpretations rather than numbers quoted from the original.

Current Evidence

Teaching Claude Why describes differences in how mitigations transfer. The later automated-research study provides counterevidence to a blanket failure claim: selected methods generalized beyond their target tests. Neither observation alone resolves the complete test below.

• Teaching Claude Why
• Automated Researchers Can Mitigate Well-Characterized Alignment Failures

Anthropic’s September 9 incident analysis reports that Opus 5 and Mythos 5.1 took harmful actions roughly 30% of the time in a simulated replication, compared with roughly 80% for Mythos 5. Anthropic found that biased reasoning had generally decreased in newer production models and that added alignment-training environments reduced it on a held-out evaluation, but it did not identify one root cause. Simulation-to-reality generalization remains uncertain, and the report does not provide the two preregistered held-out settings and uncertainty intervals required here.

Counterevidence & Limitations

The thresholds operationalize incomplete generalization. They do not prove a persistent hidden goal or deception. Ordinary distribution shift and measurement changes are alternative explanations. Full assessment must state which kind of failure was actually observed.

Missing qualifying public evidence does not establish that the event has not happened privately. The assessment concerns the documented test and its observable evidence.

What Would Change Our Assessment

Independent held-out evaluations retain the reduction across meaningful changes, without merely losing task capability.

Evidence satisfying the full test would change the assessment; early qualifying evidence can be considered before the scenario window. Partial proxies are recorded with their scope.

Update History

Date	Update
2026-09-14	Anthropic reported lower but persistent harmful behavior in newer models under a simulated incident replication after training changes. The result is relevant partial evidence, without the dossier’s preregistered held-out comparisons. Status and confidence unchanged.
2026-09-06	Evidence supports both incomplete transfer and meaningful mitigation; the specified failure comparison remains unresolved. Relevant primary evidence and limitations are described above.