External evaluators lack access to the strongest internally deployed models
Tracker summary: A lab limits a prior model-sharing commitment to external deployment, leaving foreign safety evaluators without access to the strongest internal systems.
Internal deployment is supported in part; the required evaluator-access exclusion has not been verified.
At a glance
- Assessment: Not Yet Testable
- Confidence in assessment: 75%
- Outcome: unresolved
- Timing: pending
- Evidence: insufficient
- Predicted timing: May 2027
- Primary source: AI 2027 archived scenario: May 2027: National Security, p. 14
What AI 2027 Predicted
Tracker summary: A lab limits a prior model-sharing commitment to external deployment, leaving foreign safety evaluators without access to the strongest internal systems. The source is May 2027: National Security in the AI 2027 scenario.
Scenario source: archived version and page-number reference: May 2027: National Security, p. 14.
How We Track This
Identify a named lab and a documented prior pre-deployment access commitment. Establish both consequential internal use of a more capable system and denial or contractual exclusion of that system from the UK AI Security Institute’s access. Require records or independently corroborated direct knowledge of both sides, including dates and capability basis.
The scenario names the UK institute (UK AISI). As an explicit tracker extension, apply the same commitment, internal-use and exclusion requirements to other allied-government safety institutes, reported separately.
These operational thresholds specify this tracker’s test. Where the scenario uses qualitative language, they are interpretations rather than numbers quoted from the original.
Current Evidence
Anthropic’s August report describes internally used capabilities with a July 15 coverage cutoff. That supplies only one part of the test. No qualifying combination of a prior access commitment and exclusion of that internal system was identified in this review.
Counterevidence & Limitations
Scope: METR and Accenture’s Faculty are non-government evaluators. Their access is context about a broader exclusion hypothesis, not a direct test of the scenario’s allied-government/UK AISI access claim.
On September 18, 2026, Anthropic announced: “The partnership will be led by Faculty, Accenture’s specialist AI business”. Its specified work includes evaluating and red-teaming models, alignment assessments and safeguards testing. The post describes employee-comparable access as a general feature of embedded evaluators, not a separately finalized statement of Faculty’s own access. Anthropic and Accenture each expect to invest at least $1 billion over five years. Operating details and access standards remain unsettled, and the announcement does not establish access to Anthropic’s strongest internally deployed model. Anthropic
Anthropic’s October 15 update, clarified October 16, says six METR employees have been onboarded with internal tools and systems roughly comparable to its risk-assessment teams. METR can publish its assessments, subject to predefined redaction categories whose application Anthropic ultimately controls; METR retains limited rights to describe the access it received. The announcement describes implemented evaluator access. It does not document access to the strongest internally deployed model. Anthropic evaluator update
Missing public results, ordinary NDA protection, or a model under development without internal use do not establish denied access. Capability secrecy alone is a separate existing dossier.
Missing qualifying public evidence does not establish that the event has not happened privately. The assessment concerns the documented test and its observable evidence.
What Would Change Our Assessment
The relevant evaluator receives access before or during consequential internal deployment, or the claimed access commitment never applied.
Evidence satisfying the full test would change the assessment; early qualifying evidence can be considered before the scenario window. Partial proxies are recorded with their scope.
Update History
| Date | Update |
|---|---|
| 2026-11-07 | Assessment reviewed. Status: Not Yet Testable; confidence: 75%. Anthropic’s August risk report documents internal model use; its October evaluator update documents access for non-government METR staff. Neither establishes denial of the strongest internally deployed system to UK AISI under a prior access commitment, so the specified exclusion remains unverified. Dated Faculty’s announcement September 18, 2026 and distinguished its named role from general employee-like access. METR and Faculty are not government bodies; UK AISI testing is a separate evidence category. |
| 2026-10-19 | Anthropic reported implemented METR evaluator access and defined publication/redaction terms. This is countervailing access evidence, while coverage of the strongest internal model remains unverified. Status and confidence unchanged. |
| 2026-09-21 | Anthropic announced an embedded evaluator intended to receive employee-comparable access. This is material countervailing governance evidence, but actual access to the strongest internal systems is not yet documented; status and confidence unchanged. |
| 2026-09-06 | Internal deployment is supported in part; the required evaluator-access exclusion has not been verified. Relevant primary evidence and limitations are described above. |