Methodology

1. What counts as a prediction?

AI 2027 is a narrative scenario, not a list of forecasts. It reads like a story — describing events, timelines, and dynamics in prose rather than as discrete, falsifiable claims. That makes it vivid but hard to score directly.

To track the scenario against reality, we extract individual predictions from the narrative. A statement qualifies as a trackable prediction if it meets all three of these criteria:

1. It describes something that could happen or not happen. A prediction must be about a state of the world, not a definition or framing device. "Labs race to build AGI" is a claim about the world. "AGI would change everything" is commentary.
2. It has a discernible timeframe. The scenario places events in rough chronological order, sometimes with explicit dates ("by mid-2026") and sometimes through context ("during this period of rapid scaling"). We accept both, noting when the timing is inferred rather than stated.
3. It can, in principle, be compared with evidence. We don't require a clean binary yes/no — many predictions are directional — but there must be some class of real-world evidence that would inform the assessment.

Each extracted prediction includes the original claim (exact quote or faithful paraphrase), a source reference to the relevant section of the scenario, and an explanation of how we interpret the claim in context.

We currently track 66 predictions across eight categories: model capability, agent autonomy, coding automation, governance, security, geopolitics, economic impact, and takeoff dynamics.

2. How narrative statements become trackable

Once a prediction is extracted, the next step is operationalization — converting prose into something we can monitor. This is where most of the interpretive work happens, and where we're most careful to show our reasoning.

Each prediction is mapped to one or more proxy types:

Every prediction page includes a "How We Track This" section that specifies which proxies we use and why. This is the most important section for readers who want to challenge our framing — if the operationalization is wrong, the assessment is wrong regardless of the evidence.

3. What counts as evidence?

Not all information is equally useful. We accept evidence from the following classes, roughly ordered by reliability:

1. Benchmark results and technical evaluations. Published scores on recognized benchmarks (SWE-bench, GPQA, ARC-AGI, METR task horizons, etc.), ideally from independent evaluators or the benchmark maintainers themselves.
2. Model release details. Official announcements, technical reports, and system cards from AI labs describing model capabilities, training details, and evaluation results.
3. Public product behavior. Observable, reproducible capabilities in shipping products. What can a user actually do with a tool today?
4. Official policy documents. Legislation, executive orders, regulatory filings, export control rules — not rumors or leaked drafts, but enacted or formally proposed policy.
5. Credible reporting. Well-sourced journalism from outlets with a track record (Reuters, Bloomberg, The Information, etc.). We note when an assessment relies on reporting rather than primary sources.
6. Published analyses. Research papers, think-tank reports, credible blog posts with transparent methodology. We evaluate the argument, not the author's credentials.
7. Direct statements from relevant actors. Public statements from lab leaders, government officials, or other key figures. We treat these as data points about intentions, not as ground truth about outcomes.

Evidence discipline

• Primary sources preferred. We prioritize original data over media coverage of that data.
• No single-source status changes. We don't change a prediction's status based on one report, one benchmark, or one announcement alone — unless the evidence is unambiguous (e.g., a model is publicly released).
• All evidence is dated. We note when evidence was published. Outdated evidence is flagged, not silently relied upon.
• Links are verified. Evidence URLs are checked for accessibility. Broken links are replaced or noted.
• Counterevidence is required. Every prediction page includes genuine counterarguments — not strawmen, but the strongest case against our current assessment.

4. How statuses are assigned

Each prediction carries one of six statuses. These are judgments, not mechanical outputs — we explain the reasoning on every prediction page.

Where specified, outcome records whether the claim is observed or unresolved; timing records early, within-window, late, original window passed with outcome unresolved, or pending resolution; evidence strength distinguishes direct evidence, proxies, disputed findings and insufficient evidence. A passed deadline can remain unresolved when the outcome is not observable. These fields should be read with the explanation, not as an automatic scoring formula.

How status differs from confidence

Status reflects what we think is happening. Confidence reflects how sure we are about our assessment. A prediction can be "On Track" with moderate confidence (the direction looks right but evidence is limited) or "Confirmed" with high confidence (clear, multi-source evidence). These are independent dimensions — see confidence scores below.

5. How timing mismatches are handled

A recurring challenge: the scenario gets the direction right but the timing wrong. How do we score that?

Our approach distinguishes three dimensions of a prediction:

• Direction: Is the thing happening at all? (e.g., Are AI coding tools improving?)
• Magnitude: Is it happening at the predicted scale? (e.g., Are they handling 80% of tasks, or 30%?)
• Timing: Is it happening when predicted? (e.g., By mid-2026, or later?)

We use different statuses for different mismatch patterns:

• Right direction, right timing → On Track or Confirmed
• Right direction, ahead of schedule → Ahead
• Right direction, behind schedule → Behind (with notes on expected delay)
• Right direction, magnitude unclear → Emerging or On Track (depending on signal strength)
• Direction unclear → Emerging or Not Yet Testable

We don't penalize the scenario for being directionally correct but off by a few months. The scenario describes events over a multi-year arc — expecting month-level precision from a narrative document would be unreasonable. However, we do flag timing mismatches explicitly, because a prediction that's "right but two years late" is meaningfully different from one that's "right on time."

Each prediction page includes timeline notes identifying whether the main uncertainty is about direction, timing, interpretation, measurement, or external dependencies.

6. How ambiguous claims are handled

AI 2027 is a scenario, not a specification. Many of its claims are deliberately vague, metaphorical, or embedded in narrative context that admits multiple interpretations. We handle this through several practices:

Explicit interpretation

Every prediction page includes a tracker interpretation — our reading of what the claim means in operational terms. When the original text is ambiguous, we state our interpretation clearly and explain why we chose it. Readers can disagree with the interpretation while still finding the evidence useful.

Multiple readings

When a claim supports multiple reasonable interpretations, we note them. Sometimes we'll track a narrow reading (easier to score) alongside a broad reading (closer to the scenario's spirit). We indicate which interpretation drives our current status.

Charitable but honest framing

We aim for the most reasonable interpretation — not the weakest (easy to confirm) or strongest (easy to falsify). If a claim is genuinely unclear, we say so rather than picking the reading that makes our status cleanest.

Narrative context matters

Some claims only make sense in the context of the broader scenario. "Agents can self-replicate" means something different depending on whether the surrounding narrative describes formal sandboxed evaluations or autonomous real-world behavior. We use the scenario's own context to constrain interpretation, and we explain when we do.

Unfalsifiable claims

A small number of claims are effectively unfalsifiable within any reasonable timeframe or evidence structure. We flag these as "Not Yet Testable" with a note explaining why, rather than forcing a status that implies more certainty than we have.

7. How revisions are handled

A tracker that never changes is useless. A tracker that changes without explanation is untrustworthy. Our revision practices aim to balance responsiveness with discipline.

When we update

• New evidence: When materially relevant evidence emerges — a benchmark result, model release, policy decision, or credible report — we update the affected prediction pages.
• Status changes: When accumulated evidence warrants changing a prediction's status, we do so and document the reasoning.
• Interpretation corrections: If we realize our operationalization was flawed or our interpretation of the original claim was wrong, we revise it and explain why.
• Error corrections: Factual errors, broken links, or outdated information are fixed as soon as identified.

How we document changes

• Every prediction page includes an Update History section — a chronological log of status changes, evidence additions, and interpretation revisions.
• The site-wide Changelog records significant changes across all predictions.
• Every page shows a last updated timestamp.

What we don't do

• We don't silently change statuses. Every change is logged.
• We don't retroactively adjust interpretations to make old assessments look better. If we got something wrong, we say so.
• We don't update for the sake of updating. Changes are driven by evidence, not a content calendar.

8. How external comparisons are used

This tracker doesn't exist in a vacuum. Other sources track overlapping questions — sometimes with different conclusions. We use them as inputs, not substitutes.

AI Futures Project

The AI Futures Project includes some of the authors of the AI 2027 scenario. Their later updates and self-grading provide valuable context — especially when they've revised or qualified their own views. We reference these but don't treat them as authoritative on what reality looks like. The authors' assessment of their own scenario is interesting but not the same as independent verification.

Metaculus and crowd forecasts

Metaculus runs prediction tournaments covering some of the same questions the scenario addresses. Crowd forecasts provide a useful calibration check — if our assessment diverges sharply from aggregated forecaster opinion, that's worth noting and explaining. But crowd medians aren't ground truth. We use them as one signal among many.

Other sources

We reference benchmarks (METR, SWE-bench, GPQA, ARC-AGI), lab publications, government documents, and credible analyses as they become relevant. No single external source drives our assessments.

The key principle

External sources are additional evidence and calibration, not replacements for our own assessment. We never collapse an external forecast into the original scenario — "Metaculus agrees" is not the same as "the evidence confirms." We keep the layers separate: the original claim, external views, observed evidence, and our assessment.

9. Assessing the scenario’s pace

We compare individual predictions with their specified quantities and time windows. We do not combine task benchmarks, investment and research productivity into a single current speed ratio. These measures cover different parts of the scenario and have different uncertainties.

The homepage presents published METR task-horizon measurements alongside evidence about real work and AI research. Task horizons describe human-equivalent difficulty at a specified success rate. They do not directly measure whole-job reliability or the speed of a research laboratory.

Each reviewed dossier distinguishes the observed outcome, its timing and the strength of the evidence. A proxy is labeled as a proxy. An event observed after its deadline is recorded with its lateness visible. Missing public evidence remains a limitation where the event is not observable.

Earlier dated assessments remain part of the record. The current synthesis is in AI 2027 vs reality.

10. Confidence scores

Each prediction carries a confidence score (0–95%) reflecting how certain we are about our status assessment. This is distinct from whether the prediction is "right" — it's about the strength of our evaluation.

• 90–95%: Very high confidence. Multiple strong, independent evidence sources. Little room for reasonable disagreement about the status.
• 70–89%: High confidence. Good evidence, but some ambiguity in interpretation or incomplete coverage.
• 50–69%: Moderate confidence. Evidence is mixed, the prediction is hard to operationalize, or reasonable people could disagree on status.
• Below 50%: Low confidence. Limited evidence, inherently difficult to verify, or the prediction is too vague for strong assessment.

A prediction rated "Confirmed" with 75% confidence means: we believe the evidence supports confirmation, but there's meaningful uncertainty — perhaps about scope, about whether the evidence fully captures the original claim, or about alternative interpretations.

11. Guiding principles

1. Accuracy over hype. We don't bend interpretation to make the scenario look right or wrong. If the evidence is mixed, we say it's mixed.
2. Separate source from interpretation. Every assessment involves four layers: the original claim, our interpretation, the evidence, and our assessment. We keep them visibly distinct.
3. Be explicit about uncertainty. We use statuses and confidence scores that allow ambiguity. Forcing false precision is worse than admitting uncertainty.
4. Build for skeptics. People who think AI 2027 is too aggressive — or too narrative-driven — should still find this site useful. The tracker is credible only if it's credible to people who disagree with the scenario.
5. Show your work. Every judgment is sourced and explainable. If we can't explain why we assigned a status, we shouldn't assign it.
6. Correct visibly. Revisions are a feature, not a bug. When we change an assessment, we document when, why, and what changed.