Calibrated Scenario Forecasting
Turn uncertain trends into concrete scenarios, then update the odds as evidence changes
- Difficulty
- Advanced
- Time to result
- ~ongoing to results
- Steps
- 6
- Confidence
- 96%
Daniel Kokotajlo describes forecasting as a disciplined attempt to make uncertainty concrete. Start by defining the milestone and mapping the sequence that could lead to it, then write a detailed scenario that shows how events might unfold over time. Treat the dates as points inside a probability distribution: the median marks a 50 percent estimate, while meaningful probability remains on both earlier and later outcomes. Monitor the pace of relevant trends, compare actual milestones with the scenario, and update the distribution when progress slows, accelerates, or informed people close to the work provide new evidence. The output is not a prophecy. It is a revisable model that helps people reason about consequences, prepare for plausible paths, and distinguish what is likely by default from what should happen instead.
Origin
Kokotajlo used AI forecasting inside OpenAI, later led the AI Futures Project, and co-authored concrete scenarios including AI 2027 and AI 2040 Plan A.
Core principles
- 01Make a concrete scenario without pretending it is certain
- 02Represent timing as a probability distribution rather than one fixed date
- 03Track trend pace and milestone sequence, not only the headline deadline
- 04Update estimates when observed progress or informed forecasts change
- 05Separate predictions about the default path from recommendations
How to run it
- 1
Define the milestone
State exactly what event the forecast is timing. Kokotajlo distinguishes full automation of AI research, superintelligence, and broader job automation rather than treating them as one event.
Pro tip Use an observable capability threshold instead of a vague label.
Watch out A forecast cannot be calibrated if the milestone changes whenever the date approaches.
- 2
Map the causal sequence
List the developments that would have to occur and the order in which they interact. This makes the forecast testable against intermediate progress rather than only its final outcome.
Pro tip Include strategic choices by the relevant organizations, not only technical capability trends.
Watch out Do not collapse a multi-stage process into one dramatic endpoint.
- 3
Write a concrete scenario
Describe a possible trajectory with enough time detail that readers can see how one event leads to another. Kokotajlo's AI 2027 scenario used a month-by-month path to make an uncertain future legible.
Pro tip Choose one coherent path for the scenario while preserving uncertainty around it.
Watch out Concrete detail is a reasoning aid, not a claim that every depicted event will occur.
- 4
Assign a distribution
Set a median date and recognize substantial probability on both sides of it. Communicate the 50 percent point separately from the dates used in a scenario.
Pro tip Explain what would need to be faster or slower for the tails of the distribution to occur.
Watch out Do not present the scenario's named year as the only forecast.
- 5
Update against evidence
Compare observed progress and informed views with the assumed pace. Move the distribution when the evidence changes, as Kokotajlo did when he first lengthened and later reconsidered his timelines.
Pro tip Track why each update happened so revision reflects evidence rather than mood.
Watch out Do not protect an old forecast for reputational consistency.
- 6
Separate forecast from plan
Label the default-path prediction independently from the desired intervention. Kokotajlo presents AI 2027 as a forecast scenario and AI 2040 Plan A as a recommendation.
Pro tip Show how the recommended action would deliberately alter the forecasted path.
Watch out A desirable scenario is not automatically the most probable one.
In the wild
Kokotajlo's AI 2027 scenario placed full AI-research automation in 2027, while his personal median had moved to 2028 by publication and later to 2030 as progress appeared slower. Conversations with people at Anthropic and OpenAI then pushed him to reconsider shorter timelines because they expected the milestone sooner.
→ The forecast remains a changing probability distribution rather than a defended headline date.
Illustrative example: a founder defines the milestone as three competitors launching autonomous customer-support agents, maps the required capability and adoption sequence, writes a quarterly scenario, and sets a median launch window with early and late ranges. Each month, product releases and customer pilots update the distribution.
→ The founder can time experiments and spending without treating one launch date as certain.
Common mistakes
Treating the scenario year as a promise
A concrete scenario illustrates one path; it does not eliminate probability from earlier or later outcomes.
Ignoring intermediate milestones
Watching only the final date discards the evidence needed to test whether the causal sequence is accelerating or stalling.
Confusing hope with expectation
A recommended future must be labelled separately from the path expected under current incentives.
Is it for you?
Best for
It is best for fast-changing domains where leaders need a plausible sequence, timing range, and updating discipline.
Not ideal for
It is not ideal when the target milestone is undefined or there is no observable evidence with which to update the forecast.
From the transcript
“I thought it's valuable to make a concrete guess just to sort of see what it might look like.”
“It's like a smeared out probability mass. And like the 50% mark is this particular year, but there's like a lot of possibility that it…”
“It's important to distinguish, like, this is what we recommend, this is what we want to happen from, like, this is what we actually think…”
From the episode
OpenAI Whistleblower FINALLY Speaks: “AI Has A 70% Chance Of Going Horribly Wrong!“