Precautionary Principle for Catastrophic Risk
Reject irreversible bets even when their estimated probability is low
- Difficulty
- Moderate
- Time to result
- ~days to results
- Steps
- 5
- Confidence
- 96%
Bengio invokes the precautionary principle for activities whose failure could produce catastrophic, irreversible harm. The mechanism combines consequence severity with uncertainty: first identify the worst credible outcome, then ask whether even the lower end of plausible probability estimates is acceptable given that outcome. Expert disagreement does not settle the issue in favor of proceeding; for Bengio, it shows that society lacks enough information to confidently dismiss pessimistic scenarios. The result is not a formula for calculating AI extinction risk. It is a decision rule for setting a much higher burden of proof, constraining exposure, or declining an experiment when the downside could affect everyone and cannot be undone.
Origin
Bengio connects a long-standing scientific precautionary principle to advanced-AI development in The Diary of a CEO.
Core principles
- 01Scale caution with the severity and irreversibility of harm
- 02Expert disagreement signals uncertainty rather than safety
- 03A low estimated probability can remain unacceptable
- 04The burden of proof rises when everyone bears the downside
How to run it
- 1
Define the credible catastrophe
Describe the worst outcome that informed experts consider plausible, without treating it as certain.
Pro tip Separate a plausible scenario from a prediction that it will happen.
Watch out Do not use a dramatic hypothetical that has no stated mechanism.
- 2
Test irreversibility and reach
Determine whether the damage could be repaired and how many people would be exposed.
Pro tip Raise the threshold when harms cross borders or generations.
Watch out A benefit to the decision-maker does not offset involuntary risk to everyone else.
- 3
Map the uncertainty
Record the range of credible views and what evidence is missing. Treat disagreement as uncertainty, not proof that the risk is negligible.
Pro tip Show the range rather than averaging incompatible expert judgments.
Watch out A precise percentage can create false confidence when evidence is sparse.
- 4
Set the burden of proof
Choose how much safety evidence is required before proceeding, based on the scale and reversibility of harm.
Pro tip Require stronger evidence for consequences that cannot be recalled or contained.
Watch out Do not let competitive pressure quietly lower the threshold.
- 5
Constrain or pause
If the downside remains unacceptable, reduce the activity's scope, improve safeguards, or decline to proceed.
Pro tip Define what new evidence would justify reconsideration.
Watch out Precaution manages uncertainty; it does not prove the feared outcome will occur.
In the wild
A team cannot rule out an autonomous system copying sensitive data beyond its test environment. Although engineers call the event unlikely, the team identifies broad, irreversible exposure and no reliable containment evidence. It limits the system to an isolated environment and defines the independent tests required before wider access.
→ Deployment remains bounded until the safety evidence matches the potential harm.
Common mistakes
Treating disagreement as reassurance
A wide range of expert estimates means the uncertainty is unresolved; it does not establish that the optimistic estimate is right.
Ignoring consequence severity
A small probability is not automatically acceptable when the possible harm is global and irreversible.
Is it for you?
Best for
It is best for experiments, deployments, and policies where a failure could harm many people and cannot be repaired.
Not ideal for
It is not ideal for ordinary reversible decisions whose downside is bounded and recoverable.
From the transcript
“This is a case where previous generations of scientists have talked about a notion called the precautionary principle.”
“Even if it was only a 1% probability, let's say, just to give a number. Even that would be unbearable.”
From the episode
Creator of AI: We Have 2 Years Before Everything Changes! These Jobs Won't Exist in 24 Months!