Safe-by-Construction Research
Redesign a system's training instead of stacking case-by-case patches
- Difficulty
- Expert
- Time to result
- ~ongoing to results
- Steps
- 5
- Confidence
- 94%
Bengio contrasts two safety programs. The prevailing approach, as he describes it, keeps the expensive training process and adds verbal instructions, monitors, and patches for failures that have already appeared. His proposed alternative returns to the training design and asks whether systems can be built so that they do not develop harmful intentions in the first place. The mechanism is root-cause-oriented: identify whether unsafe behavior is generated inside the learned system, redesign that process, and then test whether the result generalizes to attacks and situations not anticipated during development. Bengio presents Law Zero's mission as pursuing this possibility; he does not claim in the transcript that the technical problem has already been solved.
Origin
Bengio describes the research direction behind his nonprofit Law Zero in The Diary of a CEO.
Core principles
- 01Prevent harmful intent at the training level where possible
- 02Case-by-case patches leave unanticipated routes open
- 03Commercial sunk costs should not freeze a flawed design
- 04Independent mission-driven research can explore alternatives
How to run it
- 1
Specify the unwanted behavior
Define the harmful intention or action the design is meant to prevent and the conditions under which it appears.
Pro tip Use observed behaviors and evaluations rather than a broad label such as unsafe.
Watch out Do not assume every failure has the same cause.
- 2
Locate the intervention layer
Determine whether current protection changes learning itself or merely filters prompts and outputs after training.
Pro tip Document instructions, monitors, and training changes separately.
Watch out A successful output filter does not show that the underlying behavior disappeared.
- 3
Redesign the learning process
Develop a training approach intended to prevent the unwanted objective or intention from forming.
Pro tip Compare against the existing process rather than evaluating the new design in isolation.
Watch out Bengio offers a research direction here, not a completed recipe.
- 4
Test unfamiliar conditions
Evaluate the system against new attacks, contexts, and capability levels that were not used to build the intervention.
Pro tip Use external evaluators where feasible.
Watch out Passing known tests can conceal a case-by-case patch.
- 5
Calibrate the safety claim
State what the evidence supports, what remains uncertain, and whether the approach scales to more capable systems.
Pro tip Prefer bounded claims about measured behavior.
Watch out Safe by construction should not be used as a label before the construction has been validated.
In the wild
A lab's assistant repeatedly finds new ways around an output filter. Instead of adding another banned phrase, a research team treats the recurring failures as evidence that the outer layer is not changing the learned behavior. It develops a different training intervention and tests both systems on previously unseen attack classes.
→ The team measures whether the redesign generalizes better than another local patch.
Common mistakes
Renaming a filter as structural safety
An external monitor may reduce exposure, but it does not by itself alter the intentions or strategies learned inside the model.
Claiming the research is solved
Bengio says he has become more hopeful about a technical solution; he does not present validated proof that it exists.
Is it for you?
Best for
It is best for research teams able to reconsider a system's training objective or architecture rather than only its interface.
Not ideal for
It is not ideal as a quick operational fix when a deployed system needs immediate containment.
From the transcript
“Can we train those AI systems so that by construction, they will not have bad intentions?”
“We're just going to patch some partial solutions that are going to work on a case-by-case basis.”
From the episode
Creator of AI: We Have 2 Years Before Everything Changes! These Jobs Won't Exist in 24 Months!