Experiment Toward Truth
Improve by testing what works and learning from every result
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 95%
Keating describes experimentation as refusing to treat a strong current product as finished. Start with a specific dissatisfaction, change something, observe what works and what does not, and use the result to move closer to truth. In this view, a failed experiment is not empty because it narrows possibilities and informs the next trial. The mechanism is a learning loop rather than aimless novelty: identify a limitation, define a bounded test and its evidence, run it, interpret both positive and negative results, then update the next version. The output is steady improvement without requiring every attempt to confirm the original idea. This applies to scientific instruments, creative formats, and products wherever results can be observed safely.
Origin
Keating describes the experimentation process while discussing how The Diary of a CEO keeps improving despite already leading its category.
Core principles
- 01Improvement starts with dissatisfaction about the current version
- 02Experiments should reveal what works and what does not
- 03A negative result still produces information
- 04Learning moves the work closer to truth
- 05Strong current performance is not a reason to stop testing
How to run it
- 1
Find the current limitation
Identify one concrete reason the present product or explanation is not yet good enough.
Pro tip Choose a limitation that can produce observable evidence.
Watch out General dissatisfaction does not define a useful experiment.
- 2
Design a bounded test
Change one relevant element and decide what evidence will show whether it helped.
Watch out Do not change so many variables that the result becomes uninterpretable.
- 3
Observe both sides
Record what worked and what did not instead of collecting only confirming evidence.
- 4
Extract the learning
Treat every result as information that updates your understanding, including a result that rejects the idea.
Pro tip Write the lesson before launching the next test.
- 5
Iterate toward truth
Use the evidence to improve the product or sharpen the next question, then repeat the loop.
Watch out Iteration without updating from evidence is repetition, not experimentation.
In the wild
Keating characterizes Bartlett's process as remaining dissatisfied with the current product, testing changes, and learning what works even though the show already leads its category. A creator could similarly test one opening format, compare audience response, document the result, and use it to shape the next episode.
→ Existing success becomes a platform for evidence-led improvement rather than complacency.
Common mistakes
Calling a negative result useless
A rejected idea can narrow the search and bring the next test closer to truth.
Testing without a learning question
A change is not an experiment unless the result can update a belief or decision.
Is it for you?
Best for
It is best for researchers, creators, and product teams improving something through repeated trials.
Not ideal for
It is not ideal for irreversible decisions that cannot be tested safely on a bounded scale.
From the transcript
“see what works see what doesn't work”
“there's no such thing as a failed experiment”
“you always learn something and that brings you closer to truth”
From the episode
Brian Keating: I’m Spending $200 Million To Explore Why We Exist! How God Fits Into Science Finally Explained, Is God Real?