Ab Testing
A controlled experiment where two versions of a product or feature are compared to see which performs better.
Keywords:
ab testing, a/b testing, experimentation
Contents
Deep Dive
Metaphysical Synthesis & Analysis
The Synthesis
A/B testing is the scientific method applied to product decisions. Ron Kohavi's work at Microsoft and Amazon established it as the gold standard for validating hypotheses at scale. For PMs, the discipline lies not in running tests but in designing them: formulating falsifiable hypotheses, calculating minimum detectable effects, and understanding statistical significance versus practical significance. The Lenny corpus emphasizes that elite teams treat experimentation as infrastructure, not an occasional activity.
Metaphysical Foundation
A/B testing embodies epistemic humility—the recognition that our intuitions about user behavior are hypotheses requiring empirical validation. It transforms opinion into evidence.
Evolutionary Path
From simple split tests to sophisticated multi-armed bandits and sequential testing, the practice has evolved to handle increasing complexity: interaction effects, long-term impacts, and network effects that violate the stable unit treatment value assumption (SUTVA).
Strategic Interplays
A/B testing powers the learning-engine, providing the data that informs Prioritization. It is essential for optimizing conversion and Activation, and serves as the empirical foundation for product-led-growth.
Key Episodes
Brian Chesky
Brian Chesky’s new playbook
Dalton Caldwell
Lessons from 1,000+ YC startups: Resilience, tar pit ideas, pivoting, more | Dal...
Marty Cagan
Product management theater | Marty Cagan (Silicon Valley Product Group)
Ray Cao
Product management theater | Marty Cagan (Silicon Valley Product Group)
Elizabeth Stone
How Netflix builds a culture of excellence | Elizabeth Stone (CTO)
Kevin Weil
OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbook...
Nick Turley
Inside ChatGPT: The fastest growing product in history | Nick Turley (OpenAI)
Dr. Fei Fei Li
The Godmother of AI on jobs, robots & why world models are next | Dr. Fei-Fei Li
Fei Fei
The Godmother of AI on jobs, robots & why world models are next | Dr. Fei-Fei Li
Vikrama Dhiman
A framework for PM skill development | Vikrama Dhiman (Gojek)
Gustaf Alstromer
Lessons from working with 600+ YC startups | Gustaf Alströmer (Y Combinator, Air...
Michael Truell
The rise of Cursor: The $300M ARR AI tool that engineers can’t stop using | Mich...
Mike Krieger
Anthropic's CPO on what comes next | Mike Krieger (co-founder of Instagram)
Tobi Lutke
Tobi Lütke's leadership playbook: First principles, infinite games, and maximizi...
Elena Verna
10 growth tactics that never work | Elena Verna (Amplitude, Miro, Dropbox, Surve...
Dan Shipper
The AI-native startup: 5 products, 7-figure revenue, 100% AI-written code. | Dan...
Brandon Chu
AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff
Sander Schulhoff
AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff
Anneka Gupta
Becoming more strategic, navigating difficult colleagues, founder mode, more | A...
Manik Gupta