Ab Testing

145 episodes | 48% corpus coverage

A controlled experiment where two versions of a product or feature are compared to see which performs better.

Keywords: ab testing, a/b testing, experimentation

Contents

Deep Dive

Metaphysical Synthesis & Analysis

The Synthesis

A/B testing is the scientific method applied to product decisions. Ron Kohavi's work at Microsoft and Amazon established it as the gold standard for validating hypotheses at scale. For PMs, the discipline lies not in running tests but in designing them: formulating falsifiable hypotheses, calculating minimum detectable effects, and understanding statistical significance versus practical significance. The Lenny corpus emphasizes that elite teams treat experimentation as infrastructure, not an occasional activity.

Metaphysical Foundation

A/B testing embodies epistemic humility—the recognition that our intuitions about user behavior are hypotheses requiring empirical validation. It transforms opinion into evidence.

Evolutionary Path

From simple split tests to sophisticated multi-armed bandits and sequential testing, the practice has evolved to handle increasing complexity: interaction effects, long-term impacts, and network effects that violate the stable unit treatment value assumption (SUTVA).

Strategic Interplays

A/B testing powers the learning-engine, providing the data that informs Prioritization. It is essential for optimizing conversion and Activation, and serves as the empirical foundation for product-led-growth.

Key Episodes

Brian Chesky

Brian Chesky’s new playbook

1:13:28

Dalton Caldwell

Lessons from 1,000+ YC startups: Resilience, tar pit ideas, pivoting, more | Dal...

1:20:53

Marty Cagan

Product management theater | Marty Cagan (Silicon Valley Product Group)

1:25:15

Ray Cao

Product management theater | Marty Cagan (Silicon Valley Product Group)

1:25:15

Elizabeth Stone

How Netflix builds a culture of excellence | Elizabeth Stone (CTO)

1:13:44

Kevin Weil

OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbook...

1:31:41

Nick Turley

Inside ChatGPT: The fastest growing product in history | Nick Turley (OpenAI)

1:35:38

Dr. Fei Fei Li

The Godmother of AI on jobs, robots & why world models are next | Dr. Fei-Fei Li

1:19:34

Fei Fei

The Godmother of AI on jobs, robots & why world models are next | Dr. Fei-Fei Li

1:19:34

Vikrama Dhiman

A framework for PM skill development | Vikrama Dhiman (Gojek)

1:12:19

Gustaf Alstromer

Lessons from working with 600+ YC startups | Gustaf Alströmer (Y Combinator, Air...

1:25:36

Michael Truell

The rise of Cursor: The $300M ARR AI tool that engineers can’t stop using | Mich...

1:11:14

Mike Krieger

Anthropic's CPO on what comes next | Mike Krieger (co-founder of Instagram)

1:06:22

Tobi Lutke

Tobi Lütke's leadership playbook: First principles, infinite games, and maximizi...

1:41:43

Elena Verna

10 growth tactics that never work | Elena Verna (Amplitude, Miro, Dropbox, Surve...

1:35:10

Dan Shipper

The AI-native startup: 5 products, 7-figure revenue, 100% AI-written code. | Dan...

1:34:57

Brandon Chu

AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff

1:37:47

Sander Schulhoff

AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff

1:37:47

Anneka Gupta

Becoming more strategic, navigating difficult colleagues, founder mode, more | A...

1:08:37

Manik Gupta

Becoming more strategic, navigating difficult colleagues, founder mode, more | A...

1:08:37