Does systematic A/B-test experimentation drive product success on its own, or must it be paired with big bets and patience for slow-payoff work?

Antoine Le Nel · Oct 18, 2024

Revolut's growth comes from a compounding engine of many small additive components rather than any single launch, campaign or marketing coup, which is why growth is steady and still accelerating

Because all the components add up, growth doesn't spike and decay like campaign-driven growth; the engine keeps getting more optimized week after week

Scope: says they see no limit yet even in mature markets

10:12 20Growth: Revolut's Chief Growth Officer on The Growth Playbook Revolut Used to Scale to $2.2BN in Revenue | How Revolut Launch and Grow Products | Why the Best PMs Don't Need A/B Tests & Why CAC is a BS Metric with Antoine Le Nel

Antoine Le Nel · Oct 18, 2024

Growth comes from relentlessly optimizing many small increments across the funnel, not from moonshot bets on big new products

Like Usain Bolt, who won by hundredths of a second by optimizing his start rather than being 10x better, small gains of 1-2% at every step compound into total dominance; Revolut's success is a million small things done well at the lowest level of granularity

44:14 20Growth: Revolut's Chief Growth Officer on The Growth Playbook Revolut Used to Scale to $2.2BN in Revenue | How Revolut Launch and Grow Products | Why the Best PMs Don't Need A/B Tests & Why CAC is a BS Metric with Antoine Le Nel

Raman Malik · Nov 15, 2024

Micro-optimizations are seriously underrated because a small improvement in retention or activation raises the entire active user base

In a growth model of new plus retained plus resurrected minus churned users, a 10% retention improvement lifts the whole water level of weekly and monthly actives

Scope: micro-optimizations have diminishing returns; at some point extra gains cost too much wasted experimentation time

10:01 20Growth: Inside Perplexity's Growth Machine: What Worked, What Did Not Work | Why Paid Acquisition is a Drug and Brand Marketing is BS | The Good, Bad and Ugly of A/B Tests and Why Micro-Optimisations are Under-Rated with Raman Malik

Jean-Denis Greze · Jul 21, 2023

Metric-driven product work at most companies incentivizes incremental, measurable progress ('walking') over ambitious work that looks like failure for a long time but delivers step-change payoffs

Incremental work shows daily measurable movement in the right direction, whereas the airport-and-airplane style work shows nothing for months, so revenue pressure makes teams abandon it right before it would pay off

31:46 20VC: Why Hiring in Tech is Broken and Founders Need to be as Good at Firing as they are Hiring, Why Product Differentiation is Unsustainable & Why the Current Generation of Tech Employees are Entitled and What Needs to Change with Jean-Denis Greze @ Plai

Harry Stebbings · May 19, 2025

An experimentation-driven process struggles with things whose payoff is slow, so you have to do work with no obvious short-term gain for compounding long-term advantage.

Content is the example: it takes time to know whether it works, so an A/B test would tell you to quit in week one even though the effect shows up months later — like gaining muscle at the gym.

73:48 20VC: Duolingo Co-Founder on Why $3M is Harder than $100M to Raise | Why You Should Always Take Tier 1 VCs Even at Worse Terms | Why Europe Can't Win Unless the US Screws Up | How AI Impacts the Future of Work and Education with Severin Hacker

Raman Malik · Nov 15, 2024 · hedged

A 25% success rate on big growth swings would be a good outcome, with roughly one genuine banger campaign or feature a year

11:23 A quarter hit rate on big swings with one banger a year is a good outcome

Martin Gontovnikas · Feb 14, 2024

In B2B you should try something completely different to get results faster, and use A/B tests defensively — ship the version you believe is better and keep it as long as the metric doesn't go down

He believes a design he judges better will perform better in the future even if it's flat today; the test's job is to catch declines

10:16 Use ab tests defensively ship conviction and monitor for declines

Severin Hacker · May 19, 2025

Duolingo's success comes from a process — running thousands of A/B experiments and doubling down on what works — not from any single feature or mechanic like the streak or leaderboards.

The streak was one experiment, but roughly 300 follow-on experiments fine-tuning it are where the retention gains came from; the same test-and-double-down process drives marketing across TikTok and Instagram.

72:25 Experimentation and doubling down process drives success not single features

Severin Hacker · May 19, 2025

An experimentation program needs a portfolio of both small low-risk tests and large bets; running only tiny experiments traps a product in a local maximum

Optimizing only copy or purchase-page details can't produce innovation or relevance; big swings like adding chess or math are what open new ground

74:14 Portfolio of small tests and big bets avoids local maximum

Antoine Le Nel · Oct 18, 2024

A/B testing is the wrong tool for a company chasing 10x changes and it slows time to market, which is why Revolut largely skips it and just launches

You don't need a test to detect a 10x effect — you ship it and see it go up — and testing adds a sequential step of running the test, picking a winner, then launching

7:22 Ab testing adds unneeded steps for effects large enough to see without a test

Phil Carter · Sep 20, 2024

Seed and Series A startups should not spend time on small growth optimizations and should take bigger swings instead

Small optimizations won't move the needle at that size; if you're that early you should still have lots of low-hanging fruit (or you're on the wrong product/solution); and you don't have enough users to measure a statistically significant difference in a small A/B test

9:05 Early stage companies should skip small optimizations for bigger swings

Mike Hudack · Sep 13, 2024

Small consumer features should be shipped as carefully designed controlled experiments against a specific target metric, and unshipped if they show no statistically significant impact

With a correctly powered experiment and clean execution you will get an answer — e.g. driver chat can be measured purely by whether it reduces rider experience time versus a holdout

39:20 Ship small features as controlled experiments and remove if no measurable impact

Matt Lerner · May 31, 2024

Whether to pursue big swings or incremental optimization depends on company stage: early on optimizing is premature and you must find the big levers first, then split roughly 70/30 between squeezing working levers and hunting the next big one

You need at least one or two levers working before it's worth devoting resources to optimizing them

12:36 Stage determines the big swing versus incremental optimization split roughly 70 30 once levers are found