Do internal product metrics lose their validity once teams optimize directly for them?

Shreyas Doshi · Aug 3, 2022

It is a mistake for teams to set an adoption metric and then try to improve it directly; they should instead measure the usage behaviors that drive adoption

Adoption doesn't happen on its own — it's the end result of preconditions and of the activities users perform, so only by understanding what usage leads to adoption can you address the inputs

Scope: stated in the context of advising early-stage teams, products or startups

6:27 20 Product: Shreyas Doshi on The Three Different Types of Product Leaders and How To Hire Them, The 6 Different Product Metrics You Need To Know and What Good is For Each of Them & Table Stakes Features vs Wow Features; What To Prioritise

Harry Stebbings · Mar 8, 2023

The hardest part of objective and goal setting is making goals non-gameable.

He set his investment team a goal of meeting 20 companies a week and realised anyone can hit it by meeting trivial companies like the corner shop.

15:25 20Product: Shopify's VP Product on Why the Founder is Always the Head of Product, What Makes Truly Special Product Managers, Why The Majority of Product Managers Need to Change, Why Top-Down Decision-Making in Product is Good & How Shopify Will Be Bigger

Harry Stebbings · Mar 8, 2023

Even goals tied to converted dollars or acquisition remain gameable, because teams can hit them with low-quality leads that churn once retention is not part of the goal.

You can do whatever it takes to get customers over the line and still hit your numbers while the leads churn after two months.

16:17 20Product: Shopify's VP Product on Why the Founder is Always the Head of Product, What Makes Truly Special Product Managers, Why The Majority of Product Managers Need to Change, Why Top-Down Decision-Making in Product is Good & How Shopify Will Be Bigger

Harry Stebbings · Mar 29, 2023

Most goals set for winning are gameable — a growth team can hit a thousand-user target with paid ads and churning users

The metric can be satisfied without the underlying business outcome being achieved

25:22 20Product: Nubank's CPO on Why Product is 90% Science and 10% Art, Why Execution is Overrated and Strategic Clarity is Under-Appreciated, Why You Should Never Fall in Love With Your Own Ideas & Nubank's Biggest Product Challenges Scaling to 80M Users

Martin Gontovnikas · Feb 14, 2024

Rising email open and click rates can coexist with worse retention, so engagement metrics are a misleading measure of onboarding email quality

A holdout test where 10% of users got no emails showed the no-email group was 20% more retained; better subject lines drove opens but the confusing email bodies pushed users out of the product

Scope: based on their own holdout experiment

60:50 20Growth: How to Master Product-Led-Growth, The Biggest Mistakes Startups Make When Scaling into Enterprise, How to Assess "Bets" in Growth; Which to Take and Which to Not with Gonto, Interim CMO @ Vercel

Harry Stebbings · Feb 14, 2024

The gamification of metrics is the most dangerous thing in companies

Whatever metric you name can be trivially gamed — ask for sign-ups and someone just buys paid ads

62:29 20Growth: How to Master Product-Led-Growth, The Biggest Mistakes Startups Make When Scaling into Enterprise, How to Assess "Bets" in Growth; Which to Take and Which to Not with Gonto, Interim CMO @ Vercel

Anton Osika · Aug 18, 2025

Internal product metrics lose their meaning as soon as you start optimizing for them — e.g. thumbs-up rates on messages can be hacked by making the AI tell jokes or ask users to click

Optimizing for a metric turns it into a hackable target rather than a measure of value

Scope: given as one example

55:23 20VC: Lovable CEO Anton Osika on $120M in ARR in 7 Months | The Honest Truth About Defensibility and Unit Economics for AI Startups | The State of Foundation Models: Long Grok, Short OpenAI, Why | Replit vs Lovable vs Bolt: What Happens

David Meyer · Nov 1, 2023

It is important to make sure the thing you are measuring is not gameable

15:18 20Product: Why You Should Not Go Into Product Management, Why the CEO is Always the CPO, How to Build the Best Product Teams & Why You Should Hire People Who Aren't In Product Already with Databricks SVP Product, David Meyer

Raman Malik · Nov 15, 2024

The defense against metric gaming is to keep monitoring whether the metric still correlates with retention after you've optimized it, and to hold a rounded set of metrics rather than staring down one.

If the correlation starts to break after you tinker with the metric, you're not being honest with yourself about whether you're actually helping retention.

16:17 20Growth: Inside Perplexity's Growth Machine: What Worked, What Did Not Work | Why Paid Acquisition is a Drug and Brand Marketing is BS | The Good, Bad and Ugly of A/B Tests and Why Micro-Optimisations are Under-Rated with Raman Malik

David Meyer · Nov 1, 2023

Metrics serve only to reveal blind spots and where to ask questions; hitting the target is not the goal, and often the correct response to variance is to change the target

Targets are set from hypotheses formed before shipping; once you disprove those hypotheses the target itself was wrong, and as soon as you report on a metric it becomes the goal

11:21 Metrics exist to reveal blind spots not to be hit targets should change when hypotheses are disproven

David Meyer · Nov 1, 2023

Target-adjustment cadence should scale with company size: a startup of one revises continuously, a 6,000-person company should rarely revise more than annually — unless the situation is existential

At scale, many people have built organizationally complex flywheels to hit the goal, so frequent changes create grumpiness; but waiting six months on an existential issue is stupid

12:44 Target adjustment cadence scales with company size

David Meyer · Nov 1, 2023

Data-driven product management usually fails because teams let the data become the thing itself; the real cost is the human energy wasted producing numbers that never become actionable

Without good telemetry or BI, humans spend time counting things to put in emails nobody acts on; he admits his own daily dashboard-checking is like infinite scroll and adds no value

14:02 Data driven pm fails when data becomes the end goal wasting human energy on unused metrics

David Meyer · Nov 1, 2023

Active users and time spent are among the hardest metrics to game, because time is precious so usage implies real value to someone

If people spend time in your product there is value there, even if that value never converts to revenue — that's a separate business model question

15:45 Time spent and active usage are hard to game since time is precious

David Meyer · Nov 1, 2023

Product managers who ship a product and then don't obsessively observe its behaviour in the wild through data are insane

You spent all that energy and willpower creating the thing; not watching exactly how it interacts with the world is crazy

23:41 Obsessively observing shipped products behavior in the wild is mandatory

Glen Coates · Mar 8, 2023

Sales and marketing teams push for goals defined on metrics they fully control, which leads marketing to own top-of-funnel numbers rather than bottom-of-funnel outcomes and produces misaligned incentives.

Because they are goal-oriented and compensated on numbers, they want the number they are paid on to be one they actually control.

15:43 Gtm teams choose goals they fully control causing misaligned incentives