The value of a test is decided before it runs: a hypothesis, one variant, a metric agreed in advance, and a sample size that makes the result mean something. Tests fail most often not because the variant lost but because the team could not say beforehand what winning looked like. Running the experiment on the same event store as the analytics matters here: if the test tool and the analytics disagree on what a conversion is, the result is unarguable in the worst sense.
More in Growth
- AARRR frameworkAcquisition, Activation, Retention, Referral, Revenue: five stages used to decide which metric a team should be arguing about this quarter.
- ActivationThe moment a new user first gets the value the product promised. Everything before it is setup; everything after it is retention.
- Activation rateThe share of new users who reach that moment. The single number most onboarding work is trying to move.
- AttributionAssigning an install or a conversion to the campaign, channel or referrer that produced it.
- Average Revenue Per User (ARPU)Revenue in a period divided by active users in that period. Useful next to retention; misleading on its own.
- Churn rateThe share of users or revenue lost in a period. Its shape over time matters more than the number in any single month.