A/B testing, or split testing, is a method of comparing two versions of a feature, screen, or flow to see which one performs better on a specific metric. One group of users sees Version A (the control), another sees Version B (the variation). You measure the difference and act on the result.
For mobile games and apps, A/B testing is how you turn assumptions into decisions — whether you're optimizing onboarding, pricing, push notification copy, or paywall timing.
This glossary entry covers how to set up and analyze an A/B test, what metrics to track, and how to avoid the most common mistakes.
What Is A/B Testing?
A/B testing is a controlled experiment where two versions of a product element are shown to separate groups of users simultaneously. Version A is the original. Version B is the variation you want to test. Users are assigned randomly — that's what makes the results valid.
The logic is simple: if both groups behave the same except for the one thing you changed, any difference in outcome is caused by that change.
For example, you suspect your paywall is appearing too early. You test two versions — one showing the offer at Level 3, and another at Level 5. After two weeks, you compare paying conversion. The winning version becomes your new default.
Learn more about A/B testing in LiveOps →
How to Set Up an A/B Test
1. Define one variable. Test one change at a time. If you change the button color and the button text simultaneously, you won't know which one drove the result.
2. Set your success metric before you start. Decide upfront what you're measuring — paying conversion, D1 retention, ARPU, tutorial completion rate. Changing the metric mid-test invalidates the results.
3. Calculate your sample size. This is the step most teams skip — and it's why so many A/B tests produce unreliable results. The smaller the expected effect, the larger the sample you need. Running underpowered tests is one of the most common reasons A/B results don't hold up after shipping.
Sample size requirements vary depending on your baseline metric, expected effect size, and traffic volume. These are approximate guidelines based on standard statistical practice.
4. Run the test long enough. A minimum of 7 days covers weekly behavior cycles. For metrics like D7 retention or ARPU, you need at least 2–3 weeks. Stopping early because early results look promising is one of the most common A/B testing mistakes.
5. Assign users randomly. Random assignment is what separates a valid experiment from a biased one. In devtodev, users are assigned to test groups automatically — you define the split, and the platform handles randomization.
How to Analyze A/B Testing Results
Once your test has run long enough, look at three things:
Primary metric. Did Version B move the needle on the metric you defined at the start? Focus here first — ARPU, paying conversion, retention, or whichever KPI you set.
Statistical significance. A p-value below 0.05 means there's less than a 5% chance the result is random. devtodev calculates statistical significance and probability to be best automatically — you can see both directly in the results table alongside your primary metrics.
Secondary metrics. A version that improves paying conversion but tanks D7 retention isn't a win. Always check that your variation didn't improve one metric by damaging another.
Segment the results. Break down results by user type — paying vs. non-paying, new vs. returning, by acquisition channel. A change that hurts casual users but significantly lifts high spenders might still be worth shipping.
Learn more about A/B testing essentials: strategies, metrics, and AI →
What to Do After the Test
Version B wins — ship it to 100% of users. Document what you tested, what changed, and by how much. That record becomes your product knowledge base.
No significant difference — that's a valid result too. You learned that this particular change doesn't affect the metric. Move on to the next hypothesis.
Version B loses — keep Version A. Understand why B underperformed before writing off the hypothesis entirely. Sometimes the idea is right, but the execution is wrong.
Results conflict across segments — consider a targeted rollout. Ship Version B only to the segment where it won.
Takeaways
- A/B testing compares two versions of a feature by showing them to separate user groups simultaneously — the only variable that changes is the one you're testing.
- Define your success metric before the test starts. Changing it mid-test invalidates results.
- Sample size matters — running underpowered tests is the most common reason results don't hold up after shipping.
- Run tests for at least 7 days. For retention or ARPU metrics — 2–3 weeks.
- Check secondary metrics after the test. A version that improves conversion but hurts retention isn't a win.
- Segment results by user type — the winning version for casual users may not be the winner for high spenders.
Find out which version of your product actually works — without guesswork. Explore devtodev →