Store A/B tests — sample size, MDE and why bold variants win

TL;DR

• Store tests (Google Play experiments, Apple PPO) predict future performance with a confidence level — unlike ad-platform “A/B tests” that just report observed results — so they need a pre-computed sample size.¹
• Your variant’s boldness is your detectable effect. On Klarna-scale traffic, ~1 week detects only a ~6% relative lift, ~4 weeks ~3%; a subtle tweak (<1% MDE) may never conclude.²³
• Tier markets by traffic and conversion: no tests in low-traffic markets; only very bold variants (slightly lower confidence) in moderate ones; fine-grained tests only in high-traffic markets.⁴⁵
• Test one hypothesis, not one element — changing 3–4 screenshots that all express the same idea is fine.⁶
• Prioritise the first-impression frame (first ~4 screenshots on Play, first 2–3 on App Store) and ask: would a person notice the difference in 3 seconds?⁷
• Don’t “test everything” — build the backlog from qualitative insights first, competitor and blue-sky ideas last.⁸⁹

Details

Source context. Mateusz Wrzeszcz (ASO lead at Klarna, ex-Phiture) at ASO Conference 2023; Marina Roglic (Popcore) at ASO Conference 2022; Storemaven hosts on Google Play’s May 2022 experiment changes.

MDE in plain terms. Mateusz’s working definition: the smallest relative difference in conversion rate between default and treatment you want to reliably detect. Unlike pharma or a website redesign, ASO variants are cheap, so even +0.2% is worth having — the constraint is reliability, not cost.¹⁰ Ask: “how bold must this change be to reach validity in an acceptable time?”³

Bold vs timid, real examples (as of 2023). Coinbase and Spotify — with hundreds of thousands of installs a week — still ran fully redesigned sets (captions, fonts, features, colours): Mateusz’s guess is an MDE of 3–4%. HBO Max changing only screenshot order and Amazon Music changing “millions of songs” to “90 million songs” are, in his estimate, below 1% MDE.¹¹¹² He suggests a radical full redesign in the backlog at least once a quarter: fewer wins than trivial tests, but bigger ones.¹³

Tools. Evan Miller’s sample-size calculator is the classic frequentist one; Google Play has had a built-in calculator in experiment setup since late June 2022, and Mateusz published a Play calculator that reads a CSV export from the console.¹⁴⁵ Since May 2022 Play lets you choose confidence level and MDE yourself — Storemaven warned this makes it tempting to configure a tiny sample and “conclude” in a day (e.g. 2,000 users), which misses weekday/weekend differences and can damage organic conversion; they call high-count, low-quality testing a “fake testing culture”.¹⁵¹⁶¹⁷

Back-of-envelope for PPO. Marina estimated required impressions from the last 30-day average: one en-US test needed 300,000+ impressions and reached 99% confidence; a German test needed ~20,000 and, even with 30,000+, stayed around 20% — both were significant design changes.¹⁸¹⁹

Don’t run one creative everywhere. Expecting the same variant to reach validity in the US and the Czech Republic in the same period is a mistake.⁶

Disagreements / caveats

• Steve P. Young (2026) advises one variable per test (Screenshot A/B tests — themes to test and published results); Mateusz argues single-element tests “won’t get you far” unless you have huge traffic. Low-traffic apps should side with Mateusz.⁶
• The 6%/3% figures come from one Klarna-traffic chart; your numbers depend on baseline conversion and volume.
• Play experiments are frequentist and PPO Bayesian (Mateusz, 2023) — calculators are not interchangeable; see Reading store A/B test results — when to stop, applying ties, validating winners.²⁰

See also

• Reading store A/B test results — when to stop, applying ties, validating winners — when to stop and what to apply
• Apple PPO limitations and how practitioners work around them — getting tests to run at all on iOS

Footnotes

1. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 03:09 — Mateusz Wrzeszcz ↩

2. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 13:53 — Mateusz Wrzeszcz ↩

3. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 12:59 — Mateusz Wrzeszcz ↩ ↩²

4. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 20:35 — Mateusz Wrzeszcz ↩

5. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 21:29 — Mateusz Wrzeszcz ↩ ↩²

6. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 24:49 — Mateusz Wrzeszcz ↩ ↩² ↩³

7. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 23:52 — Mateusz Wrzeszcz ↩

8. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 25:35 — Mateusz Wrzeszcz ↩

9. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 09:38 — Mateusz Wrzeszcz ↩

10. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 12:09 — Mateusz Wrzeszcz ↩

11. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 16:29 — Mateusz Wrzeszcz ↩

12. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 18:03 — Mateusz Wrzeszcz ↩

13. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 23:07 — Mateusz Wrzeszcz ↩

14. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 19:43 — Mateusz Wrzeszcz ↩

15. What Are the New Marketing and ASO Capabilities in Google Play, and What Do They Mean? with Esthe... @ 12:32 — Jonathan Fishman ↩

16. What Are the New Marketing and ASO Capabilities in Google Play, and What Do They Mean? with Esthe... @ 13:19 — Esther Shatz ↩

17. What Are the New Marketing and ASO Capabilities in Google Play, and What Do They Mean? with Esthe... @ 14:51 — Jonathan Fishman ↩

18. Making the most out of PPOs | Marina Roglic | ASO Conference 2022 @ 13:00 — Marina Roglic ↩

19. Making the most out of PPOs | Marina Roglic | ASO Conference 2022 @ 13:48 — Marina Roglic ↩

20. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 04:47 — Mateusz Wrzeszcz ↩