--- title: Store A/B tests — sample size, MDE and why bold variants win slug: aso-ab-test-sample-size-and-bold-variants summary: How much traffic a screenshot/icon test needs, what minimum detectable effect (MDE) means for ASO, Klarna's market-tier rule for where to test, and why "one hypothesis, bold change" beats "one small change at a time". domain: mobile-apps tags: [aso, ab-testing, statistics, product-page-optimization, google-play-experiments, screenshots] related: [aso-ab-test-reading-results-and-ties, ppo-limitations-and-workarounds, app-store-screenshot-ab-test-results, app-store-screenshot-production-and-ppo-setup, aso-audit-and-prioritization] sources: [yt:lW0kO3KvyFE, yt:-YsOukrqcWU, yt:fnzx89yKxHY] updated: 2026-10-24 --- # Store A/B tests — sample size, MDE and why bold variants win ## TL;DR - Store tests (Google Play experiments, Apple PPO) **predict** future performance with a confidence level — unlike ad-platform "A/B tests" that just report observed results — so they need a pre-computed sample size.[^1] - **Your variant's boldness is your detectable effect.** On Klarna-scale traffic, ~1 week detects only a ~6% relative lift, ~4 weeks ~3%; a subtle tweak (<1% MDE) may never conclude.[^2][^3] - **Tier markets by traffic and conversion:** no tests in low-traffic markets; only very bold variants (slightly lower confidence) in moderate ones; fine-grained tests only in high-traffic markets.[^4][^5] - **Test one hypothesis, not one element** — changing 3–4 screenshots that all express the same idea is fine.[^6] - Prioritise the **first-impression frame** (first ~4 screenshots on Play, first 2–3 on App Store) and ask: would a person notice the difference in 3 seconds?[^7] - Don't "test everything" — build the backlog from qualitative insights first, competitor and blue-sky ideas last.[^8][^9] ## Details **Source context.** Mateusz Wrzeszcz (ASO lead at Klarna, ex-Phiture) at ASO Conference 2023; Marina Roglic (Popcore) at ASO Conference 2022; Storemaven hosts on Google Play's May 2022 experiment changes. **MDE in plain terms.** Mateusz's working definition: the smallest relative difference in conversion rate between default and treatment you want to *reliably* detect. Unlike pharma or a website redesign, ASO variants are cheap, so even +0.2% is worth having — the constraint is reliability, not cost.[^10] Ask: "how bold must this change be to reach validity in an acceptable time?"[^3] **Bold vs timid, real examples (as of 2023).** Coinbase and Spotify — with hundreds of thousands of installs a week — still ran fully redesigned sets (captions, fonts, features, colours): Mateusz's guess is an MDE of 3–4%. HBO Max changing only screenshot order and Amazon Music changing "millions of songs" to "90 million songs" are, in his estimate, below 1% MDE.[^11][^12] He suggests a radical full redesign in the backlog at least **once a quarter**: fewer wins than trivial tests, but bigger ones.[^13] **Tools.** Evan Miller's sample-size calculator is the classic frequentist one; Google Play has had a built-in calculator in experiment setup since late June 2022, and Mateusz published a Play calculator that reads a CSV export from the console.[^14][^5] Since May 2022 Play lets you choose confidence level and MDE yourself — Storemaven warned this makes it tempting to configure a tiny sample and "conclude" in a day (e.g. 2,000 users), which misses weekday/weekend differences and can damage organic conversion; they call high-count, low-quality testing a "fake testing culture".[^15][^16][^17] **Back-of-envelope for PPO.** Marina estimated required impressions from the last 30-day average: one en-US test needed 300,000+ impressions and reached 99% confidence; a German test needed ~20,000 and, even with 30,000+, stayed around 20% — both were significant design changes.[^18][^19] **Don't run one creative everywhere.** Expecting the same variant to reach validity in the US and the Czech Republic in the same period is a mistake.[^6] ## Disagreements / caveats - Steve P. Young (2026) advises one variable per test ([[app-store-screenshot-ab-test-results]]); Mateusz argues single-element tests "won't get you far" unless you have huge traffic. Low-traffic apps should side with Mateusz.[^6] - The 6%/3% figures come from one Klarna-traffic chart; your numbers depend on baseline conversion and volume. - Play experiments are frequentist and PPO Bayesian (Mateusz, 2023) — calculators are not interchangeable; see [[aso-ab-test-reading-results-and-ties]].[^20] ## See also - [[aso-ab-test-reading-results-and-ties]] — when to stop and what to apply - [[ppo-limitations-and-workarounds]] — getting tests to run at all on iOS [^1]: yt:lW0kO3KvyFE @ 03:09 — Mateusz Wrzeszcz [^2]: yt:lW0kO3KvyFE @ 13:53 — Mateusz Wrzeszcz [^3]: yt:lW0kO3KvyFE @ 12:59 — Mateusz Wrzeszcz [^4]: yt:lW0kO3KvyFE @ 20:35 — Mateusz Wrzeszcz [^5]: yt:lW0kO3KvyFE @ 21:29 — Mateusz Wrzeszcz [^6]: yt:lW0kO3KvyFE @ 24:49 — Mateusz Wrzeszcz [^7]: yt:lW0kO3KvyFE @ 23:52 — Mateusz Wrzeszcz [^8]: yt:lW0kO3KvyFE @ 25:35 — Mateusz Wrzeszcz [^9]: yt:lW0kO3KvyFE @ 09:38 — Mateusz Wrzeszcz [^10]: yt:lW0kO3KvyFE @ 12:09 — Mateusz Wrzeszcz [^11]: yt:lW0kO3KvyFE @ 16:29 — Mateusz Wrzeszcz [^12]: yt:lW0kO3KvyFE @ 18:03 — Mateusz Wrzeszcz [^13]: yt:lW0kO3KvyFE @ 23:07 — Mateusz Wrzeszcz [^14]: yt:lW0kO3KvyFE @ 19:43 — Mateusz Wrzeszcz [^15]: yt:fnzx89yKxHY @ 12:32 — Jonathan Fishman [^16]: yt:fnzx89yKxHY @ 13:19 — Esther Shatz [^17]: yt:fnzx89yKxHY @ 14:51 — Jonathan Fishman [^18]: yt:-YsOukrqcWU @ 13:00 — Marina Roglic [^19]: yt:-YsOukrqcWU @ 13:48 — Marina Roglic [^20]: yt:lW0kO3KvyFE @ 04:47 — Mateusz Wrzeszcz