--- title: Reading store A/B test results — when to stop, applying ties, validating winners slug: aso-ab-test-reading-results-and-ties summary: Why you shouldn't stop a Google Play experiment when it first shows a winner, how to read performance bars as confidence intervals and apply "positive ties", using a Bayesian calculator for PPO, and three ways to cross-check a winner. domain: mobile-apps tags: [aso, ab-testing, statistics, google-play-experiments, product-page-optimization] related: [aso-ab-test-sample-size-and-bold-variants, ppo-limitations-and-workarounds, app-store-screenshot-ab-test-results, app-store-screenshot-production-and-ppo-setup] sources: [yt:lW0kO3KvyFE, yt:M2e6pJuA8hA, yt:fnzx89yKxHY] updated: 2026-10-24 --- # Reading store A/B test results — when to stop, applying ties, validating winners ## TL;DR - **Statistical significance ≠ validity.** Don't apply a variant the moment Google Play recommends it; first check the pre-computed sample size was reached.[^1][^2] - In a 1,000-run A/A simulation (identical variants), **771 reached 90% significance and 531 reached 95%** at some point — peeking produces false winners.[^3] - Performance bars are **confidence intervals**; the midpoint is the best-supported estimate (not "most likely" as Google's docs say). Applying ties whose bar is mostly green can keep lifting conversion.[^4][^5] - Apple PPO is Bayesian: convert impressions × conversion rate into downloads and use a **Bayesian** calculator (e.g. abtestguide.com/bayesian), applying when "chance to be best" is ≥75–80%.[^6][^7] - Treat any result as a **directional cue, not an oracle** — cross-check big decisions with an A/B/B test, a backward test, or a before/after read.[^8] ## Details **The case that changed Mateusz's mind (Klarna, 2023).** A Play experiment ran 7+ days, the variant led the whole time and the bar was fully green (+7% to +23%); ~15–20% of the conference audience would apply it.[^9] Rerun and left four weeks longer, it became a "negative tie" — the variant was the worst performer.[^10] Rule: compute minimum sample size per market *before* the test and wait for it.[^2] **Ties.** In an ASO Slack poll about a third of practitioners never apply ties.[^11] Mateusz thinks that limits optimisation: within a 95% interval every value is equally "likely" in his reading, but evidence is densest at the centre, so a −2% to +8% bar suggests ~+3%.[^4][^12] His spreadsheet applies a tie when a set share of the bar is green — 75% for a "moderate" risk appetite.[^5] He has an app where *no* test was fully positive in a year, yet conversion rose month over month from applied ties.[^13] **PPO specifics.** PPO's own read-out ("+0.13%, 1% confidence") tells you little; re-analyse it in a Bayesian tool. Don't use frequentist calculators (Evan Miller) on Bayesian PPO data.[^6][^7] Simon Thillay (AppTweak, Feb 2022) ran an A/A test right after PPO launched and already saw noise; he called the 90% confidence model prone to false positives, as AppTweak had shown for Google Play.[^14] **Google Play changes (May 2022).** Play replaced the "scaled installs" metric — a projection multiplying a 10% sample by 10, which Storemaven called imaginary because audiences behave differently at scale — with actual sample and install counts, and let you set confidence and MDE.[^15][^16] **Validation methods.**[^8] 1. **A/B/B** — two identical treatments; if they disagree, distrust the result (Simon Thillay wrote this up at AppTweak). 2. **Backward test** — apply the winner, then test the old default as the treatment. 3. **Sequential (before/after)** — compare periods, controlling for seasonality and external events. ## Disagreements / caveats - Mateusz's tie rule trades validity for velocity; it suits high-volume apps where small lifts compound, and is his practice, not a statistical standard. - His reading of intervals ("same chance anywhere in the bar") is a simplification he admits a statistician would contest.[^12] - Steve P. Young (2026) reads PPO through App Store Connect analytics and applies winners directly ([[app-store-screenshot-production-and-ppo-setup]]); none of his notes discuss peeking. ## See also - [[aso-ab-test-sample-size-and-bold-variants]] — compute the sample before you start - [[ppo-limitations-and-workarounds]] [^1]: yt:lW0kO3KvyFE @ 28:58 — Mateusz Wrzeszcz [^2]: yt:lW0kO3KvyFE @ 29:48 — Mateusz Wrzeszcz [^3]: yt:lW0kO3KvyFE @ 30:40 — Mateusz Wrzeszcz [^4]: yt:lW0kO3KvyFE @ 33:01 — Mateusz Wrzeszcz [^5]: yt:lW0kO3KvyFE @ 34:38 — Mateusz Wrzeszcz [^6]: yt:lW0kO3KvyFE @ 37:12 — Mateusz Wrzeszcz [^7]: yt:lW0kO3KvyFE @ 38:06 — Mateusz Wrzeszcz [^8]: yt:lW0kO3KvyFE @ 38:54 — Mateusz Wrzeszcz [^9]: yt:lW0kO3KvyFE @ 27:21 — Mateusz Wrzeszcz [^10]: yt:lW0kO3KvyFE @ 28:11 — Mateusz Wrzeszcz [^11]: yt:lW0kO3KvyFE @ 31:28 — Mateusz Wrzeszcz [^12]: yt:lW0kO3KvyFE @ 33:47 — Mateusz Wrzeszcz [^13]: yt:lW0kO3KvyFE @ 35:29 — Mateusz Wrzeszcz [^14]: yt:M2e6pJuA8hA @ 31:09 — Simon Thillay [^15]: yt:fnzx89yKxHY @ 10:03 — Esther Shatz [^16]: yt:fnzx89yKxHY @ 10:52 — Esther Shatz