SSubida Skills

Esc to close · results from all notes, skills and people

Mobile apps›App Store Optimization

Reading store A/B test results — when to stop, applying ties, validating winners

Why you shouldn't stop a Google Play experiment when it first shows a winner, how to read performance bars as confidence intervals and apply "positive ties", using a Bayesian calculator for PPO, and three ways to cross-check a winner.

Updated 2026-10-24·3 sources·asoab-testingstatisticsgoogle-play-experimentsproduct-page-optimization

Reading store A/B test results — when to stop, applying ties, validating winners

TL;DR

  • Statistical significance ≠ validity. Don’t apply a variant the moment Google Play recommends it; first check the pre-computed sample size was reached.12
  • In a 1,000-run A/A simulation (identical variants), 771 reached 90% significance and 531 reached 95% at some point — peeking produces false winners.3
  • Performance bars are confidence intervals; the midpoint is the best-supported estimate (not “most likely” as Google’s docs say). Applying ties whose bar is mostly green can keep lifting conversion.45
  • Apple PPO is Bayesian: convert impressions × conversion rate into downloads and use a Bayesian calculator (e.g. abtestguide.com/bayesian), applying when “chance to be best” is ≥75–80%.67
  • Treat any result as a directional cue, not an oracle — cross-check big decisions with an A/B/B test, a backward test, or a before/after read.8

Details

The case that changed Mateusz’s mind (Klarna, 2023). A Play experiment ran 7+ days, the variant led the whole time and the bar was fully green (+7% to +23%); ~15–20% of the conference audience would apply it.9 Rerun and left four weeks longer, it became a “negative tie” — the variant was the worst performer.10 Rule: compute minimum sample size per market before the test and wait for it.2

Ties. In an ASO Slack poll about a third of practitioners never apply ties.11 Mateusz thinks that limits optimisation: within a 95% interval every value is equally “likely” in his reading, but evidence is densest at the centre, so a −2% to +8% bar suggests ~+3%.412 His spreadsheet applies a tie when a set share of the bar is green — 75% for a “moderate” risk appetite.5 He has an app where no test was fully positive in a year, yet conversion rose month over month from applied ties.13

PPO specifics. PPO’s own read-out (“+0.13%, 1% confidence”) tells you little; re-analyse it in a Bayesian tool. Don’t use frequentist calculators (Evan Miller) on Bayesian PPO data.67 Simon Thillay (AppTweak, Feb 2022) ran an A/A test right after PPO launched and already saw noise; he called the 90% confidence model prone to false positives, as AppTweak had shown for Google Play.14

Google Play changes (May 2022). Play replaced the “scaled installs” metric — a projection multiplying a 10% sample by 10, which Storemaven called imaginary because audiences behave differently at scale — with actual sample and install counts, and let you set confidence and MDE.1516

Validation methods.8

  1. A/B/B — two identical treatments; if they disagree, distrust the result (Simon Thillay wrote this up at AppTweak).
  2. Backward test — apply the winner, then test the old default as the treatment.
  3. Sequential (before/after) — compare periods, controlling for seasonality and external events.

Disagreements / caveats

  • Mateusz’s tie rule trades validity for velocity; it suits high-volume apps where small lifts compound, and is his practice, not a statistical standard.
  • His reading of intervals (“same chance anywhere in the bar”) is a simplification he admits a statistician would contest.12
  • Steve P. Young (2026) reads PPO through App Store Connect analytics and applies winners directly (Producing screenshots and running Product Page Optimization tests); none of his notes discuss peeking.

See also

Footnotes

  1. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 28:58 — Mateusz Wrzeszcz ↩

  2. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 29:48 — Mateusz Wrzeszcz ↩ ↩2

  3. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 30:40 — Mateusz Wrzeszcz ↩

  4. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 33:01 — Mateusz Wrzeszcz ↩ ↩2

  5. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 34:38 — Mateusz Wrzeszcz ↩ ↩2

  6. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 37:12 — Mateusz Wrzeszcz ↩ ↩2

  7. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 38:06 — Mateusz Wrzeszcz ↩ ↩2

  8. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 38:54 — Mateusz Wrzeszcz ↩ ↩2

  9. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 27:21 — Mateusz Wrzeszcz ↩

  10. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 28:11 — Mateusz Wrzeszcz ↩

  11. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 31:28 — Mateusz Wrzeszcz ↩

  12. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 33:47 — Mateusz Wrzeszcz ↩ ↩2

  13. Mastering A/B Testing for ASO | Mateusz Wrzeszcz | ASO Conference 2023 @ 35:29 — Mateusz Wrzeszcz ↩

  14. Mastering ASO in 2022 with Simon Thillay of AppTweak @ 31:09 — Simon Thillay ↩

  15. What Are the New Marketing and ASO Capabilities in Google Play, and What Do They Mean? with Esthe... @ 10:03 — Esther Shatz ↩

  16. What Are the New Marketing and ASO Capabilities in Google Play, and What Do They Mean? with Esthe... @ 10:52 — Esther Shatz ↩