SSubida Skills

Esc to close · results from all notes, skills and people

Mobile apps›Web2app funnels

Running funnel experiments — problem first, big swings, document everything

Built With Science's experiment process for web quiz, onboarding and paywalls — find the biggest drop-off, list problems before solutions, test bundled big swings when traffic is limited, write every test down (≈70% fail first time), and re-check winners months later by segment (Simply's soft-paywall lesson).

Updated 2026-10-24·2 sources·web2appexperimentationconversionpaywallonboardingprocess

Running funnel experiments — problem first

TL;DR

  • Step 1 — data: in your analytics (BwS uses Amplitude), find the web/onboarding steps with the biggest drop-off and the most room to improve.1
  • Step 2 — problems: for the chosen page, list as many reasons it underperforms as possible (lack of trust? unclear value prop?) and agree on the biggest one — before any solutions.12
  • Step 3 — solutions: only then brainstorm fixes for that problem (trust → testimonials, credibility) and ship the test.2
  • Write it all down. ~70% of BwS experiments fail on the first try; documentation lets you see whether execution or the problem diagnosis was wrong and iterate (their no-trial toggle went from losing to a strong lift this way).23
  • Big swings, bundled: with limited traffic, test several changes against one problem together rather than button colours; strip back if it fails.45
  • Port winners: messaging that wins in the quiz gets tested in the app, and vice versa.6

Details

Re-check “winners” later by segment (Simply). A soft paywall at the end of Simply Piano onboarding won clearly on ARPU and was celebrated for months — until analysis showed many subscribers only used the touch-screen courses (no piano at home) and were unlikely to get a year’s value. Deeper segmentation found kids did get value; Simply kept the soft paywall only for accounts with kid profiles.789

Quiet launches (product side). BwS ships new features to a ~100-person veteran group and ~1,000 beta testers for feedback, then releases publicly without announcing, so engagement metrics aren’t inflated by launch emails; they announce only once the feature proves itself.1011

Disagreements / caveats

  • David Barnard notes small copy changes (CTA wording) can move conversion a lot — granular tests aren’t worthless, just lower priority at low traffic. See Paywall CTA and microcopy that converts.12
  • 70% failure rate is BwS’s own figure.

See also

Footnotes

  1. Why 90% of their paid traffic goes to a quiz, not the App Store @ 38:04 — Ethan Ethier ↩ ↩2

  2. Why 90% of their paid traffic goes to a quiz, not the App Store @ 38:51 — Ethan Ethier ↩ ↩2 ↩3

  3. Why 90% of their paid traffic goes to a quiz, not the App Store @ 56:03 — Ethan Ethier ↩

  4. Why 90% of their paid traffic goes to a quiz, not the App Store @ 41:16 — Ethan Ethier ↩

  5. Why 90% of their paid traffic goes to a quiz, not the App Store @ 42:31 — Ethan Ethier ↩

  6. Why 90% of their paid traffic goes to a quiz, not the App Store @ 40:02 — Ethan Ethier ↩

  7. How Simply Finally Cracked Facebook Ads with Web Funnels – Yoav Sharon, Simply @ 1:01:43 — Yoav Sharon ↩

  8. How Simply Finally Cracked Facebook Ads with Web Funnels – Yoav Sharon, Simply @ 1:02:30 — Yoav Sharon ↩

  9. How Simply Finally Cracked Facebook Ads with Web Funnels – Yoav Sharon, Simply @ 1:03:15 — Yoav Sharon ↩

  10. Why 90% of their paid traffic goes to a quiz, not the App Store @ 44:17 — Ethan Ethier ↩

  11. Why 90% of their paid traffic goes to a quiz, not the App Store @ 45:13 — Ethan Ethier ↩

  12. Why 90% of their paid traffic goes to a quiz, not the App Store @ 41:16 — David Barnard ↩