---
title: Running funnel experiments — problem first, big swings, document everything
slug: funnel-experiment-process-problem-first
summary: Built With Science's experiment process for web quiz, onboarding and paywalls — find the biggest drop-off, list problems before solutions, test bundled big swings when traffic is limited, write every test down (≈70% fail first time), and re-check winners months later by segment (Simply's soft-paywall lesson).
domain: mobile-apps
tags: [web2app, experimentation, conversion, paywall, onboarding, process]
related: [web-paywall-annual-default-and-trial-framing, web2app-funnel-structure, web-quiz-vs-direct-to-store, hard-vs-soft-paywalls, freemium-vs-hard-paywall-free-tier, product-experiment-process-okrs-and-activation]
sources: [yt:EyoYQU6JyV8, yt:Xv04J1cr16Q]
updated: 2026-10-24
---

# Running funnel experiments — problem first

## TL;DR
- **Step 1 — data:** in your analytics (BwS uses Amplitude), find the web/onboarding steps with the biggest drop-off *and* the most room to improve.[^1]
- **Step 2 — problems:** for the chosen page, list as many reasons it underperforms as possible (lack of trust? unclear value prop?) and agree on the biggest one — *before* any solutions.[^1][^2]
- **Step 3 — solutions:** only then brainstorm fixes for that problem (trust → testimonials, credibility) and ship the test.[^2]
- **Write it all down.** ~70% of BwS experiments fail on the first try; documentation lets you see whether execution or the problem diagnosis was wrong and iterate (their no-trial toggle went from losing to a strong lift this way).[^2][^3]
- **Big swings, bundled:** with limited traffic, test several changes against one problem together rather than button colours; strip back if it fails.[^4][^5]
- **Port winners:** messaging that wins in the quiz gets tested in the app, and vice versa.[^6]

## Details
**Re-check "winners" later by segment (Simply).** A soft paywall at the end of Simply Piano onboarding won clearly on ARPU and was celebrated for months — until analysis showed many subscribers only used the touch-screen courses (no piano at home) and were unlikely to get a year's value. Deeper segmentation found kids *did* get value; Simply kept the soft paywall only for accounts with kid profiles.[^7][^8][^9]

**Quiet launches (product side).** BwS ships new features to a ~100-person veteran group and ~1,000 beta testers for feedback, then releases publicly *without announcing*, so engagement metrics aren't inflated by launch emails; they announce only once the feature proves itself.[^10][^11]

## Disagreements / caveats
- David Barnard notes small copy changes (CTA wording) can move conversion a lot — granular tests aren't worthless, just lower priority at low traffic. See [[paywall-cta-copy]].[^12]
- 70% failure rate is BwS's own figure.

## See also
- [[web-paywall-annual-default-and-trial-framing]] — a winner from this process
- [[hard-vs-soft-paywalls]]
- [[people/ethan-ethier]], [[people/yoav-sharon]]

[^1]: yt:EyoYQU6JyV8 @ 38:04 — Ethan Ethier
[^2]: yt:EyoYQU6JyV8 @ 38:51 — Ethan Ethier
[^3]: yt:EyoYQU6JyV8 @ 56:03 — Ethan Ethier
[^4]: yt:EyoYQU6JyV8 @ 41:16 — Ethan Ethier
[^5]: yt:EyoYQU6JyV8 @ 42:31 — Ethan Ethier
[^6]: yt:EyoYQU6JyV8 @ 40:02 — Ethan Ethier
[^7]: yt:Xv04J1cr16Q @ 1:01:43 — Yoav Sharon
[^8]: yt:Xv04J1cr16Q @ 1:02:30 — Yoav Sharon
[^9]: yt:Xv04J1cr16Q @ 1:03:15 — Yoav Sharon
[^10]: yt:EyoYQU6JyV8 @ 44:17 — Ethan Ethier
[^11]: yt:EyoYQU6JyV8 @ 45:13 — Ethan Ethier
[^12]: yt:EyoYQU6JyV8 @ 41:16 — David Barnard