SSubida Skills

Esc to close · results from all notes, skills and people

Marketing›Measurement

Designing incrementality tests (duration, budget, CTV and brand)

Practical rules for geo/holdout incrementality tests — run upper-funnel and CTV tests 4–6 weeks (two weeks shows nothing), give them enough budget not to be swamped, avoid launching into seasonality, and judge brand spend on a finance timeline.

Updated 2026-10-24·2 sources·measurementincrementalitygeo-testingctvbrand-marketingexperiments

Designing incrementality tests

TL;DR

  • Duration matches the channel. For CTV and other awareness channels, a two-week test “is just not enough time to even see anything”. Haus sees most CTV lift in the back half of 4–6 week tests.
  • Budget must be big enough. A tiny long test gets absorbed by the big channels and you can’t separate the effect.
  • Engineer the test. Don’t switch a channel on just as seasonality ramps up. Collinearity makes the result unreadable, and you’ll have the same debate again next year.
  • Prefer geo-targetable channels. CTV can be geo-tested cleanly. Linear TV, bought nationally, mostly can’t.
  • Budget changes must be large. Moving spend 5% teaches nothing; halve it or switch it off in test regions.
  • Brand spend is not exempt. Test it, then rank its payback timeline against other uses of the money.

Details

Test length. Olivia Kory: Haus’s big recent learning is that CTV lift arrives late. Cutting a test at 3–4 weeks “you are missing all the value”.1 Different channels have different payback windows, so you must decide what a dollar back next month is worth versus a dollar back today.2 Eric Seufert‘s caveat: “good luck going to a CFO” asking for $500K over three months before any conclusion.2

Budget and design. Seufert: a longer test on a tiny budget hits the same wall, because the effect is absorbed by bigger channels. Marketing data “suffers from collinearity anyway”, so tests must be deliberately engineered.3 Kory’s recurring failure case is linear TV. Teams plan a geo test, don’t run it, turn TV on as seasonality ramps, can’t see any impact, turn it off, and repeat the conversation next year.4 Seufert notes that channels with low minimums and flexible budgets and targeting are the ones you can test at all. A $1M campaign minimum rules out smaller companies.5

Size of the change. In a fully digital business you can cut a channel in half or to zero in test regions. A 5% change inside wide confidence intervals tells you nothing.6 Many organisations won’t move a channel more than 5–10% for fear of missing the quarter. That conservatism is the main practical blocker.7

CTV playbook (Michael Kaminsky)

  1. Launch small and look for any sign of life: the vendor’s measurement panel (with a grain of salt) and post-checkout survey answers mentioning TV.8
  2. If there’s a signal, scale it and run a geographic lift test. CTV’s geo targeting makes that clean, while national linear TV doesn’t extrapolate from local buys.9
  3. Expect wide ranges, e.g. “CPA probably $50–75”. Second-screen leakage to Meta means you’ll never get it precise.10 Decide anyway, using the range plus strategic reasons such as diversifying away from Facebook.11

Brand and upper-funnel. Seufert argues for testing in both directions. “Brand will absolutely work” is as absurd as “brand can’t work”.12 Even if it works, weigh the time value of money and the spend needed before returns show, and rank it against other investments.13 Kory: finance teams care about brand only as far as it drives downstream outcomes. Untied brand spend “is going to be the first thing on the chopping block”.14

Vendor-funded tests. Newer channels (CTV sellers, AppLovin’s e-commerce product) fund Haus tests with ad credits to lower the risk.15 This is useful, but Kory admits ad credits make objectivity a “gray area”.16

Disagreements / caveats

  • Long-test evidence comes from Haus’s own published data. Kory admits independent research on long-term effects is thin.4

Footnotes

  1. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 29:37 — Olivia Kory ↩

  2. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 28:46 — Olivia Kory / Eric Seufert ↩ ↩2

  3. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 30:24 — Eric Seufert ↩

  4. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 41:17 — Olivia Kory ↩ ↩2

  5. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 43:40 — Eric Seufert ↩

  6. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 28:19 — Eric Seufert ↩

  7. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 28:19 — Michael Kaminsky ↩

  8. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 42:34 — Michael Kaminsky ↩

  9. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 41:48 — Michael Kaminsky ↩

  10. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 45:02 — Michael Kaminsky ↩

  11. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 45:54 — Michael Kaminsky ↩

  12. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 38:49 — Eric Seufert ↩

  13. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 39:37 — Eric Seufert ↩

  14. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 36:13 — Olivia Kory ↩

  15. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 37:04 — Olivia Kory ↩

  16. S07, E21: AppsFlyer and the new dynamics of mobile attribution (with Olivia Kory) @ 04:54 — Olivia Kory ↩