---
title: Designing incrementality tests (duration, budget, CTV and brand)
slug: incrementality-test-design
summary: Practical rules for geo/holdout incrementality tests — run upper-funnel and CTV tests 4–6 weeks (two weeks shows nothing), give them enough budget not to be swamped, avoid launching into seasonality, and judge brand spend on a finance timeline.
domain: marketing
tags: [measurement, incrementality, geo-testing, ctv, brand-marketing, experiments]
related: [measurement-stack-attribution-incrementality-mmm, last-click-attribution-biases, when-last-click-is-good-enough, mmm-evaluation-and-open-source, evaluating-new-ad-channels, apple-ads-brand-vs-non-brand, paid-to-organic-uplift]
sources: [yt:qZj6_mJGOMg, yt:tJzzjP29f-k]
updated: 2026-10-24
---

# Designing incrementality tests

## TL;DR
- **Duration matches the channel.** For CTV and other awareness channels, a two-week test "is just not enough time to even see anything". Haus sees most CTV lift in the back half of 4–6 week tests.
- **Budget must be big enough.** A tiny long test gets absorbed by the big channels and you can't separate the effect.
- **Engineer the test.** Don't switch a channel on just as seasonality ramps up. Collinearity makes the result unreadable, and you'll have the same debate again next year.
- **Prefer geo-targetable channels.** CTV can be geo-tested cleanly. Linear TV, bought nationally, mostly can't.
- **Budget changes must be large.** Moving spend 5% teaches nothing; halve it or switch it off in test regions.
- **Brand spend is not exempt.** Test it, then rank its payback timeline against other uses of the money.

## Details
**Test length.** [[people/olivia-kory]]: Haus's big recent learning is that CTV lift arrives late. Cutting a test at 3–4 weeks "you are missing all the value".[^1] Different channels have different payback windows, so you must decide what a dollar back next month is worth versus a dollar back today.[^2] [[people/eric-seufert]]'s caveat: "good luck going to a CFO" asking for $500K over three months before any conclusion.[^2]

**Budget and design.** Seufert: a longer test on a tiny budget hits the same wall, because the effect is absorbed by bigger channels. Marketing data "suffers from collinearity anyway", so tests must be deliberately engineered.[^3] Kory's recurring failure case is linear TV. Teams plan a geo test, don't run it, turn TV on as seasonality ramps, can't see any impact, turn it off, and repeat the conversation next year.[^4] Seufert notes that channels with low minimums and flexible budgets and targeting are the ones you can test at all. A $1M campaign minimum rules out smaller companies.[^5]

**Size of the change.** In a fully digital business you can cut a channel in half or to zero in test regions. A 5% change inside wide confidence intervals tells you nothing.[^6] Many organisations won't move a channel more than 5–10% for fear of missing the quarter. That conservatism is the main practical blocker.[^7]

**CTV playbook** ([[people/michael-kaminsky]])
1. Launch small and look for any sign of life: the vendor's measurement panel (with a grain of salt) and post-checkout survey answers mentioning TV.[^8]
2. If there's a signal, scale it and run a **geographic lift test**. CTV's geo targeting makes that clean, while national linear TV doesn't extrapolate from local buys.[^9]
3. Expect wide ranges, e.g. "CPA probably $50–75". Second-screen leakage to Meta means you'll never get it precise.[^10] Decide anyway, using the range plus strategic reasons such as diversifying away from Facebook.[^11]

**Brand and upper-funnel.** Seufert argues for testing in both directions. "Brand will absolutely work" is as absurd as "brand can't work".[^12] Even if it works, weigh the time value of money and the spend needed before returns show, and rank it against other investments.[^13] Kory: finance teams care about brand only as far as it drives downstream outcomes. Untied brand spend "is going to be the first thing on the chopping block".[^14]

**Vendor-funded tests.** Newer channels (CTV sellers, AppLovin's e-commerce product) fund Haus tests with ad credits to lower the risk.[^15] This is useful, but Kory admits ad credits make objectivity a "gray area".[^16]

## Disagreements / caveats
- Long-test evidence comes from Haus's own published data. Kory admits independent research on long-term effects is thin.[^4]

[^1]: yt:qZj6_mJGOMg @ 29:37 — Olivia Kory
[^2]: yt:qZj6_mJGOMg @ 28:46 — Olivia Kory / Eric Seufert
[^3]: yt:qZj6_mJGOMg @ 30:24 — Eric Seufert
[^4]: yt:qZj6_mJGOMg @ 41:17 — Olivia Kory
[^5]: yt:qZj6_mJGOMg @ 43:40 — Eric Seufert
[^6]: yt:tJzzjP29f-k @ 28:19 — Eric Seufert
[^7]: yt:tJzzjP29f-k @ 28:19 — Michael Kaminsky
[^8]: yt:tJzzjP29f-k @ 42:34 — Michael Kaminsky
[^9]: yt:tJzzjP29f-k @ 41:48 — Michael Kaminsky
[^10]: yt:tJzzjP29f-k @ 45:02 — Michael Kaminsky
[^11]: yt:tJzzjP29f-k @ 45:54 — Michael Kaminsky
[^12]: yt:qZj6_mJGOMg @ 38:49 — Eric Seufert
[^13]: yt:qZj6_mJGOMg @ 39:37 — Eric Seufert
[^14]: yt:qZj6_mJGOMg @ 36:13 — Olivia Kory
[^15]: yt:qZj6_mJGOMg @ 37:04 — Olivia Kory
[^16]: yt:qZj6_mJGOMg @ 04:54 — Olivia Kory