SSubida Skills

Esc to close · results from all notes, skills and people

Marketing›Measurement

Evaluating a marketing mix model (and whether Robyn/Meridian are biased)

Assume any MMM is wrong until it passes three checks — parameter recovery on simulated data, out-of-sample forecasting of a period with real budget changes, and agreement with lift tests; open-source Robyn and Meridian are not rigged for Meta/Google, but most MMM vendors are thin wrappers around them.

Updated 2026-10-24·2 sources·measurementmmmrobynmeridianeconometricsvendor-selectionai-analysis

Evaluating a marketing mix model

TL;DR

  • Default assumption: the MMM is wrong. Inferring causality from aggregate time series is “almost impossibly difficult”.
  • Before buying or building one, set up the evaluation:
    1. Parameter recovery. Simulate realistic data where you know the truth and check that the model recovers it.
    2. Holdout forecast. Train on data up to two months ago and predict the last two months, ideally a period when budgets actually changed.
    3. Lift-test calibration. Does it agree with your geo/holdout experiments, past and new?
  • Instability is the warning sign. Rerunning a week later, changing a setting, or switching libraries flips the “best channel”. That is a random number generator.
  • Robyn (Meta) and Meridian (Google) aren’t rigged. The code is open, there are no pre-trained weights, and you can swap channel labels to check. More than 80% of MMM vendors are reportedly wrappers around them.
  • Judge vendors on whether the model is right, not on dashboards.

Details

Why skepticism is the default. Michael Kaminsky (Recast, an MMM vendor): simulate data with realistic properties, such as performance that drifts as creative changes, correlated channels and seasonality. Most MMMs then fail to return the true channel effects.1 In practice teams see results change week to week, or contradict well-run Geolift or user-level experiments, and “the shine is starting to wear off”. He thinks the industry has passed “peak MMM”.23 Run the same data through two MMMs, or Meridian with two settings, and one says Facebook is best while the other says TikTok.4

The three checks, in order.56

  1. Parameter recovery. Encode how you believe your business works (lag lengths, how much channel performance moves over time), simulate, and fit.
  2. Predictive holdout. Withhold the last two months from the vendor or internal team and ask for a forecast. “If it’s terrible, then you shouldn’t trust it for anything else.”7
  3. Experiment agreement. If the model says TV is great, does a lift test agree?

The work is real: it takes code, there are few tools for it, and it’s like building leakage-free validation in ML but with slower feedback.7

Feedback speed. MMM estimates are wide, e.g. Facebook CPA “between $50 and $75”. A forecast is hard to falsify within a week unless spend changes a lot.8 All-digital businesses with cheap or free first purchases (games, fitness apps) have short lags and get faster feedback. Even so, budget moves must be large.9

Open source and “grading their own homework”. Open source means anyone can read, change and run the code. These are frameworks trained only on your data, with no starting weights.10 Kaminsky’s sanity tests: have an LLM (Codex, Claude) review the repo for anything favouring Meta, or swap the “Facebook” and “Google” labels and rerun; nothing changes.11 His explanation for the release: agency and consultancy MMMs were often structurally biased toward TV, the agencies’ historical cash cow, and against digital. The open libraries are a counterweight, and better than “some random regression” from an agency.12 Thomas Petit relays that over 80% of MMM companies are a layer on Meridian or Robyn.13

Why so many vendors. Cloud automation replaced analyst-built decks, open-source models appeared, and ATT made marketers anxious, so many vendors launched. Buyers can’t evaluate correctness, so they compare dashboards instead.14 Kaminsky rejects the claim that MMM is a crystal ball that replaces attribution.15

AI changes who can do this. Marketing leads now pull data and run analyses with Claude or Codex. One team that was blocked for months got their data assembled in 30 minutes. The value is in finding inconsistencies in their own beliefs.16

Disagreements / caveats

  • Kaminsky’s validation bar, while correct, favours sophisticated (paid) MMM. Petit says no MMM solution is perfect and clients keep switching.13
  • An MMM is channel-agnostic. You can add any new channel’s daily spend or impressions, which is its main structural advantage.17

Footnotes

  1. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 12:45 — Michael Kaminsky ↩

  2. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 13:38 — Michael Kaminsky ↩

  3. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 35:09 — Michael Kaminsky ↩

  4. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 20:51 — Michael Kaminsky ↩

  5. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 21:39 — Michael Kaminsky ↩

  6. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 22:27 — Michael Kaminsky ↩

  7. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 24:04 — Michael Kaminsky ↩ ↩2

  8. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 27:26 — Michael Kaminsky ↩

  9. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 26:38 — Michael Kaminsky ↩

  10. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 03:28 — Michael Kaminsky ↩

  11. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 06:46 — Michael Kaminsky ↩

  12. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 10:08 — Michael Kaminsky ↩

  13. S07, E30: MDM Mailbag #8 (with Thomas Petit) @ 05:46 — Thomas Petit ↩ ↩2

  14. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 31:00 — Michael Kaminsky ↩

  15. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 32:43 — Michael Kaminsky ↩

  16. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 40:08 — Michael Kaminsky ↩

  17. S07, E: Demystifying open source MMM (with Michael Kaminsky) @ 41:48 — Michael Kaminsky ↩