A practical guide to incrementality testing

Attribution tells you which ads were present before a sale. Incrementality tells you which ads caused it. The difference matters most when budgets are tight: a channel can look efficient in attribution reports while mostly capturing customers who would have bought anyway.

What an incrementality test measures

The idea is simple. Split your audience into two comparable groups, show ads to one and hold them back from the other, then compare conversion rates. The gap between the groups is the lift your ads actually created. Everything else is demand that already existed.

Choosing how to split

For most advertisers, geography is the most practical split. Pick matched sets of regions with similar historical sales, keep ads running in one set and pause them in the other. User-level holdouts are more precise but need platform support and enough volume to reach significance.

Running a clean test

Keep the test long enough to cover a full purchase cycle, usually three to six weeks. Change nothing else in the test regions during that time: no new creative, no price changes and no promotions that only run in one group. Decide your success metric before you start, and write it down.

Size the holdout so the result is readable. As a rule of thumb, holding back 10 to 20% of a channel’s spend is enough to detect a lift of 10% or more. In Reticle, you can set this up as a geo_holdout experiment and the sample size is calculated for you.

Reading the result

Divide the extra conversions in the exposed group by the spend that produced them to get your incremental cost per acquisition. Compare it with the platform-reported CPA. If the incremental figure is far higher, the channel is claiming credit for sales it did not drive.

Making it a habit

A single test is a snapshot. The teams that get the most from incrementality run a small rolling programme: one channel at a time, every quarter, feeding the results back into how they weight attribution. Reticle uses these results to calibrate its attribution model, so forecasts reflect lift rather than credit. If you want help designing your first test, talk to our team.