What a Holdout Group Is
A holdout group is the control group in a marketing experiment. You choose a set of users, devices or regions at random and keep them from seeing a campaign. Everyone else is exposed as normal. After the campaign, you compare the two groups. The difference in installs, purchases or revenue is the campaign's incremental effect: what it added beyond what would have happened anyway.
Holdout groups are the core of incrementality testing. They answer the question attribution cannot: did this campaign cause these results?
Why Attribution Is Not Enough
Attribution credits conversions to campaigns that touched the user before converting. That is useful for operations, but it says nothing about cause. Retargeting ads shown to loyal users, brand search ads shown to people already searching for you, and broad campaigns that happen to reach people about to install all collect attributed conversions they did not create.
A holdout separates the two. If the exposed group converts at 5% and the holdout at 4%, the campaign added one percentage point, not five.
Types of Holdouts
User-level holdouts. Individual users or devices are randomly assigned to exposed or held-out groups. Common for retargeting, push, email and CRM campaigns, where you control the audience. Ad platforms such as Meta and Google also offer conversion lift studies that run user-level holdouts inside their systems.
Geo holdouts. Whole regions are held out. Used for TV, outdoor, and channels without user targeting, or when platform lift studies are not available. Regions need to be matched carefully on size, seasonality and past performance.
Time-based holdouts. A campaign is paused for a period and results are compared with periods when it ran. Easier to run, but weaker, because many other things change over time.
Running a Good Holdout Test
- Define one metric. Installs, first purchases or D30 revenue. Pick before the test.
- Randomize properly. Assignment must be random, or the groups will differ for reasons unrelated to the campaign.
- Size it for the lift you expect. Small expected lifts need larger holdouts or longer tests. Run a power calculation before starting.
- Keep the holdout clean. Make sure held-out users or regions are not reached through other campaigns with the same message.
- Measure everything, not just attributed conversions. Compare total conversions in each group, including organic, because the point is to see the full effect.
- Run long enough. Cover at least one full purchase or usage cycle.
Common Pitfalls
- Too small to read. A holdout that cannot detect the expected lift wastes the test.
- Contamination. Held-out users see the campaign through another channel or device.
- Stopping early. Calling a result as soon as it looks significant inflates false positives.
- Ignoring cost. A positive lift is only good news if the incremental revenue beats the incremental spend.
Holdouts and Attribution Together
Holdouts and attribution answer different questions, and growth teams need both. Attribution, including multi-touch and data-driven views, tells you which campaigns touch converting users every day. Holdout tests, run a few times a year on the biggest budgets, tell you how much of that attributed volume is real lift. The ratio between the two, sometimes called an incrementality factor, can then adjust attributed ROAS for planning.
Linkrunner's analytics and cohort reporting give you the attributed side and the post-install metrics, such as retention and revenue, that holdout tests usually compare. Segments built from cohorts can be exported as CSV to define audiences on ad platforms.