What Is a Holdout Group in Marketing?

A holdout group is a randomly selected subset of your audience that does *not* receive a particular campaign or message. Everyone else, the group that does receive the campaign, is called the treatment group. The idea is straightforward: if you randomly split your audience and only send the campaign to one group, the difference in outcomes (revenue, purchases, engagement) between the two groups tells you how much impact the campaign actually had. Think of it like a clinical trial. Doctors do not just give a new drug to patients and see if they get better. They compare patients who received the drug to patients who received a placebo. The holdout group is your marketing placebo. For example, imagine you have a winback campaign targeting 10,000 lapsed customers. You randomly hold out 1,000 of them (10%) and send the campaign to the remaining 9,000. After two weeks, you compare revenue per customer in both groups. If the treatment group generated $4.20 per customer and the holdout group generated $3.10, your campaign drove an incremental $1.10 per customer. That is the real lift.

Without a holdout group, you have no way to separate correlation from causation. Consider this scenario: you send a winback email to customers who have not purchased in 60 days. Some of them buy afterward. Success, right? Not necessarily. Some of those customers were going to buy anyway. They were already browsing your site, had items in their wishlist, or simply happened to return on their own. Without a holdout group, you are counting all of that organic revenue as campaign-driven revenue. This is not a minor problem. Studies have shown that email campaigns routinely overcount their impact by 20% to 50% because of this attribution inflation. That means your "5x ROI" campaign might really be a 2.5x ROI campaign. Still good, but the inflated number leads to bad decisions about where to invest your marketing budget. Holdout groups give you the honest answer. They tell you exactly how much revenue happened *because* of your campaign versus how much would have happened regardless.

Setting up a holdout group involves a few key decisions: **1. Choose your holdout percentage.** A common range is 5% to 15% of your target audience. Smaller holdouts preserve more revenue (since more people receive the campaign), but they need more time to reach statistical significance. For most lifecycle campaigns, 10% is a good starting point. **2. Randomize properly.** The holdout must be truly random. Do not hold out your least engaged customers or your smallest spenders. The whole point is that the two groups are statistically identical before the campaign runs. Many ESPs let you create random segments, or you can use a hash-based approach that assigns users deterministically based on a hash of their user ID and campaign ID. **3. Keep the holdout clean.** The holdout group should not receive the specific campaign you are testing. They can still receive other emails (transactional messages, other campaigns), but they should not get the one you are measuring. **4. Measure the right outcomes.** Compare revenue per customer, conversion rate, or whatever KPI matters most for the campaign. Make sure you measure over a reasonable window, typically 7 to 30 days after the campaign sends. **5. Check for statistical significance.** A raw difference between groups is not enough. You need to verify that the difference is unlikely to be due to random chance. A standard two-proportion z-test or chi-squared test works well here. Look for a p-value below 0.05 or, equivalently, a 95% confidence interval that does not include zero. Scalversion automates all of this. Every campaign runs with a deterministic holdout group, and the platform computes incremental lift with confidence intervals automatically. No spreadsheets, no manual segment management.

**"We cannot afford to hold out revenue."** This is the most common objection, and it misunderstands the purpose. The holdout is not permanently withholding email from those customers. It is a temporary, small-percentage test that gives you the data to make much better decisions about where to invest your budget. The insight you gain far outweighs the marginal revenue from the holdout group. **"Our campaigns obviously work, so why test?"** If you are confident, the holdout will confirm it. If you are wrong, you will find out before wasting more budget. Either way, you win. Many teams that "know" their campaigns work discover that some campaigns have zero incremental lift when they finally measure properly. **"It is too complicated."** With the right tooling, holdout testing is no harder than setting up a normal campaign. The analysis is automated, and the results are clear. The hard part is the cultural shift to demanding proof rather than accepting vanity metrics.

Holdout groups are the gold standard for measuring whether your marketing campaigns actually work. They are simple in concept, powerful in practice, and essential for any team that wants to move beyond vanity metrics to real, causal measurement. If you are not running holdout tests today, you are making budget decisions based on inflated numbers. The good news is that getting started is straightforward, and the payoff in better decision-making is immediate.