Definition
An email holdout group is a randomly selected segment of subscribers who are excluded from receiving marketing email communications for a defined period. The holdout group serves as a control or baseline against which the behaviour of the treatment group, which receives normal email communications, is compared. The difference in conversion rate between the two groups represents the incremental impact of email marketing.
Holdout group sizing for statistical significance depends on the expected lift, baseline conversion rate, and desired confidence level. A typical holdout group is 5-10% of the total subscriber base. For programmes with 100,000 subscribers, a 5,000-10,000 person holdout group is sufficient to detect a 10-20% relative lift in conversion rate at 95% confidence. The required holdout size increases as the expected lift decreases. Programmes with small lists or low baseline conversion rates may need larger holdout groups, up to 15-20%, to achieve statistical significance.
Holdout duration must be long enough to capture the full conversion cycle but not so long that the holdout group's behaviour becomes unrepresentative. For e-commerce businesses with a 7-14 day conversion cycle, a 30-day holdout period is typically sufficient. For B2B businesses with longer sales cycles, 60-90 days may be required. For ongoing measurement, some organisations maintain a permanent holdout group of 2-5% of subscribers that never receives marketing email, providing a continuous baseline for incrementality measurement.
Best Practices
- Randomise holdout group selection to ensure representative comparison: Use a random sampling method such as a random number generator rather than selection criteria such as engagement level or sign-up date. Randomisation ensures the holdout group mirrors the full subscriber population and eliminates systematic bias that would invalidate the comparison.
- Set holdout group size based on statistical power requirements, not convenience: Calculate the minimum holdout size needed to detect your expected lift with 80% statistical power at 95% confidence. Use a sample size calculator with your baseline conversion rate and expected minimum detectable effect. A 5% holdout is not automatically correct; size it based on your data.
- Document holdout group membership and exclusion periods in your ESP: Tag holdout group subscribers with a specific attribute or segment so they are consistently excluded from all marketing campaigns during the holdout period. Document the exclusion rules, start date, and end date. Ensure your ESP's automation workflows respect the holdout exclusion.
- Rotate holdout group membership periodically if maintaining a permanent holdout: For permanent holdout groups maintained for ongoing incrementality measurement, rotate members every 90-180 days. Permanent exclusion from all marketing for the same subscribers creates ethical concerns and may skew the holdout group's behaviour if they become aware they are not receiving communications.
- Report holdout test results with confidence intervals and statistical significance: When presenting lift results, include the confidence interval, p-value, and sample sizes for both groups. A lift result that appears large but is not statistically significant should be reported as inconclusive rather than as a confirmed lift. Transparent reporting builds trust in the incrementality measurement methodology.
Related Glossary Terms
Email Analytics
The measurement, collection, analysis, and reporting of email performance data to understand subscriber behaviour and optimise campaign results.
Email Cannibalisation
Email cannibalisation occurs when email marketing reduces the performance of other marketing channels, such as organic search or paid social, by capturing traffic or conversions that would have occurred anyway.
Email Conversion Lift
Email conversion lift measures the incremental increase in conversion rate attributable to email campaigns using holdout group testing and statistical significance analysis.
Email Deliverability Testing
Email deliverability testing uses seed lists and inbox placement monitoring to measure what percentage of emails reach the inbox versus the spam folder across major ISPs and mailbox providers.
Email Testing
Systematic verification of email rendering, deliverability, functionality, and content quality before campaign deployment.
Frequently Asked Questions
The holdout group provides a counterfactual baseline that answers the question: what would have happened if these subscribers had not received email? By comparing the holdout group's conversion rate to the treatment group's rate, marketers can isolate email's true incremental impact from conversions that would have occurred through other channels or organic behaviour.
A holdout group should be 5-10% of the total subscriber base for most programmes. The exact size depends on your baseline conversion rate, expected lift, and desired statistical confidence. Use a statistical power calculator to determine the minimum size. Very large programmes may use smaller percentages such as 2-3% while still achieving reliable results due to the absolute sample size.
The holdout period should cover at least one full conversion cycle. For e-commerce, 30 days is typically sufficient. For B2B or high-consideration purchases, 60-90 days may be needed. For ongoing measurement, a permanent holdout of 2-5% provides continuous baseline data. The key is running the test long enough to capture the delayed conversion effects that email may influence.
Measure the same metrics you track for the treatment group: conversion rate, average order value, revenue per subscriber, site visit rate, and any other key performance indicators relevant to your business. The lift is calculated as the treatment group metric minus the holdout group metric, divided by the holdout group metric, expressed as a percentage.
Withholding marketing email from a randomly selected group for a limited period is standard research practice and is generally considered ethical provided the holdout period is not excessive, the holdout group is not deprived of essential communications such as transactional messages, and the results are used to improve the email programme for all subscribers. Most subscribers do not notice a 30-day holdout period. Clearly document your holdout policy for internal governance.