Definition
An email holdout group is a randomly selected segment of subscribers who are excluded from receiving marketing email communications for a defined period. The holdout group serves as a control or baseline against which the behaviour of the treatment group, which receives normal email communications, is compared. The difference in conversion rate between the two groups represents the incremental impact of email marketing.
Holdout group sizing for statistical significance depends on the expected lift, baseline conversion rate, and desired confidence level. A typical holdout group is 5-10% of the total subscriber base. For programmes with 100,000 subscribers, a 5,000-10,000 person holdout group is sufficient to detect a 10-20% relative lift in conversion rate at 95% confidence. The required holdout size increases as the expected lift decreases. Programmes with small lists or low baseline conversion rates may need larger holdout groups, up to 15-20%, to achieve statistical significance.
Holdout duration must be long enough to capture the full conversion cycle but not so long that the holdout group's behaviour becomes unrepresentative. For e-commerce businesses with a 7-14 day conversion cycle, a 30-day holdout period is typically sufficient. For B2B businesses with longer sales cycles, 60-90 days may be required. For ongoing measurement, some organisations maintain a permanent holdout group of 2-5% of subscribers that never receives marketing email, providing a continuous baseline for incrementality measurement.
Best Practices
- Randomise holdout group selection to ensure representative comparison: Use a random sampling method such as a random number generator rather than selection criteria such as engagement level or sign-up date. Randomisation ensures the holdout group mirrors the full subscriber population and eliminates systematic bias that would invalidate the comparison.
- Set holdout group size based on statistical power requirements, not convenience: Calculate the minimum holdout size needed to detect your expected lift with 80% statistical power at 95% confidence. Use a sample size calculator with your baseline conversion rate and expected minimum detectable effect. A 5% holdout is not automatically correct; size it based on your data.
- Document holdout group membership and exclusion periods in your ESP: Tag holdout group subscribers with a specific attribute or segment so they are consistently excluded from all marketing campaigns during the holdout period. Document the exclusion rules, start date, and end date. Ensure your ESP's automation workflows respect the holdout exclusion.
- Rotate holdout group membership periodically if maintaining a permanent holdout: For permanent holdout groups maintained for ongoing incrementality measurement, rotate members every 90-180 days. Permanent exclusion from all marketing for the same subscribers creates ethical concerns and may skew the holdout group's behaviour if they become aware they are not receiving communications.
- Report holdout test results with confidence intervals and statistical significance: When presenting lift results, include the confidence interval, p-value, and sample sizes for both groups. A lift result that appears large but is not statistically significant should be reported as inconclusive rather than as a confirmed lift. Transparent reporting builds trust in the incrementality measurement methodology.
Related Glossary Terms
Email A/B Test Confidence Level
The confidence level in an email A/B test indicates the probability that the observed result is genuine and not due to random chance, with 95% being the standard threshold.
Email A/B Test Examples
Email A/B test examples show practical testing scenarios that help marketers improve open rates, click-through rates, and conversions through data-driven experimentation.
Email A/B Test Minimum Detectable Effect
The minimum detectable effect (MDE) is the smallest improvement an A/B test can reliably detect given the available sample size, directly affecting how long a test must run.
Email A/B Test Sample Size
A/B test sample size is the minimum number of recipients needed per variant to achieve statistically significant results, determined by baseline conversion rate, minimum detectable effect, and confidence level.
Accessibility Testing
Accessibility testing evaluates how well your emails can be read and understood by people with disabilities, covering screen readers, colour contrast, keyboard navigation, and more.
Email Analytics
The measurement, collection, analysis, and reporting of email performance data to understand subscriber behaviour and optimise campaign results.
Frequently Asked Questions
The holdout group provides a counterfactual baseline that answers the question: what would have happened if these subscribers had not received email? By comparing the holdout group's conversion rate to the treatment group's rate, marketers can isolate email's true incremental impact from conversions that would have occurred through other channels or organic behaviour.
A holdout group should be 5-10% of the total subscriber base for most programmes. The exact size depends on your baseline conversion rate, expected lift, and desired statistical confidence. Use a statistical power calculator to determine the minimum size. Very large programmes may use smaller percentages such as 2-3% while still achieving reliable results due to the absolute sample size.
The holdout period should cover at least one full conversion cycle. For e-commerce, 30 days is typically sufficient. For B2B or high-consideration purchases, 60-90 days may be needed. For ongoing measurement, a permanent holdout of 2-5% provides continuous baseline data. The key is running the test long enough to capture the delayed conversion effects that email may influence.
Measure the same metrics you track for the treatment group: conversion rate, average order value, revenue per subscriber, site visit rate, and any other key performance indicators relevant to your business. The lift is calculated as the treatment group metric minus the holdout group metric, divided by the holdout group metric, expressed as a percentage.
Withholding marketing email from a randomly selected group for a limited period is standard research practice and is generally considered ethical provided the holdout period is not excessive, the holdout group is not deprived of essential communications such as transactional messages, and the results are used to improve the email programme for all subscribers. Most subscribers do not notice a 30-day holdout period. Clearly document your holdout policy for internal governance.