Definition
A control group, also called a holdout group, is a randomly selected subset of subscribers who are excluded from receiving a specific email campaign or programme. The behaviour of the control group is compared against the behaviour of the exposed group to measure the incremental lift caused by the email. Lift is calculated as the difference in the target metric (revenue, conversion, engagement) between the two groups. Without a control group, it is impossible to distinguish between behaviour driven by the email and behaviour that would have occurred anyway.
Control groups typically comprise 5 to 20 per cent of the target audience, depending on the statistical power needed to detect the expected effect size. A larger control group increases statistical confidence but reduces the number of subscribers reached by the campaign. For high-traffic programmes, a 5 per cent holdout may be sufficient. For programmes with fewer subscribers or smaller expected effects, 10 to 20 per cent is more appropriate. The control group must be randomly selected and must not differ systematically from the exposed group in any characteristic that affects the outcome metric.
Best Practices
Always run control group tests for campaigns where you need to prove business impact. Without a control group, you can report email performance metrics but cannot isolate the incremental effect of the email from organic behaviour, other marketing channels, or seasonal trends.
Randomise your control group at the subscriber level using a deterministic method such as a hash of the subscriber ID modulo 100. Ensure randomness is verifiable by checking that the control and exposed groups have similar distributions of key characteristics such as recency, frequency, and monetary value.
Determine control group size based on the minimum detectable effect and desired statistical power. Use a sample size calculator before launching the test. For a typical campaign targeting 100,000 subscribers with a minimum detectable effect of 5 per cent and 80 per cent power, a 10 per cent control group of 10,000 is adequate.
Measure lift on a per-subscriber basis rather than aggregate totals. Calculate the average revenue per subscriber in the exposed group and subtract the average revenue per subscriber in the control group. This per-subscriber lift multiplied by the number of exposed subscribers gives the total incremental impact.
Do not include control group members in other campaigns that could confound the measurement. If control group members receive a different email channel or a different campaign, the lift measurement is contaminated. Maintain a centralised suppression list for all active control groups.
Related Glossary Terms
Email Attribution Fraud
The misattribution of email marketing credit for conversions that would have occurred without the email influence, inflating reported email performance.
Campaign Analysis
Campaign analysis is a structured framework for evaluating email performance after send, comparing results against benchmarks and previous campaigns to identify optimisation opportunities.
Email Campaign Audit
A comprehensive post-campaign review evaluating performance against goals, process efficiency, content quality and opportunities for improvement.
Email Campaign Diminishing Returns
The principle that each additional email send generates less incremental revenue or engagement as frequency increases and inbox competition grows.
Email Campaign Optimization Cycle
The iterative process of sending campaigns, measuring results, identifying improvements, implementing changes, and measuring again to continuously improve performance.
Email Content Audit
A systematic review of all email content across campaigns, automations and lifecycle stages to identify gaps, redundancies and improvement opportunities.
Frequently Asked Questions
An A/B test compares two different versions of an email to determine which performs better. A control group compares sending versus not sending the email at all. A/B tests optimise the creative or targeting of a send. Control groups measure whether the send itself drove incremental value.
Measurement windows depend on the purchase cycle. For low-consideration products with short purchase cycles, measure for seven to fourteen days after the send. For high-consideration products with longer cycles, measure for thirty to sixty days. The window should capture the full response curve.
Reusing the same subscribers as a control group across multiple campaigns introduces bias because those subscribers are consistently not receiving email, which may alter their behaviour over time. Rotate control group assignments or use a fresh random sample for each test or measurement period.
Welcome email series typically show among the highest incremental lifts of any email programme. Lift varies by industry but is commonly 50 to 200 per cent in early-period revenue compared to a control group that receives no welcome emails. This makes welcome series strong candidates for control group validation.
For ongoing automations such as welcome series or browse abandonment, control groups are set up at the point of entry into the automation. New subscribers entering the automation are randomly assigned to receive or not receive the emails. The lift is measured over the full lifecycle of the automation.