Definition
A control group, also called a holdout group, is a randomly selected subset of subscribers who are excluded from receiving a specific email campaign or programme. The behaviour of the control group is compared against the behaviour of the exposed group to measure the incremental lift caused by the email. Lift is calculated as the difference in the target metric (revenue, conversion, engagement) between the two groups. Without a control group, it is impossible to distinguish between behaviour driven by the email and behaviour that would have occurred anyway.
Control groups typically comprise 5 to 20 per cent of the target audience, depending on the statistical power needed to detect the expected effect size. A larger control group increases statistical confidence but reduces the number of subscribers reached by the campaign. For high-traffic programmes, a 5 per cent holdout may be sufficient. For programmes with fewer subscribers or smaller expected effects, 10 to 20 per cent is more appropriate. The control group must be randomly selected and must not differ systematically from the exposed group in any characteristic that affects the outcome metric.
Best Practices
Always run control group tests for campaigns where you need to prove business impact. Without a control group, you can report email performance metrics but cannot isolate the incremental effect of the email from organic behaviour, other marketing channels, or seasonal trends.
Randomise your control group at the subscriber level using a deterministic method such as a hash of the subscriber ID modulo 100. Ensure randomness is verifiable by checking that the control and exposed groups have similar distributions of key characteristics such as recency, frequency, and monetary value.
Determine control group size based on the minimum detectable effect and desired statistical power. Use a sample size calculator before launching the test. For a typical campaign targeting 100,000 subscribers with a minimum detectable effect of 5 per cent and 80 per cent power, a 10 per cent control group of 10,000 is adequate.
Measure lift on a per-subscriber basis rather than aggregate totals. Calculate the average revenue per subscriber in the exposed group and subtract the average revenue per subscriber in the control group. This per-subscriber lift multiplied by the number of exposed subscribers gives the total incremental impact.
Do not include control group members in other campaigns that could confound the measurement. If control group members receive a different email channel or a different campaign, the lift measurement is contaminated. Maintain a centralised suppression list for all active control groups.
Related Glossary Terms
Email Experimentation
Email experimentation applies the scientific method to email marketing with hypothesis-driven A/B and multivariate testing. Statistical significance at 95% confidence is standard.
Email Frequency Engagement
Email frequency vs engagement analysis finds the optimal send frequency per subscriber. Over-sending increases unsubscribes and dormancy; under-sending diminishes brand recall. Testing methodology identifies the frequency sweet spot.
Email KPI Tree
Email KPI hierarchy organises leading and lagging indicators into a metric tree. Primary metrics drive reporting while secondary metrics diagnose performance.
Email Landing Page
Email landing pages align post-click experience with email messaging using dedicated pages per campaign to optimise conversion rates typically between 2 and 5 per cent.
Email OKR
Objectives and Key Results for email marketing align team efforts with business goals. Leading and lagging indicators track subscriber engagement and revenue impact.
Email Preview
Email preview tools test rendering across 100+ email clients, providing spam scoring, code analysis, accessibility checks and collaboration features for quality assurance.
Frequently Asked Questions
An A/B test compares two different versions of an email to determine which performs better. A control group compares sending versus not sending the email at all. A/B tests optimise the creative or targeting of a send. Control groups measure whether the send itself drove incremental value.
Measurement windows depend on the purchase cycle. For low-consideration products with short purchase cycles, measure for seven to fourteen days after the send. For high-consideration products with longer cycles, measure for thirty to sixty days. The window should capture the full response curve.
Reusing the same subscribers as a control group across multiple campaigns introduces bias because those subscribers are consistently not receiving email, which may alter their behaviour over time. Rotate control group assignments or use a fresh random sample for each test or measurement period.
Welcome email series typically show among the highest incremental lifts of any email programme. Lift varies by industry but is commonly 50 to 200 per cent in early-period revenue compared to a control group that receives no welcome emails. This makes welcome series strong candidates for control group validation.
For ongoing automations such as welcome series or browse abandonment, control groups are set up at the point of entry into the automation. New subscribers entering the automation are randomly assigned to receive or not receive the emails. The lift is measured over the full lifecycle of the automation.