Definition
Email omission testing — also called holdout testing — measures what happens when a randomly selected group of subscribers does not receive a specific email or set of emails. By comparing the behaviour of the omitted group against the group that received the email, marketers can calculate the true incremental lift the email generated. Omission testing is the most reliable method for measuring causal impact because it eliminates attribution bias and accounts for conversions that would have occurred without the email.
How It Works
- A randomly selected holdout group (typically 5-10% of the target segment) is excluded from receiving the email
- The test group receives the email as normal
- Conversion rates and revenue are compared between the two groups
- The difference represents the email's true incremental impact
- Holdout group members are often compensated with delayed exposure or a different offer
Best Practices
- Randomise holdout selection to ensure groups are comparable
- Maintain the holdout group consistently to build reliable incremental measurement data
- Use holdout testing for high-value campaigns where accurate attribution matters most
- Combine holdout testing with standard attribution models to calibrate attribution methods
- Inform stakeholders that holdouts are a necessary analytical investment, not wasted opportunities
Related Glossary Terms
Control Group
Control or holdout group testing withholds a random subscriber segment from a campaign to measure incremental lift in engagement, revenue, and conversion.
Email Attribution Fraud
The misattribution of email marketing credit for conversions that would have occurred without the email influence, inflating reported email performance.
Email Breakeven
Breakeven analysis for email campaigns identifies the minimum conversions or revenue needed to cover total campaign costs. It enables data-driven budget allocation and campaign go/no-go decisions.
Campaign Analysis
Campaign analysis is a structured framework for evaluating email performance after send, comparing results against benchmarks and previous campaigns to identify optimisation opportunities.
Email Campaign Breakeven Analysis
Calculating the minimum conversions or revenue a campaign must generate to cover all costs including creative production, ESP fees, and team time.
Email Campaign Optimization Cycle
The iterative process of sending campaigns, measuring results, identifying improvements, implementing changes, and measuring again to continuously improve performance.