Definition
Email omission testing — also called holdout testing — measures what happens when a randomly selected group of subscribers does not receive a specific email or set of emails. By comparing the behaviour of the omitted group against the group that received the email, marketers can calculate the true incremental lift the email generated. Omission testing is the most reliable method for measuring causal impact because it eliminates attribution bias and accounts for conversions that would have occurred without the email.
How It Works
- A randomly selected holdout group (typically 5-10% of the target segment) is excluded from receiving the email
- The test group receives the email as normal
- Conversion rates and revenue are compared between the two groups
- The difference represents the email's true incremental impact
- Holdout group members are often compensated with delayed exposure or a different offer
Why It Matters
This matters because the choices you make here show up directly in your results. Omission testing is the most reliable method for measuring causal impact because it eliminates attribution bias and accounts for conversions that would have occurred without the email. When this is handled well it supports engagement, delivery, and the trust subscribers place in your brand; when it is neglected, the effects tend to show up in declining performance and harder-to-fix problems further down the line.
Best Practices
- Randomise holdout selection to ensure groups are comparable
- Maintain the holdout group consistently to build reliable incremental measurement data
- Use holdout testing for high-value campaigns where accurate attribution matters most
- Combine holdout testing with standard attribution models to calibrate attribution methods
- Inform stakeholders that holdouts are a necessary analytical investment, not wasted opportunities
Was this useful?
Related Glossary Terms
Control Group
Control or holdout group testing withholds a random subscriber segment from a campaign to measure incremental lift in engagement, revenue, and conversion.
Email Acquisition Channel Attribution
The practice of assigning credit for new subscriber sign-ups to the specific marketing channel, campaign and touchpoint that generated them.
Email Attribution Fraud
The misattribution of email marketing credit for conversions that would have occurred without the email influence, inflating reported email performance.
Email Attribution Last Touch
Last-touch attribution in email marketing gives full conversion credit to the final email campaign a subscriber engaged with before converting. It is the simplest attribution model but often oversimplifies the customer journey.
Email Attribution Model Comparison
Comparing different attribution models — first-touch, last-touch, multi-touch, linear, time-decay and data-driven — to understand how each affects campaign measurement.
First-Touch Attribution
First-touch attribution credits the entire conversion to the first email that introduced or acquired the subscriber, useful for measuring acquisition.
Frequently Asked Questions
Good practice here means handling Email Omission Testing in a way that is relevant, timely, and honest for your audience. A controlled experiment where a group of subscribers receives no email to measure the true incremental impact of a campaign or programme. Done well, it improves engagement and builds trust; done poorly, it creates friction that costs you results.
Because it touches the parts of email that drive outcomes: relevance, trust, and delivery. Small improvements compound, while repeated mistakes quietly erode the health of your programme.
The most common problems are treating Email Omission Testing as a one-off task, ignoring what the data says, and copying competitors without testing. All three lead to effort that does not translate into better results.
Compare the metrics it should influence — engagement, conversions, and deliverability — before and after you make changes. Trends over time matter far more than any single send.
It supports the same goal as the rest of your email programme: the right message to the right person at the right time. Aligned with segmentation and automation, it reinforces everything else rather than competing with it.