Definition
Email conversion lift measures the incremental increase in conversion rate that is directly attributable to email marketing, as distinct from conversions that would have occurred regardless. Lift measurement answers one of the most important questions in email marketing: how many additional conversions does email actually drive beyond what would have happened organically?
A/B holdout testing is the standard methodology for measuring conversion lift. In a holdout test, subscribers are randomly divided into two groups. The treatment group receives the email campaign as normal. The holdout group, typically 5-10% of the audience, is excluded from the campaign. Both groups' conversion behaviour is tracked over a defined measurement period. The conversion lift is calculated as the treatment group conversion rate minus the holdout group conversion rate. If the treatment group converts at 6% and the holdout group at 4%, the email conversion lift is 2 percentage points.
Lift calculation must account for the counterfactual: what the treatment group would have done without email. The holdout group provides this counterfactual estimate. The incremental conversion rate is the absolute difference between the two groups' rates. The percentage lift is this difference divided by the holdout rate. In the example above, the percentage lift is 50%, meaning email increased conversions by 50% over the baseline. Both absolute and percentage lift should be reported because they communicate different aspects of performance.
Best Practices
- Use holdout groups of at least 5% of your audience for statistically reliable results: The minimum holdout size depends on your expected lift and baseline conversion rate. Use a statistical power calculator to determine the required sample size. For typical email lift tests, a 5-10% holdout group provides sufficient statistical power to detect a 10-20% relative lift.
- Randomise holdout group assignment to avoid selection bias: Use a true random assignment method rather than convenience-based selection such as every Nth subscriber or last name alphabetical. True randomisation ensures the holdout group is representative of the full population and eliminates systematic differences between the groups.
- Measure lift over a defined time window that matches your conversion cycle: Set the measurement window based on your typical conversion lag time. For e-commerce, 7-14 days after send is common. For B2B with longer sales cycles, 30-60 days may be required. The measurement window should capture the full conversion effect without including too much unrelated activity.
- Calculate statistical significance before declaring a lift result: Use a two-proportion z-test or chi-squared test to determine whether the observed conversion difference is statistically significant. Report the confidence level alongside your lift number. Results that are not statistically significant at the 95% confidence level should be treated as inconclusive rather than as zero lift.
- Run lift tests across different campaign types and subscriber segments: Conversion lift varies significantly by campaign type and subscriber segment. Promotional emails typically show higher lift than newsletters. New subscribers show higher lift than long-term subscribers. Segment-level lift analysis identifies which campaigns and audiences are genuinely influenced by email versus those that would convert without it.
Related Glossary Terms
Email Analytics
The measurement, collection, analysis, and reporting of email performance data to understand subscriber behaviour and optimise campaign results.
Email Cannibalisation
Email cannibalisation occurs when email marketing reduces the performance of other marketing channels, such as organic search or paid social, by capturing traffic or conversions that would have occurred anyway.
Email Deliverability Testing
Email deliverability testing uses seed lists and inbox placement monitoring to measure what percentage of emails reach the inbox versus the spam folder across major ISPs and mailbox providers.
Email Holdout Group
An email holdout group is a randomly selected subset of subscribers excluded from email campaigns to provide a baseline for measuring incremental lift in opens, clicks, conversions, and revenue.
Email Testing
Systematic verification of email rendering, deliverability, functionality, and content quality before campaign deployment.
Frequently Asked Questions
Conversion rate is the percentage of recipients who converted after receiving an email, which includes both conversions driven by the email and conversions that would have occurred anyway. Conversion lift isolates the incremental conversions caused by the email by subtracting the baseline conversion rate observed in a holdout group that did not receive the email.
The minimum sample size depends on your expected lift, your baseline conversion rate, and your desired statistical power. For typical email programmes, a minimum of 5,000 to 10,000 subscribers per group is needed to detect a 10-20% relative lift at 80% statistical power and 95% confidence. Smaller lists may need to accept lower confidence levels or combine results from multiple campaigns.
A good conversion lift depends on the campaign type and baseline conversion rate. Promotional campaigns to engaged subscribers typically show 20-50% lift. Lifecycle campaigns such as welcome series and abandoned cart shows 30-80% lift. Newsletters to broad lists often show 5-15% lift. Results below 5% lift suggest the email is having minimal incremental impact and the campaign strategy should be reviewed.
The holdout test should run long enough to capture the full conversion cycle. For e-commerce, 7-14 days is sufficient for most products. For high-consideration purchases and B2B, 30-60 days is more appropriate. The measurement period starts when the email is sent, not when it is opened, because subscribers may open and convert at different times.
Without a holdout group, you cannot measure true incremental lift because you lack a counterfactual. Alternative approaches such as before-and-after comparisons and matched-market studies have significant methodological limitations and tend to overstate lift. Holdout testing remains the industry gold standard for reliable conversion lift measurement.