Definition
Statistical significance is a mathematical measure of confidence that an observed difference in email test results — such as variant A having a higher open rate than variant B — reflects a genuine effect rather than random variation. It is expressed as a confidence level, typically 95%, meaning there is only a 5% probability that the observed difference occurred by chance.
Without statistical significance testing, email marketers risk optimising on noise — chasing random fluctuations as if they were meaningful patterns. A variant that appears better after 100 sends may perform identically to the control after 10,000 sends if the initial difference lacked statistical weight.
Key Concepts
- P-value: The probability that the observed result occurred by chance. A p-value below 0.05 is commonly the threshold for significance
- Sample size: Larger samples produce more reliable results. Small lists rarely reach significance
- Minimum detectable effect: The smallest meaningful difference the test can reliably detect
- Confidence interval: The range within which the true value is likely to fall
Best Practices
- Do not end a test early — early results are often misleading
- Calculate required sample size before launching a test
- Use a single metric as the primary success measure rather than evaluating multiple metrics simultaneously
- If your list is too small to reach significance, focus on qualitative feedback instead of chasing statistical certainty
Related Glossary Terms
A/B Testing
A/B testing in email marketing is the practice of sending two variations of an email to a small sample of your list to determine which version performs better before sending the winner to the remaining subscribers.
ARPU (Average Revenue Per User)
ARPU (Average Revenue Per User) is a metric that measures the average revenue generated per email subscriber over a specific period, used to evaluate list value and campaign effectiveness.
Attention Rate
Attention rate is the percentage of email opens that last longer than 5 seconds, distinguishing genuine reads from passive opens, preview-pane views, or Apple MPP auto-loads.
Bounce Rate
Email bounce rate is the percentage of emails that were rejected by the receiving server before reaching the recipient. It is a key indicator of list health and data quality.
Campaign Internal Rate of Return (IRR)
Campaign IRR is the annualised return rate earned by an email campaign, accounting for the timing and magnitude of costs and revenue across the campaign's full lifecycle.
Campaign Lifted Revenue
Campaign lifted revenue is the additional revenue generated by an email campaign beyond what would have occurred without it, measured via a holdout group or statistical lift analysis.