Definition
Confidence level answers: how sure can you be that variant B genuinely outperformed variant A? A 95% confidence level means there is a 95% probability the result is real and only a 5% chance it is due to random variation. Most email A/B tests use 95% as the threshold for declaring a winner. Lower confidence (80-90%) is acceptable for low-risk tests. Higher confidence (99%) is used for high-stakes decisions. Confidence is calculated using the sample size, the conversion rates of each variant, and the variability of the data. Testing with insufficient sample size reduces confidence regardless of the apparent difference.
Related Glossary Terms
A/B Testing
A/B testing in email marketing is the practice of sending two variations of an email to a small sample of your list to determine which version performs better before sending the winner to the remaining subscribers.
Abandoned Cart Email
An abandoned cart email is an automated message sent to customers who added items to their online shopping cart but left without completing the purchase. It is one of the highest-converting email types in ecommerce.
ARPU (Average Revenue Per User)
ARPU (Average Revenue Per User) is a metric that measures the average revenue generated per email subscriber over a specific period, used to evaluate list value and campaign effectiveness.
Attention Rate
Attention rate is the percentage of email opens that last longer than 5 seconds, distinguishing genuine reads from passive opens, preview-pane views, or Apple MPP auto-loads.
Bounce Rate Threshold
A bounce rate threshold is the percentage of hard or soft bounces at which email platforms and ISPs begin to penalise sender reputation, typically around 2-5%.
Bounce Rate
Email bounce rate is the percentage of emails that were rejected by the receiving server before reaching the recipient. It is a key indicator of list health and data quality.