CPP A/B Testing estimates test duration by calculating how many users (taps) each variant needs to reach your desired precision, then converting that requirement into days based on traffic.
This article explains which inputs are used and how the duration estimate is calculated. To learn about how the CPP A/B Testing algorithm works, go to Statistical background of CPP A/B Testing.
What the system uses to estimate duration
CPP A/B Testing calculates the required sample size per variant and then converts it into a duration estimate using:
- The original ad group’s conversion rate for the last 28 days (p)
- The confidence level (z) used for duration planning
Example: for 90% confidence, z = 1.65 - The desired precision you choose, which is treated as the margin of error (ε)
Example: ±1 percentage point (0.01) - Your daily average taps
Step 1: Calculate required sample size (n)
The system uses the sample size formula:
n = z² × p(1−p) / ε²
Where:
- n is the required number of users per variant
- z is the confidence value used in duration planning
- p is the benchmark conversion rate from the last 28 days
- ε is the margin of error (desired precision)
Step 2: Convert sample size into test duration (days)
After the system calculates the required sample size (n), it estimates duration using:
Test Duration (days) = n / Daily average taps
Example calculation
Assume the following inputs:
- p = 0.05 (5% conversion rate)
- ε = 0.01 (±1 percentage point margin of error)
- z = 1.65 (90% confidence level)
Sample size:
n = (1.65² × (0.05 × 0.95)) / 0.01² = 1293
This means approximately 1,293 users are required per variant.
If you are getting 500 taps per day total, this results in 250 taps per variant per day. The duration is then calculated as:
1293 / 250
What this means in practice
Test duration is a balance between:
- The benchmark conversion rate (p)
- The confidence level value used in planning (z)
- The margin of error you accept (ε, your desired precision)
- How many taps you receive each day
In general:
- Higher precision (smaller ε) increases required sample size and duration.
- More daily taps reduce the estimated duration.
Related links
- Statistical background of CPP A/B Testing
- How are switch periods chosen in CPP A/B Testing
- Test health and issue management in CPP A/B Testing
- What are the requirements and limits for CPP A/B Testing
Need more help?
If you have further questions on the process, contact your dedicated Customer Success Manager or contact the support team via live chat!