PlanMyTest.com

Display Options

Show explanatory text

Users

Users per Day

Users per Week

Users per Month

Users per 2 Months

Provide the number of cumulative unique users over each of the following periods. Linear interpolation is used between the provided values.

Experiment Design

Business Cycle

days

Number of Variants

Baseline Conversion Rate

MDE (Relative)

Business Cycle: If you have different types of users, or they behaive differently over time, you should avoid making experiment decisions on a fraction of a cycle. Instead, plan you test for a whole number of cycles.

Number of Variants: All else being equal, more variants will take more time to reach statistical significance. However, if additional variants mean one of them has a larger effect, this could let you use a larger value for MDE, thus shortening the experiment overall. This number counts the control (so an A/B test is 2 variants), and assumes equal traffic distribution between all variants.

Baseline Conversion Rate: This is the conversion rate of the experiment's primary metric, expressed as conversions / impressions * 100. A lower baseline conversion rate will be harder to detect changes against, so you can get results sooner with a higher baseline conversion rate.

MDE: This is the minimum size of effect (caused by the treatment variant) that you want to be able to detect, relative to the conversion rate of the baseline. In other words, what is the smallest impact that is meaningful and worht running the test for. A smaller MDE will mean that the experiment takes longer to run.

Statistical settings

Power

Confidence

Apply Bonferroni correction

Bonferroni Correction: Experiments with more than 2 variants have a false positive rate (each comparison between treatment and control is a chance to detect an uplift that doesn't actually exist). A Bonferroni Correction automatically adjusts the number of users required to keep error rate constant when comparing more than 2 variants (and there is no impact to tests with only 2 variants).

Experiment Plan

Users Required

After users have been exposed to your experiment, you'll be able to detect a change as large as (the MDE). This means that if your actual observed effect is or larger, you'll reach stat sig at users.

—
Experiment Duration (Days)

You should expect to achieve statistical significance on day , if your observed effect is or bigger.

—
Stat Sig in 1 Cycle?

Whether significance is likely to be achieved within the first business cycle.

—
Cycles to Reach Stat Sig

Number of business cycles required to detect a change of . If you are using a T-Test, you should only consider results valid at the end of cycle . If you are using a sequential testing calculator, you can check results after each cycle and if there is a larger-than-expected effect, you can stop the test early.

—
Users to Target

If your observed uplift is or larger, you can set your experiment targeting to of your total users and still achieve stat sig in cycles.

—
Or Lower MDE to

If you run for the full cycles and don't reduce the targeting percent, you can expect to detect a change as small as .

—

Scenarios

Desired Experiment Duration (cycles)

Drag the slider to see the impact of a shorter or longer test on MDE, Baseline Conversion Rate or User Targeting

Your MDE would be

You would be able to detect a change of or larger.

—
OR your baseline CR needs to be

If you pick an experiment metric a conversion rate of or higher, you can reach stat sig in cycles.

—
OR you would need number of users exposed to be

You would need to change experiment targeting criteria, or change the page/screen where the experiment runs, to expose users in cycles.

—