How Long Should an A/B Test Run? Duration, Peeking and Stopping Rules

Your A/B test has been running for a few days. The new version is performing better, and the dashboard is already showing a statistically significant result. Then someone asks if you can stop the test and make the change live. You want to say yes, especially when the numbers look good. But a few days of positive results don’t always mean the new version will keep performing better.

So how do you know when it’s actually safe to stop? There’s plenty of advice online about running tests for a certain number of days or weeks. But the right duration depends on your traffic, the size of the improvement you’re trying to measure, and how much data you need to trust the result.

In this guide, we’ll explain how to decide how long an A/B test should run, why checking results too early can be misleading, and how to set clear stopping rules before the test starts. That way, you’re making the decision based on the data rather than pressure to launch the winning version.

Why the advice on A/B test duration keeps contradicting itself

If you have searched for an answer, you have probably noticed that the numbers do not agree. AB Tasty puts the average recommended test time at around two weeks and then tells readers to work out the right length from their own goals. In an expert roundup on Convert’s blog, one practitioner waits for at least 100 conversions per variation and says that usually takes four to six weeks, while another prefers to cap tests at four weeks.

These are all reasonable habits for the traffic and conversion rates those people work with. They disagree because duration is not something you pick. It comes out of your sample size and your traffic, so a number borrowed from another site deserves some suspicion.

How long should an A/B test run? Start with sample size

An A/B test should run until it reaches the sample size you calculated before launch, and for at least one full business cycle. Duration comes from dividing that sample size by the eligible visitors entering the test each day, then rounding up so the test ends on a complete cycle.

Convert says duration depends on traffic volume, minimum detectable effect, significance threshold and power. In practice that gives you four numbers to settle before you build anything.

1. Your baseline conversion rate for the page or step being tested.
2. The minimum detectable effect, which is the smallest lift worth acting on.
3. Your significance threshold, which Convert says most teams set at 95 percent.
4. Your statistical power, which Convert says most teams set at 80 percent.

Choose a minimum detectable effect you can defend

The minimum detectable effect is the setting you control most, and it changes the answer a great deal. Convert points out that a smaller effect needs a larger sample, and gives the example of a test projected at 12 weeks that could drop to four with a larger minimum detectable effect, at the cost of missing subtler changes.

It also warns against picking a number your site has never produced. If a 25 percent lift has never happened on your store, a test built to detect it was never going to tell you anything useful. A better starting point is the size of lift your past tests have actually delivered.

Turn the sample size into days

Once a calculator gives you the total sample size, divide it by the number of eligible visitors who enter the test each day. That gives you the raw collection time. Then check it against how your customers buy.

AB Tasty offers a useful example. If your purchase cycle is three weeks and the test runs for one, the sample will not represent your audience. Its advice is to test across at least one business cycle and ideally two. The same page explains why early numbers mislead, because the first visitors in a test rarely look like your full audience, which is one reason a lead on day four deserves caution.

What peeking does to an A/B test

Checking on a running test is a natural thing to do, and it is hard to resist when a number looks good. The trouble starts when the check turns into a decision.

Evan Miller’s well-known piece, How Not To Run an A/B Test, explains that the significance calculation assumes the sample size was fixed in advance. If you run a test until it looks significant instead, the significance levels your dashboard reports stop meaning what they say. His example uses a 50 percent conversion rate and a test stopped at the first sign of 5 percent significance or after 150 observations. Even when the change does nothing, that approach finds a false winner 26.1 percent of the time. He calls this close to a worst case because it checks after every observation, but he notes it is not unheard of.

In other words, a dashboard that shows 99 percent confidence after ten looks is describing something closer to 95 percent.

Look at the dashboard, but do not decide from it

Miller says peeking is fine as long as you do not stop the experiment before it has run its course. That gives you a workable rule. Check a live test to confirm it is healthy, and leave the winner question for the planned end date.

Convert suggests assigning someone to monitor the test for problems such as sample ratio mismatch, with the condition that this person cannot call a winner. We cover how consent settings and cookies create that kind of mismatch in our post on broken A/B test data.

Bayesian testing does not remove the problem on its own

Miller’s article points to sequential and Bayesian designs as ways to look more often. Sequential designs let you set checkpoints in advance and give you correct significance levels at each one. Bayesian reporting is often described as safe to check at any time, but an engineer at RevenueCat ran simulations and found otherwise. More frequent peeking raised the chance of calling a winner when none existed, and a fixed 95 percent threshold did not control that error. If your team wants to check continuously, the method needs to be chosen for that purpose.

Write your A/B test stopping rules before launch

Stopping early is not always wrong. Convert’s roundup describes three situations where it makes sense, which are clear harm to the business, no realistic chance of reaching significance, and a broken test. The difference between those cases and the day-four celebration is that the rules were agreed before anyone saw a number.

These are the steps we suggest writing down before each test starts.

1. Set the sample size and the minimum run length, with the run length covering at least one full business cycle.
2. Name the person who can call the test, and keep that person separate from whoever watches for problems.
3. Decide what level of harm on which metric would end the test early.
4. Decide how you will judge futility. One practitioner in the Convert roundup treats a conditional power below 20 percent by day 14 as a sign the test cannot produce a conclusive result.
5. List the integrity problems that stop a test, such as a bug in the variant, a sample ratio mismatch or a tracking failure.
6. Write down what you will do if the end date arrives without a clear result.

The fifth step depends on your setup being sound before launch. Our post on building A/B tests that do not break your site covers the QA work that catches most variant bugs before traffic reaches them.

On the last row, Convert’s advice is to let the test run. Reaching significance early can mean you caught a moment of variance, and calling the winner then carries the same risks as any other early stop.

Running a test too long has a cost too

Waiting is not free either. Convert notes that the longer a test runs, the easier it is for the sample to be polluted by seasonal shifts, returning visitors who have already seen a variant and outside events such as a news cycle.

Duration also shapes how much you can learn in a year. Convert gives the example of four concurrent tests that each take six weeks, which yields roughly two test cycles in a quarter. Shorter, properly planned tests could fit in a third. When tests stretch, teams also tend to drift toward safer ideas that show results faster and away from bolder ones.

What to do when the end date arrives and the result is flat

A flat result at the planned end date is still a result. Record it as inconclusive, and note the smallest effect your sample could have detected. Then decide whether the idea deserves a bolder second attempt or whether the traffic is better spent elsewhere.

Extending the test because it is almost significant is the same mistake as stopping early because it looks good. Both let the data decide the rules after the fact. If your analytics setup is part of why results keep coming back unclear, our post on why A/B tests keep failing covers the infrastructure side.

A/B test duration FAQ

How long should an A/B test run?

Long enough to reach the sample size you calculated before launch, and for at least one full business cycle. AB Tasty recommends one cycle at minimum and ideally two, and notes that around two weeks is a common average.

Can I stop an A/B test early if it already looks significant?

Convert advises against it. Early significance can be a swing in the data, and Evan Miller shows that stopping at the first significant reading inflates the false positive rate and makes the reported significance misleading.

Is two weeks enough for an A/B test?

Sometimes. AB Tasty gives two weeks as an average, but also says the right length depends on your own goals and purchase cycle. If your customers take three weeks to buy, two weeks will miss part of your audience.

How many visitors do you need for an A/B test?

It depends on your baseline conversion rate, the minimum detectable effect, your significance threshold and your power. One practitioner in the Convert roundup uses 100 conversions per variation as a practical minimum, and notes the number varies by test.

Is it okay to check results while a test is running?

Yes, as long as you do not stop the test because of what you see. Evan Miller says peeking is fine when you can resist ending the experiment before it has run its course.

Set durations and stopping rules before your next test

If your tests keep getting called on day four, bring them to us. Request a free mini audit and we can help you set durations and stopping rules before the next test goes live.