Running Multiple Tests At A Time

Running Multiple Tests At A TimeThe most dogged misperception about split testing is that you should only run one test at a time. For example there are two tests conceived of by two different people in your organization – one on the home page and one in the checkout – and someone says, “We can’t run them at the same time because we won’t get clean results.” Should you believe them? It depends. Most of the time you can run tests simultaneously without worrying. Having the flexibility allows you to learn more quickly.

There are scenarios when you have to be careful however. Let’s setup one of those examples.

Test #113: Home Page
Recipe A (Style Blue) vs. Recipe B (Style Red)
Test #114: Checkout
Recipe A: (Style Red) vs. Recipe B (Style Purple)

Let’s say if tested in isolation Test #113 shows higher conversion rates with Style Blue. And in isolation Test #114 shows higher conversion rate with Style Purple.

But if #113-B (Red) is combined with #114-A (Red) that combo produces a huge lift – bigger than the lift when you select the two winners from the tests in isolation (#113-A Blue + #114-B Purple). Chalk one up for consistency of design!

So in this made-up example the right way to setup the test was as a multivariate test (MVT). You can see all the combinations and pick the best one. You setup this up as a MVT so you got the best answer. Through your skill you avoided to three suboptimal ways to setup this up:

  • Run them at the same time so everyone is in both tests, but not setup as a MVT
  • Run them at the same time but split the traffic so that half the traffic sees only #113 and the other half sees only #114
  • Run them sequentially where all traffic sees #113 and then when that test is over all the traffic sees #114

These were all suboptimal setups because there was no way to read results of the visitors who saw both #113-B and #114-A. So you would not see that this was the peak performing combination.

Wait. Did that example just prove that you should always run tests at the same time, setup as a MVT? Absolutely not! How practical would that be?

You would have to have all the test ideas you were ever to have and test them all at once. You’d run 10,000 tests in the world’s most gigantic MVT all in a 2-week period and then you’d be done. If you could do that you’d have the most optimal version of your website and you could retire because there’s no more work to do.

Many people I’ve met have stored in their head “only test one thing at a time” and have translated that (erroneously) to “only put one person in one test at a time”. But as you can see from the example above even if you tested #113 in June and #114 in July you STILL would not have found the optimal mix. At the end of July you’d have Blue on the home page and Purple on the Checkout. So separating them out didn’t help. But you lost a month of time.

No matter what you have on the site today, you’re going to change it tomorrow. And run more tests. And it is in the realm of possibility that some test you do in the future would have influenced the results of some test you did in the past IF you ran them both at the exact same time with the MVT setup and analysis. Deal with it. Unless you’re planning on running all your best 10,000 ideas in a single month and then stopping, you’re going to have to live with the fact that your site is never going to be at it’s theoretical peak of performance.

What you can and should do however is this: if you have two tests on the docket that have a reasonable chance of interacting, combine them into a single MVT. And look for interaction effects. But you should be able to articulate the theory as to why they might interact. If what you’re looking to do is optimize a particular page and you’ve got a bunch of different elements you are thinking of changing, then certainly you should use MVT for that test design as well.

Be careful at falling into the trap of “never put someone in more than one test a time” because it slows you down and doesn’t teach you anything new.

The experimental approach has us controlling for variables, not eliminating variation. So it’s ok that not all your site visitors have the same demographics, propensities or behaviors. It’s ok because you’ve randomized the visitors into the test recipes you don’t get lumpiness where all 65+ year old visitors fall into Recipe B. It’s the same with running multiple tests at a time. It’s ok that there is variation in what people see, so long as that variation in evenly distributed across the recipes. If you are putting all your visitors into 5 totally separate A/B tests, then there ought to be 5 random numbers that distribute the visitor into each one of those tests. That will ensure the variables from the other tests are evenly distributed across the recipes of the one you are analyzing.

Hopefully this explanation will work for you and will free you up to do lots more testing. Use MVT when appropriate and simultaneous testing the rest of the time.

Trackback URL for this post:


The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

Enter the code shown in the image: