If you were a fly on my wall, you would hear this:

Marketer: “Jared, could you get tell me how long I will need to run this test for?”
Jared: “Sure. May I inquire why you're asking?"
Marketer: “Because there are more tests I've put on the testing calendar right behind this one. I need to make sure this can finish up quickly.”

Estimating test length is fairly easy with one big catch:

How much is your test recipe going to impact your success metric?

(Yes, you also need to know how many visitors you can put through the test and what the baseline rate is on your success metric. But estimating those are often straightforward.)

The problem is that the marketer generally has a tough time estimating how much of a lift they need to detect. And so their first estimate (say +5%) rapidly migrates to a much larger lift estimate (say +25%) once they hear how long a 5% lift will take to detect. Often they're really early in any kind of learning plan, so no strong data set exists for them off which to base an estimate. Since it's all guesswork, it must seem harmless to adjust it upwards so that the test fits within the available window. Or maybe it's just optimism.

The optimism shows up in another place too. The test driver wants to know how long they need to give this test so that they can schedule another test right behind it. This implies one of two things about the thought process:

Assumption #1: “The test is going to produce a winner.”

If you assume this particular test will yield a clear winner, then sure – figure out how long it will take and schedule a different test right behind it.

But for some of the best testing organizations only 1 in 10 tests produces a clear winner. Is it just optimism that this test will be that one?

Assumption #2: “If we don’t get a clear winner in this time frame, we’re okay to abandon this theme altogether.”

If you schedule an unrelated test right behind it, have you just "abandoned ship"? If the organization is so willing to just move on after trying only one angle, why did we test that in the first place? If it were based on solid research or logic, wouldn't you want to try again from a different angle and see how you can make it work?

So I'll present another way to go about planning your marketing experiments:

1. List out the themes for A/B testing that you believe in.

Ideally, you believe in these themes because there’s already solid data from other companies or from customer research or market research that pointed you in this direction in the first place. Or maybe you believe in these themes because your CEO is a visionary and rarely wrong (i.e., Is your CEO named Steve Jobs?)

2. Pick your strongest testing themes.

Once you realize that testing your way into growth will take time, you’ll find you have to focus your resources. What are the themes that are the most extensible across your business lines? What are the themes that will stand the test of time?

3. Give them enough time to play out.

For a given theme, you may need 4 or 5 rounds of testing to get it right. Amazon.com – a giant in the area of A/B testing - will pick a strategic direction in terms of the product or business areas they believe represent future growth. And then they test their way into success in in that area. Their track record has shown them it takes 2-3 years to develop the right formula for that new business area.

Wait a second. Am I suggesting that your business act strategically? Have a long-term vision? Look beyond this quarter’s earnings? And use A/B testing to win at that strategy? Um, yeah.

If that sounds too outlandish, don’t fret. Website optimization can and does yield quick wins too. And enough small and quick wins can add up to serious money. If that’s what you’re using it for, that’s fine. But beware: if your competition is using A/B testing and multivariate testing to optimize all the parts of their business – the 5 Ps or Marketing (product, price, promotions, place, and people) – your small and quick wins may not be enough.

“How long do I need to run this test?” Instead, ask yourself “what kinds of tests would I be willing to devote 2-3 years to?” Paradoxically, those are tests that will keep you on the leading edge.

This is a rich topic, and I'll have plenty more to write on the subject. I'd like to hear your thoughts, so I encourage your comments below.