In any organization that does a/b testing, ideas on what to test are rarely in short supply. Test ideas seem to fly out of every corner of the company. So if there are plenty of ideas floating around, what’s the problem?

Most of them are crap shoots. Blind guesses. Shots in the dark.

Plenty of the ideas in the concept bucket are based on precious little. If I were to ask for $40,000 in cash to go do one of these tests, you probably wouldn't give it to me. But there's a pervasive notion that "testing on the web is free". So when that same idea comes in the form of "Hey, let's just test it because it's free to do so", you might green-light it.

While the "it's free" theory may right for some small nimble companies (that unfortunately may not even have the traffic to do much testing), it's not quite true for large companies. In larger companies, testing involves non-trivial fixed and variables costs.

A Marketing Experiment Is An Investment.

It’s an investment of your employees’ time or your program budget. Before I make an investment into something, I want to see some evidence my investment is going to pay off. Through hard data or pure logic, I want to understand why this test has a solid chance at winning.

How Not To Pitch A Test Idea

“The top goal of this division this year is to grow Product Z by 30% this year. Our test hypothesis is that by up-leveling Product Z on the home page with larger font and images and downplaying product Y we can grow the sales of Product Z.’’

What’s the problem? It’s a pretty typical test hypothesis, but it contains no data or logic.

Wishing Well

First, as stated, this hypothesis is not based on any data or logic. It’s based more on a wish. We wish this would sell, so we’re going to plaster it over the site and hope it does.

To find out if there’s any solid data or logic to support this hypothesis, I’d ask questions like: Is there voice-of-customer research showing that prospects would rather be buying Product Z than Product Y? Was Product Z really hard to find before, but Product Y can be found from multiple places? Are competitive products similar to Z doing better in the marketplace right now than products similar to Y, thereby justifying a bigger push on this product?

One time I found myself involved in a live web test of a new offering. Instead of starting with traditional market research to understand the market need (this kind of research was deemed slow and needlessly expensive), a live test was done of this offering. After months of having it promoted on the site, it had a total of seven customers. Had the notion being the offering been predicated on some true grasp of the market reality, we could have made better use of the opportunity to run a test, by running it against an offering that had a real market behind it.

Misguided Missile

The second - and more egregious error - with the pitch shown earlier is that the goal is wrong. The goal of your company is not to sell more Product Z. It is to make money. By hiding one product and promoting another one, you may be more likely to sell more of that new product. But can you NET more money for the company overall? That’s always the sticky wicket.

Sometimes it’s hard to measure “total impact”. In fact it’s nearly impossible to measure the full impact to something as far-reaching as Life Time Value. However, if there’s any reason to suspect your test will cause SHIFT, you must measure it. Mix shift or channel shift can get you closer to the impact that is incremental gain, and must be factored into whatever analysis you’re doing when a shift can be reasonable to assume.

Elements Necessary for a Good Test Concept

1. Based on data (past tests, customer research, market trends, proven principals of persuasion, etc.) or,
2. Based on logic (e.g., an obvious win-win for customer and company)
3. Moves the whole organization forward, not just one part

These elements are necessary, but not sufficient.

Rounding on the list, I’d like to see that my investment is going to be managed well and highly leveragable. In other words, will it cost me a lot to learn this thing and then I can only apply it once? Or can I learn it on the cheap and keep reusing the insight year after year in all my business units.

Testing ideas are a dime-a-dozen, but even taking these few criteria into consideration will drastically cut down the number of ideas worthy of the serious investor’s time and money.

--

Subscribe for free via RSS to get the next post entitled:
"Reverse Engineering A/B Tests" (Oct 12, 2009)