Trader Joe's Gives a Lesson in Measuring A/B Test Success

I was in Trader Joe’s the other day. My kid was at a little drawing table they have setup with markers and blank paper. My wife was in the cashier’s line. I was watching my kid, but other than that pretty useless. So I had time to really take in all the beautiful signage in the store.

“Are all these signs handmade?” I asked the three friendly-looking employees behind the Customer Service desk.

“Yes, our employees hand make these signs. Sign Artists in each store create these beautiful handmade signs every day alongside their regular duties like stocking shelves and working the cash register.”

The Trader Joe's Magic Touch

Trader Joe’s is such a phenomenal success. It has sales per square foot double the industry average. And it’s one of the few supermarket chains to go from coast-to-coast. Parking lots are always full, and still customers can’t help themselves but to gather at parties and talk up how great TJs is.

How did this company do it? Everyone knows supermarkets operate on razor thin margins. It’s a volume business dominated by huge corporations squeezing out efficiencies everywhere. Safeways, Stop & Shop and their ilk turned to the Club Card concept decades back so they could crunch reams of personal shopping behavior. This would enable them to expertly target and anticipate customer wants and needs. Trader Joe's hasn't been swept up in this.

And yet here is Trader Joe’s doing double the per square foot sales, and soaking up all the customer praise.

The mega-chains sell name brand products with combined advertising dollars behind them in the billions. Safeway and friends spends umpteen millions each on TV advertising.

And little Joe is stuck with only private label products with little to nothing in the advertising coffers.

TJ's Secret Sauce

Now back to these beautiful, colorful handmade signs. With all the apparent disadvantages TJs already suffers vs the Big Boys, how in the name of all that is labeled Organic can Trader Joe’s afford to waste its money on sign makers? Sign makers for each store? Each day? With products that change all the time?

(True story, my paternal grandfather was a sign-maker and I just figured his were about the last signs to be pained by hand.)

What Trader Joe knows is something you should know too. Measuring only revenue, or measuring only cost savings is doing yourself a disservice.

While other companies are focused entirely on volume or efficient operations, Trader Joe’s has their eye on a third set of metrics – Loyalty and Word-of-Mouth. And the handmade signage, as well as the weekly folksy newsletter, as well as the kids drawing table, and helium balloon tank – these are all investments towards Loyalty and Word-of-Mouth.

These fluffy sounding words are typically very problematic to measure accurately. And so they are often given a bit of lip-service but little else. But get these things right and you have positively impacted your companies ability to make money more than just about any other thing going.

I say this because the contributors to high Loyalty and Word-of-Mouth are typically outrageously economical vs the cost-savings they create.

For most companies acquiring new customers, and re-acquiring customers that have done business with them previously are two of the most expensive items on the books. Tens of millions to hundreds of millions of dollars expended. This included brand advertising (TV, Radio, Print), unrelenting direct mailings, and the like.

Compare this to costs of some helium tanks, a kids table from Ikea and some markers, and yes – staff time devoted to making artful signs. Granted, there’s much more to Trader Joe’s success than these things I mentioned. TJs succeeds in large part to its unique and ever-changing variety of pre-packaged or semi-prepared foods from around the world at reasonable prices. But even here, these differentiators don’t necessarily cost more money, but they do contribute to Loyalty and Word-of-Mouth.

Setting the Right A/B Testing Goals

What does this all have to do with website optimization and a/b testing?

Simple. The typical success metric used for A/B tests is either conversion rate lift or revenue per visitor (RPV) lift. Both of these are some version of measuring the dollars coming in through the door in the here-and-now.

But this single-minded A/B test success metric ignores two other important dimensions that are critical to understanding whether your test recipe is really moving your business forward.

One is Loyalty and Word-of-Mouth. For the Marketing organization, this relates directly to your Cash Flow statement. Growth from Acquisition and Retention burns cash. Get it under control on the cheap, and you've lifted that crucial part of your financials.

The other is Operating Costs. For the Marketing organization, this is headcount to a large degree. Is where you are taking your testing plans going to reduce the ongoing cost of running your business?

Rarely do I see test hypotheses touch on these two metrics.

In many corporate environments, “Website Conversion” and “Website Traffic” are viewed as two independent work flows. The acquisition teams bring ‘em in, and the website marketing team closes ‘em down to become customers. In this structure, the ultimate goal for the “Website Conversion” workflow should not just be revenue or conversion rate.

The Ultimate Testing Success Metric

The goal should be something like “increase incremental revenue while simultaneously increasing loyalty & WOM and reducing operating expense”.

If the website marketing team can do all that, I have no doubt it is making money for the company. If you’re just showing me you’re increasing revenue but I don’t know anything about the other two, I should still have real concerns.

Trackback URL for this post:


The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

Enter the code shown in the image: