Calculating sample size is an important first step in the process of running an a/b split test. Many people ignore this step, or do not understand the purpose it serves. Let’s explore why to calculate sample size and how to do it.

What is Statistical Power
First let’s explain why to calculate sample size. It’s all about the test’s “Power”. By using a sample size calculator before your marketing experiment you can ensure a certain “power”. What is Statistical Power? It is the chance that your test will detect a difference if that difference exists. A test with a small sample size may have low power. Also a test that is trying to pick up a very weak signal may also have low power.

Sometimes Adobe Test & Target is the right tool for your A/B testing needs. Or maybe Google Website Optimizer fits the need (and budget) better. How about SiteSpect or Optimost? But what if your particular business requires you to build your own A/B testing or MVT framework from scratch? That's the situation that confronted me during the past year. It can feel a little daunting, but what I've learned is that if you stick with it, it could be worth the effort to code your own website optimization system.

Why would you want to build your own website optimization platform?

When you’re optimizing a landing page for leads or sales, and those leads or sales are going to happen right away, an off-the-shelf solution may be just perfect. Adobe Test & Target will track how many people come into each test recipe, and how many people convert. You’ve got a conversion rate on your control and your test treatments. Voila – you’ve got a working A/B testing solution.

For subscription businesses or software-as-a-service (SaaS) businesses, this may not be so simple. A given recipe may outperform control on some

I was in Trader Joe’s the other day. My kid was at a little drawing table they have setup with markers and blank paper. My wife was in the cashier’s line. I was watching my kid, but other than that pretty useless. So I had time to really take in all the beautiful signage in the store.

“Are all these signs handmade?” I asked the three friendly-looking employees behind the Customer Service desk.

If you were a fly on my wall, you would hear this:

Marketer: “Jared, could you get tell me how long I will need to run this test for?”
Jared: “Sure. May I inquire why you're asking?"
Marketer: “Because there are more tests I've put on the testing calendar right behind this one. I need to make sure this can finish up quickly.”

Short answer:
Not unless you have magical oompa-loompas that you don’t need to pay, and these oompa-loompas are really good at persuasion architecture, design, analytics, & web development.

Can you spot the differences?

One of the fun things I like to do when I have some free time is to try to learn from A/B tests other companies are running. My thinking is, "Why run a test when you don't have to?" There might be something you can learn and adapt to your business, or it might spark some new thinking.

Have you ever gulped down an Apple Beer? Me neither. But Matthew Wright has. He also has some great experience in A/B testing at Hewlett-Packard. So pull up a stool and join our conversation. Jared Waxman chats with Matthew Wright, Manager of HP.com Worldwide Analytics, about A/B testing within large organizations.

As part of series on A/B Testing on http://www.benchmark-analytics.com these two practitioners share their thoughts on topics including: What are the benefits? What are the misconceptions? What to look for when hiring analysts?