That's totally random!

Actually, no it's not.

When you run a test and split the traffic "evenly" it's unlikely you will get exactly the same number of visitors in each recipe. The most common method of splitting traffic is via a random number stored in a cookie. Some other methods are used as well: "round robin" and "time splitting". Assuming you are using a random number generator there are two main sources of variance off a completely even split. First is that the random number generator isn't completely random. In this illustrator we stick to a common method of generating a fairly random number which does a pretty good job, as you can see when you increase the sample size per round. We must admit however that computers can't generate something purely random, but they can get very close. In Javascript we flip a coin like:

randomnumber = Math.floor(Math.random() * 2) + 1;

The remaining variance is due to "sampling error". The sampling error is the variation that occurs due to the fact that you are only sampling some members of a larger population, and those samples may not be perfect representations of the larger population. This is also known as statistical noise. In this simulator you can adjust the sample size per round and visually see the kinds of variations you can expect to get when using samples of this size.

You may try my random number generator simulation which will open in a new window. This will allow you to play with the sample size and see the impact on the variance that is due mainly to sampling error.


Trackback URL for this post: