User loginNavigationInsights-to-Income Model
Search |
Is Testing On the Web Really “Free”?
But as your organization gets more deeply into A/B testing, you might soon you have so many test concepts you’ve built a small army of salaried professionals to plan, execute, and analyze them. A/B testing at any decent-sized organization – once you get going – is definitely not “free”. But Is It Close to Free?To me, $50,000 is a lot of money. If I came home and bought a brand-new BMW and told my wife it was “free”, she’d probably want to know more. When I explain that it was really $50,000 which is close enough to “free”, she’d probably think I lost my mind. And yet, a typical A/B test consumes $50,000 of a company’s resources.
This estimate may be off by a factor for two for your particular organization. Still, it's not free, and whatever you spend money on should be understood in terms of the return you get on your investment. So How Do I Estimate Return on Investment?
Operating costs:As shown above, it’s fairly easy to get a rough estimate of how much internal resources are being consumed by your tests. The nice thing here is you will have historical data. Even if people didn’t keep track of every minute of their time, you can fairly easily take a poll of everyone who was involved in the test and ask them for how much time they put in. Do that a handful of tests and you’ll have a pretty good idea which is based on historic data of the operating costs. Instead of simply recording the average, you should compile this data into a histogram. Win Rate:Similarly, you can use historic data to get a win rate for your tests. Count up how many tests you ran, and how many of them yielded a winning recipe. Now let’s talk about what is much more of unknown. Predicted Lift from Test:This is always the thing executives want to you to tell them. “How much money are you to make from this test?” It’s also the estimate most prone to being wrong. Why? Because you’re generally asking the person whose idea it was in the first place to make a prediction about something in the future. And the larger that number is the more likely that person is to get their idea funded and their name in lights. Pretty obvious why that number is suspect. Surely, you need some number to plug in for what the upside is, right? Ok. Again, let's use actual historic data. Of the tests that produced a winner, what was the actual projected lift after you ran the test? It may be much more accurate to use average revenue gains from actual test winners than to concoct a customized prediction for each test you’re thinking about running. Such a prediction would be prone to too much bias. Now, let’s put it together into the new Simple A/B Test ROI Formula:
The only number out of that I would change per test is the Operating Expense value. I’d look at the range of expenses associated with the past tests. You have a histogram of the expenses related to past tests. Have your design, development, and analytics members take a crack at where on that distribution they believe this new test will fall.
But if not, there is just too much subjectivity and personal interests involved to just left to a marketer or business leader to guestimate. I'm more okay allowing a web developer to input on how much time the project would take because I'm not usually concerned s/he has any particular bias towards making the project look exciting. Usually, the person is interested in having their time managed well, which should lead to a realistic assessment. Clearly you should be focusing on the big themes that you believe will drive huge growth over time. By all means, apply proven quantitative approaches to highlighting these big themes (not covered in this post). Making minor changes to marketing copy on a low-grossing product line will probably not produce a positive ROI. But once you’ve identified the big themes, then focus on the tests that have lower Operating Expense and use objective, historical data to assess win rate and potential lift if it’s a winner. A/B Testing is not free. When managed as an investment, however, it can be the most highly leveraged, highest ROI marketing program you have. Please comment below and tell me how you think through the costs vs benefits of A/B testing. Trackback URL for this post:http://www.benchmark-analytics.com/d/?q=trackback/30
|
Short answer:
Why not also take a crack at how big an impact this test will have? If you can figure out a way to do this objectively, based somehow on data, and you have a way of comparing your projections to actuals to recalibrate your methodology, great. By all means do that.
13% YoY Growth
A/B testing does cost money but the point of A/B testing is to make money from it. :)
When I worked at ACDSee.com we did over 1 million downloads per month and my job was to convert free users into becoming paid users. I grew my channel by 13% year over year. How? A/B testing. I’ll tell you how.
With a team of developers, designers, copy writers etc…same team as you listed on your blog, we were still profitable because we made A/B testing our mantra. I also had a GREAT TEAM working with me.
Here is how we managed 6 software products in 12 languages.
Metrics and templates. We had the developers build very sophisticated tracking software that I can’t disclose here. It was so smart that I knew everything about a user. When they clicked, bought, location... I even knew what they had for dinner. (wink wink). We then developed templates that we knew was converting at xx%.
We also calculated conversion rates from every single touch point. That way we could calculate (with some margin of error) every campaign change in advance to see what the changes might be to our bottom line.
My mission was then to try to beat these templates by not using designers and programmers. I did this by having systems and processes in place where I could manage testing environments myself. That way I had high costs upfront in building the AB tests (designs etc...), but when the actual test started I managed it myself via the system we had.
E.g. I knew that “landing page 1” performed at xx% but LP 3 performed at a higher % so by looking at traffic sources, where in the buying cycle they were I could change the messaging on the spot to boost conversion rates.
This is hard to explain but think of my approach as Lego or building blocks. I could easily replace one “bad block” with a “block” that I knew was performing, thus boosting conversion rates.
A/B testing is all how you set it up and track progress. Word of caution though, if you start with good material it will make your AB Tests a lot easier because you don’t have to re-develop designs, programming etc. Do it right the first time and the rest will be easier to manage.
I still use the same process today and it works great but it takes practice and time to set up. Once it's done it should be profitable.
The true "lift" is unmeasurable
The most important impacts of structured testing and data based decision making are inseparable from overall business success. The better job your group does in asking the right guiding questions during campaign design and analysis, the better everything will be implemented and the more sucessful everyone will be with everything they set out to do. When done right, the real lift is orders of magnitude bigger than just the lift from A vs. B.
Factorial models
I think you got the cost side down. Clearly if B gives lift over A, the control or current version, then the returns are clear. But if B has no lift over A, the company has saved money if there was a plan to implement B.
This is a good reason for factorial models where instead of a test control comparison, factors of each website are identified and the impact of each factor is quantified.
Georgette Asherman
Direct Effects, LLC
Costs even when we don't test
I don't really believe this for a few reasons:
What are the total costs in the marketplace of selecting A vs. B without testing and deciding wrong? How many years do we have to carry those costs forward? How is that incorporated into the calculation. What is our success rate in guessing?
While it may be easy to quantify the costs of gaining knowledge as you have tried, it is much harder to quantify the cost of ignorance.
Also, the calculation assumes NO time would be taken if we didn't test. Usually going with the blind guess takes much of the time outlined to pick and plan for that blind guess. I am guessing some of the cost of developing an A/B test would support the implementation of whatever proved best, A or B. (For example, say A wins, the time spent perfecting the A message, coding the A approach, selling A to mgmt, analyzing the performance of A). In a sense you need to compare all time spent before during and after "final solution implementation" with an without testing. I don't believe the costs outlined here are truly incremental.
Some of the staff in your calculation is salaried, and not doing the test does not decrease the costs to the department. If you are currently leasing a car and you have 24 months left on the lease, and if you could have a 24 month lease for that BMW for the same as your current lease payment, the car has an additional transportation cost impact of 0. I bet even your wife would agree. Sure if you used this logic to conduct a large number of tests, eventually it breaks down, and more staff is necessary, but for a small number of mission critical tests, I only believe they add the outsourced portion of the staff segment.
Can't manage what you can't measure
Lynda - There's a ton of truth to the points you bring up. Here's how I'd respond.
My post was in reaction to a number of things I've observed.
1) Companies moving towards testing "everything" and really thinking very little (if at all) about the costs side of the equation. So there's no balance between the realistic payoff and the amount of effort that goes into developing the test. The simple question of "are we likely to make more than we spend?" gets asked very rarely! The question typically stops at, "are we likely to make more?".
At the extreme, it's somebody at the top who has an idea, and somebody in the middle who say, "yes we'll go do that right away". And the critical assessment of whether it's really worth it - or just how to go about it efficiently - is completely lost. So that's what I'm hoping to get into a dialogue about.
2) Overly optimistic estimates on how much a new test idea is likely to bring in. And this in sense feeds the notion that we could throw 10 employees at a project for a month and it'd be worth it. Why does this happen? It's not unusual for the justification to run a new test include numbers that are created in a vacuum. For example, if the baseline conversion rate for visitors who see the product page is 4%, the justification for a test might be, "What if we could just increase the rate 1 percentage point... from 4% to 5%. That would be $2M!" And the other marketers and managers agree, "Yes, how hard could it be to increase it just 1 point?".
Had they taken into account that over the last 2 years we've already run 10 different A/B tests on that same page, and the distribution of outcomes ranged from -0.2% to +0.2 (in other words a relative drop or lift of 5% from the baseline), then the idea that some new test could raise the rate from 4% to 5% (a relative increase of 25%!) would seem somewhat preposterous.
Lynda - You're taking a much more expansive view, which is great!
You bring up quite a few points, some of which (like how to carry costs forward) simply relate to refining the model and putting into to operation. But there are two components to your argument that I'd like to weight in on.
First is that companies are saddled with a certain headcount, or fixed overhead. Your premise is that if they're not doing testing, they're doing something else, so the cost doesn't really disappear as I've supposed it does, even if we were to eliminate a particular test.
I agree - up a point. As you suggest, to run a few mission critical tests, you're using the same folks you've got on staff already. Now, in some companies on the bleeding edge of optimization, there are non-trivial sized groups devoted almost entirely to A/B testing, behavioral targeting, data mining, and the like. When managed right, I believe these groups would turn up at the very top of the list in terms of contribution margin or ROI.
But if in theory we had a completely accurate assessment of ROI for each individual work group, and compared it against all other workgroups in the company, and some particular group had a relatively low ROI, I have no doubt they'd be cut in the next round. So I'm not convinced that the exercise is pointless.
At many companies there are essentially dedicated resources to testing, and if the ROI wasn't high relative to other workgroups, it would be an opportunity to trim expenses. Companies are not saddled with salaried employees they can't cut. They cut them all the time. Though I'm not sure it's always because management has a real, true sense of who is contributing to the bottom line and by how much.
If every workgroup - from Testing & Analytics to Corporate Communications to Direct Mail to Engineering - could boil down their work into an equation for ROI, then each would be in a much better position to raise their ROI. Back to the idea that "you can't manage what you can't measure". Does Corporate Communications have a way to measure their contribution margin or ROI? If not, then how to they know which actions raise it? Luckily, in A/B testing we are in a good position to be able to measure, and therefore manage our contribution.
I believe groups involved with A/B Testing, behavioral targeting, data mining, and predictive analytics would likely be listed at the highest contributors. By putting some rigor around quantifying costs, I think these groups can tune their processes to find even HIGHER efficiencies.
Your second argument I'd like to address is the notion that companies are going to make these changes or run these projects even in the absence of testing. If so, I'm overestimating the costs associated with A/B testing specifically. Many of the costs would be borne anyway, and the testing component is only slightly incremental.
When I think back on all the last hundred of so A/B tests I've been involved with, I think that statement rings true in quite a few of the situations. Just how many though, I'd have a hard time estimating. I'd like to think that management wouldn't have decreed frequent changes to title text, call-to-action buttons, product line-up, or top navigation had not they had some way to measure it. Though if I go back in my mind to the world "pre-A/B testing", I recall we believed we did have measurements methods, only they were a bit more crude in some ways -- surveys, usability studies, clickstream analysis, etc. So after thinking about this argument you bring up, I'm tending to agree!
Ripple effects?
As a 'From the Hip' calculation, you pretty much described how most companies which we've dealt with perceive ROI. An underestimated value of these experiments is the factor of how the 'journey' plays into the value of ROI. Particularly, with the impact on the culture of the company internally, the intrinsic value of testing is the breakdown in barriers which prevent people from looking at pages subjectively, and persuasion design/usability objectively.
The exposure to really insightful testing from experienced and qualified candidates is a ripple in a pond. Further more its iterative, and, by nature, supported with quantifiable results which removes the contentious layer of the HiPPO or ''Rat Race" type atmosphere which EVERY company has. When a platform for incongruent focus can be given a forum for productive discussion to a productive ends, the outcomes move the decimal point off the page.
The way I see it, if a company or serious player in internet technologies has not, or simply refuses to move to make testing part of their curriculum for improvement, they/it, in essence, are passively sentencing the future of their innovation to a major roadblock. Whatever the cost (you mention $50,000 which I think for most companies is probably not in their budget under current constraints), the measureable ROI will eventually justify the investment through the periphery impact of having introduced a progressive style of active growth management to all parties involved, and, created a catalyst for communication between organization left and right brained clusters. On paper, the numbers turn the simple tangible ROI calculation into a fraction of its exponential ROI impact distributed across new angles for thinking and cooperation.
I know this answer is from non-material calculation field of the response class, however, it would seem to me, after our experiences, that this, more than anything is where the true value in A/B, Multivariate, Usability, or any other testing would reside. Therefore, I would contest that this is, and should be, the ROI calculation objective. Quantifiable, maybe on some level, necessarily quantified...not really...perceived...most certainly...proven...Look at CableOrganizer, Google, Crutchfield, Intuit, WebMD....if you think they've found it profitable on their diversified budgets, then you can bet a program exists to leverage the same impact with whatever budget can be applied.
Sorry for the long-winded response. )o0(
-Daniel Shields
Founder, Wicked|Sciences
free is not cheap enough!
Jared:
You make a great point. Many times, we've been faced with the "it's free, what do I have to lose? I'll just *try* it" conundrum.
The trouble also comes when you try to define an ROI -- people always have their own assumptions for inclusion into the calculation.
Fortunately, there is innovation in this space of A/B/n and multi-variate testing (MVT) ... no longer are marketers left hunting for the "big change" in A/B-land that will get them the +ROI they're seekign. Nor must they have $$$ and loads of traffic to waste on MVT.
What marketers really need is:
- a tool that allows lots of little changes to add up to a +ROI
- a way to execute MVT without hundreds of thousands of visitors
- a low capital outlay and little-to-no time expenditure in executing the testing
Not easy to find, but definitely out there.
CJ
Post new comment