XChange Redux: "In Analytics We Trust"?

trust1_0_0.jpgReading Brooks Bell's Insights From XChange post caused me to contrast the last three years of XChange. Brooks and I both attended the huddle entitled "Testing, Testing, Testing : Building Consensus and Evaluating Results" led by HP's Matthew Wright. I showed up a few minutes late but was immediately struck by how much more engaged and experienced the attendees were this year on the topic of website optimization.

Last year at XChange I had the privilege of co-leading the huddle on A/B Testing with Dylan Lewis. And last year of the three huddles I led the Testing huddle had, suprisingly, the lowest percentage of attendees with deep hands-on experience in the subject. This year was different. It was just as engaging and interactive as any other huddle I attended that day, which I take as an indication that the level of expertise is rising.

So that's progress.

Brook accurately emphasized trust as a recurring theme. This lack of trust is an obstacle to the prudent use of data to drive better decisions.

Now, lack of trust can be looked at from many, many angles. From my experience, one often overlooked aspect of this mistrust is a general unease with the very concept of uncertainty. And yet uncertainty is a part of website optimization and testing (e.g., "we're only 80% confidence A is better than B").

Many people in business seem to have a visceral mistrust of data that contains within it in some level of uncertainty. For lots of otherwise intelligent folks, something about the concept of making "data driven decisions" seems to fight with the notion of "there are some things we cannot be 100% certain of". And yet website optimization involves both hard data and a dash of uncertainty.

Of course the uncertainty can usually be quantified. But now we're speaking of abstract things. And business is plain and simple, right? "If I wanted to deal with uncertainty, I'd have gone into quantum mechanics".

So it may take some formal training in the basics, as well as practical experience, to get over this unease. Then the Executives can begin again to take swift and decisive actions. Emerging with an enhanced grasp of "distributions of outcomes", the Company is more ready to act strategically and respond quickly when the future takes an unexpected turn.

Lack of trust around data may also stem from one's own experience being on the receiving end of someone else using "data as a weapon". We've probably all seen it in our organizations at one time or another. The storyline is written on some great success or some dismal failure, and the proof is in the numbers presented. But we have inside knowledge that those numbers were misused or misleading.

We can only hope that building up the overall bench skills around quantitative analysis would serve as a form of inoculation - at least for the most egregious abuses of data. And at the same time, those of us who have seen the profit potential realized will bring along some others.

In Analytics We Trust? I'm not so Pollyanna to envision a time when data fluency will be second nature to all. I do, however, believe analytics' ultimate proof will be in the longevity, EPS, and P/E of the companies who Trust the most.

I want YOUR opinion. Feel free to comment below, or vote for what you think is the root cause in this 1-question poll.

Subscribe via RSS To Get These Upcoming Blog Posts
  • "Choosing Your Testings Goals" (Sept 29, 2009)
  • "Evaluating Test Ideas" (Oct 5, 2009)

Trackback URL for this post:



KevinErtell's picture

Good post, Jared. As we discussed in some of the huddles I was in, uncertainly is part of web analytics and just about all other business metrics, for that matter. I believe we run into more problems when we give the illusion of precision by giving specific numbers when we have no business giving specific numbers. I discussed this topic in one of my blog posts called "How sales forecasts are like baby due dates." It's here: http://www.retailshakennotstirred.com/retail-shaken-not-stirred/2009/09/...

You allude to the same point when you reference being "80% confident that A is better than B." To me, the reality of recognizing some uncertainly allows us to make more reasoned decisions and plan for contingencies. If we as executives insist on a specific number when one cannot reasonably exist, we are doing ourselves a disservice and we inevitably end up disappointed with the results of the decisions we make.

To me, the big key is to change the conversation. As we discussed at X Change, let's start talking about web analytics in statistical terms instead of accounting terms.