Coffee, Sun & Technology

The accuracy monster never seems to die. Judging by the number of accuracy inquiries from customers, Web analytics users seemed to understand, in simple terms, that their favorite WA tool can’t be 100% accurate (and that it’s OK). However we are clearly seeing these inquiries picking up again. People are legitimately concerned about how the cookie deletion problem is affecting the numbers – especially if money is involved.

Briefly, I want to give three fundamental reasons why it is not reasonable to ask for 100% accuracy from your favorite Web analytics vendor.

First, recording Web site events is not an exact science. For example, for an ASP package (or a software solution) to capture that User X landed on your homepage from Google, dozens of things need to work perfectly right – some of these things are purely unpredictable (server load, network load, JavaScript support on the client side, reverse proxies, etc).

Second, even if by magic we could guarantee that all events on the site will be recorded without any loss, sometimes the information we’re looking for is just unavailable. Consider the simple scenario where a long time loyal visitor and customer of your site just has purchased a new PC, and is therefore clear of any customer cookie. How would be know that this user is a returning user?

Last but not least, many online metrics are subject to interpretation. What is a click? Just a simple click on a link, or, a click on a link combined with an actual view on the target URL? Ask your Google rep – then ask your CFO. You’ll get different answers.

There are dozens of other reasons. In the end, like a coworker of mine puts it, there is a Heisenberg principle in analytics. But it doesn’t diminish in any way the core value of analytics, which is to drive action and change on your site.

Business KPIs are easy. User experience KPIs are challenging to define because no one really knows what matters. How about (for retail sites):

- Page display time
- Error pages per thousand sessions
- (Page A -> B -> A) loop count per thousand sessions
- Checkout form entry error rate (per attempted checkout)
- Failed login attempts per session (for returning users)
- Forgot my password requests

For this week vs last week with per percentage variation + historical average.

I had a nice brainstorm on A|B testing today with a customer of ours. First, I was impressed that they thought building (or buying) an A|B testing platform was a key component of their online strategy. They want to run tests every single day and will assign dedicated resources to constantly develop challenger content.

Second, I was even more impressed with their longer term vision. They view A|B testing as a tool not only to immediately increase conversion rates, revenue, etc for a given campaign/landing page - but also to build longer term Intellectual Property for the site.

A|B testing as a research tool to better understand how our customers shop. The idea is not new but few have executed on it.

More thoughts on the ScanAlert article on Window Shopping (see previous post on the subject), as Nigel Ravenhill was kind enough to send me a PDF copy of the full report:

First, we know that window shopping visitors use shopping carts to save the product they’re interested in. Therefore, why not make it a little more friendly to returning users - for example: Thanks for coming back. Is there anything we can do to help you make a decision? (Email, Phone, Chat etc).. Still undecided? Have you considered these products:… (Show a Cross-sell or related product).

Second, more strategically, how can we win the window shopping battle against competitors? The ScanAlert report indicates that (a) Price and Availability, then (b) Safety and Trust are key factors driving window shopping sales. But these alone will not necessarily tilt the balance in anyone’s favor. I think we’ve got to see the cart page for the returning user as part of the sales cycle, since users re-accessing the cart are actually even more qualified to buy. In other words, better market to these return visitors (discounts, coupons, etc…).

Thinking of Shopping Carts as qualifying entry points rather than exit points - that is the thought provoking mind-shift needed to win the window shopping battle.

Excellent article on “window” shopping from ScanAlert,

The average time delay between a consumer’s first visit to a Web site and their first purchase was just over 19 hours, with over 20% of shoppers delaying their buy decision for more than three days. About one-third (35%) of shoppers took more than 12 hours to make a buy decision. Twenty-one percent took more than three days, with 14% of these “cautious shoppers” taking more than one week to decide where to buy.

ScanAlert said the study was conducted on more than 8 million shoppers and confirms the behavior I have seen with my clients (as far as the time it takes for a purchase to happen). If you plug these numbers into my original Excel spreadsheet to assess the impact of users deleting their cookies (see this post), you will find that 3.78% are missed because of users deleting their cookies (over 28 days). See the revised spreadsheet here.

Again, plug your own numbers in there to make your own assessment. In any event, try to understand the impact of the cookie deletion issue on your business before you reach any radical conclusion about the use of cookies.

Customer Experience Pros need to decide if they’re running a Web site or a Web business. For the ones running a Web business - i.e the ones who entered the conversion rate contest - it’s all about results: they are the magicians in charge of finding the right formula for higher site conversions:

- How many steps in the checkout?
- How do we help users find the right product, quickly?
- What should our home page look like? ETC.

Jakob Nielsen’s answer: Look around. If you are actively looking for usability studies, advice, tips, I highly recommend Nielsen’s Alertbox’ newsletters, where you will learn to distinguish between good and bad design. I have researched myself the usability topic and listened to my customer brainstorming about usability issues - problems are very much the same across all retail sites and so are the solutions. As Nielsen wrote in 1999:

Web design is easy: If you are thinking about how to design a certain page element, all you have to do is to look at the twenty most-visited sites on the Internet and see how they do it.

In that same newsletter, he talks about the idea of Web standards and when/why they should be implemented. Heidi Adkisson from the University of Washington wrote an excellent thesis (”Identifying De-Facto Standards for E-Commerce Sites”) on the topic as well which nicely complements Nielsen’s own research.

But here is the key challenge: Since usability and design are by and large all figured-out, how do we achieve high usability (and therefore results) while being different?