Visitor Monitoring What Every Retailer Wants for Christmas | Keynote

What Every Retailer Wants for Christmas

Retailers look forward anxiously to the holiday season, counting on their online channel

He’s making his list, and checking it twice, and the entire retail industry is hoping that everyone on it will be marked “nice.” This year’s holiday retail season is being anticipated variously with tentative optimism and trepidation, and everyone, from the retailers themselves to economists, the stock market and the government, is hoping for a pleasant surprise.

The most watched forecast, from the National Retail Federation, predicts a conservatively pessimistic decline of one percent for the November-December holiday season, to US $437.6 billion. While this would make it the second-worst holiday season in more than four decades, the “glass is half full” set sees a silver lining in this projection. It’s not nearly as bad as last year’s 3.4 percent holiday tumble, which made 2008 the worst year since the government began tracking retail revenue. 1National Retail Federation, “NRF Forecasts One Percent Decline in Holiday Sales,” news release, 10/6/09


While overall retail is anticipated to be down, some are predicting online’s share of holiday sales to be up by 10 percent or more. The NRF projects that five percent of holiday sales, or US $156.1 billion, will occur online. That’s more than a 10 percent jump over last year’s online sales of $141.3 billion. According to a Burst survey reported in Internet Retailer, 85 percent of consumers plan to shop online, and 69% plan to buy online. And a third of consumers who did not buy online last holiday season plan to make purchases online this year. 2Internet Retailer, “Consumers say their holiday budgets are tight, but 85% will shop online,” 10/7/09 Online could end up being the bright spot in the holiday season, for those merchants who have an effective Web presence that performs to consumers’ higher and higher expectations.


The fate of the holiday retail season — and for overall 2009 retail sales — depends on the mood and confidence of consumers, which has been anything but reliable. Any kind of negative economic report in October or November, whether it’s bad unemployment news, stock market woes, or a spike in oil prices, could send consumers running for cover, again.

The current consumer mood, though, has taken a tentatively positive turn. In shopper surveys conducted by Retail Forward in September, 9 percent of shoppers indicate that they will spend more this holiday season than last, a 2 percent gain over August. And in the spirit of this economy, where less bad news is considered to be good news, 40 percent of shoppers say they intend to spend less this year than last — but that’s down from the 43 percent in August who said they’d be more Scrooge-like. 3Retail Forward, “Teetering Back to Cautious Near-Term Spending Intentions,” ShopperScape™ Newsletter, October 2009 Affirming a slight positive shift in mood, chain store sales in September were up by a tenth of a percent year-over-year, which was the first real positive movement in over a year. 4Internet Retailer, “The Worst Might Be Over for Retail, Two Groups Report,” 10/9/09

And in another good omen, Reuters and the University of Michigan report that their consumer sentiment index in September reached its highest point since January of 2008. The September index, at 73.5, gained significantly from the August level of 65.7. 5NASDAQ, “Indicators Offer Mixed Signs for Economy,” by Bill Laforme, 10/9/09


A few retailers are feeling upbeat about the coming holiday season. Best Buy is planning to do more holiday hiring than last year, partly because of strengthening demand for flat-screen TVs, smartphones, and netbook computers, and partly because of market share they captured following last year’s demise of Circuit City. 6Business Week, “Best Buy Plans to Boost Holiday Hiring,” by Matthew Boyle, 9/30/09 Target plans to have 26 new stores open in time for the holidays, including five “Super Target” stores with full grocery departments; this, and more aggressive pricing, are moving Target closer to a head-to-head brawl with Wal-Mart. 7Inside Futures, “Target Opening Stores in Time for Christmas,” by Ockham Research Staff, 10/7/09

Most retailers, however, are simply looking for ways to maximize their share of holiday revenue. Deep and earlier discounting is expected to be widespread. And in a throwback to an earlier era, and at least partly in response to a growing paucity of consumer credit, the layaway plan is making a comeback. K-Mart and its parent company, Sears, are both offering not only in-store layaway plans, but also online versions as well. Consumers can reserve their merchandise and make payments online, and when they are paid in full, pick up their merchandise in the store. 8bNet, “Holiday Challenges Continue to Loom, Yet Kmart Adapts the Steidtmann Way,” by Mike Duff (blog post), 10/8/09 Whatever it takes, even down to giving away free gift cards, retailers are doing to get consumers to buy, either in the store or online.


With retailers counting more and more on their online channel to help make their holiday numbers, online performance is — or should be — a top priority as the season approaches. And Webmasters and Internet marketing executives have good reason to be concerned, as consumers have approached a zero tolerance level for sites that deliver less than near-instant gratification.

How fast is fast? Consumers today want sites to load literally twice as fast as they did three years ago — 47 percent expect pages to load in two seconds or less, according to a Forrester Consulting study reported in Internet Retailer. More than four times as many pin site loyalty on quick page loading. And perhaps the most ominous finding of all: four out of five visitors are less likely to return to a site that’s left them hanging, and more than a quarter are less likely to visit that retailer’s brick-and-mortar locations. 9Internet Retailer, “The new page-loading threshold: 2 seconds,” 9/15/09

“The question is latency tolerance,” says Donald Foss, director of global testing services for Keynote Systems. “How much patience do you have? Consumers today don’t have much. And even if the problem is line speed, users will still tend to blame the site.”

For online marketers, the conclusion is clear: there is a direct correlation between site performance and revenue.

“Especially now when everyone’s got incredibly high speed Internet connections into their home, they’re used to pages popping very, very quickly.” says Sears Canada AVP of eCommerce Simon Rodrigue, “One of the reasons we focus on it so much is that we have seen this correlation — that by improving performance, even when you’re running good performance — by improving availability incrementally or improving performance incrementally, we do see a return from a conversion perspective, and hence a revenue perspective.”

Conversely, incremental decreases in performance result in incremental decreases in revenue.

“When you factor the economy into it, people with limited dollars are looking for the best deals,” notes Keynote Systems’ Senior Consulting Manager Ben Rushlo. “And the difference between making a sale or not could very well be site quality, especially in highly competitive areas like electronics. Consumers want the functionality — the reviews, the ability to zoom in on photos, and so on. But if it’s not working well, if it’s not fast, they’ll jump over to the competitor’s site where it is working.”


Just a few years ago, when broadband envy was a common condition and DSL was considered high-speed, testing Web site performance was largely an internal affair. Follow basic best practices in building the site, make sure the servers were up and the pipes open, and you were ready for business. Consumers coming off the painful slowness of dial-up had far lower expectations and far greater patience.

But that all seems like such a long time ago. While the U.S., at 60 percent broadband penetration, ranks 20th (!) globally (far behind South Korea’s 95 percent), a solid majority of U.S. users now have fast connections. 10Ars Technica, “US 20th in broadband penetration, trails S. Korea, Estonia,” by Nate Anderson (quoting Strategy Analytics study), 6/19/09 They’re paying for speed, and they expect sites to be fast. Add in rich functionality, video, Flash, etc., and you’re looking at an experience made or broken by site performance — performance that can no longer be effectively measured from the inside out.

Believe it or not, some major U.S. retailers still do not have an organized regimen for external testing of their sites, or for determining if they can handle a heavy surge in holiday traffic.

“Some big players — I won’t mention any names — still have no external monitoring in place,” says Rushlo. “They have a mentality that ‘we don’t need that piece,’ believing that internal monitoring and instrumentation can handle it and give them the data they need. But nothing can stand in for measuring your user experience from out in the Internet where customers live.”


In the broadest sense, testing falls into two categories: ongoing evaluation and tweaking to optimize daily performance, and peak load testing to determine overall site capacity and potential breakpoints.

In either case, the only way to quantify user experience is to measure what users are actually experiencing. Unless you’re the local bike shop serving only your immediate area, the testing needs to be done across a wide geography and multiple backbones. And it needs to use an actual Web browser, and go through the same type of page view sequences and transactions as a typical user would. There’s simply no other way to get a true perspective on what users are really experiencing.


Ongoing performance testing, done on a routine basis or following upgrades, patches, or redesigns, can be as simple or complex as the site itself. Assuming the back of the house is in order — that servers are performing up to speed and all of the infrastructure is optimized — the focus of the testing is on the front end. Individual pages are tested and analyzed, and those performing below par are deconstructed to identify the structural problems or elements that are causing the slowdowns.

A single element could be dragging down the page load, or it could be less-than-ideal placement of JavaScript on the page. The point is that testing and identifying the problem pages or transactions is only half the problem. The other half is being able to pinpoint the causes and correct them. A robust performance evaluation regimen accomplishes this.

“We’ve gone in and found simple things, simple changes in settings that were able to improve performance 60 to 80 percent,” Rushlo says. “Data and metrics can be overwhelming. You don’t want performance to become just another bunch of noisy data. You have to have the ability to take it and make decisions, and make improvements.”


Every holiday season, there are stories of the sites that broke in one way or another under a sudden crush of shoppers and transactions. Losing a site for a few minutes or a few hours on Black Friday, Cyber Monday, or in the days leading up to Boxing Day is likely to have a deleterious effect on retailer revenues — not just for the transactions lost in the downtime, but in the customers lost because of the bad experience.

This year could test the limits of retail sites more than ever before. Not only are more consumers planning to do more of their shopping online, significantly more of them are carrying iPhones and other smartphones and will be accessing sites on the go. Scanning bar codes while you’re in the store and running a comparison price check? Yes, there’s an app for that, and it creates a whole new stream of site traffic that hasn't been encountered before.


There is more to determining site capacity than sheer volume of concurrent users. While that certainly gives an indication of break-point, it does not necessarily provide a true, real-world perspective on site capacity.

“The Web doesn’t work that way,” Foss explains. “Customers are coming and going all the time; they’re not all there at once. A more sophisticated load test uses ‘arrival rate’ methodology, for example, that reflects more what you would see in Web analytics — page views and time spent, among other things.”

Unlike a scheduled event — President Obama’s inauguration, for example, when news sites could expect with certainty a tremendous surge in users in a specified time frame — the retail traffic pattern is that users are continuously coming and going. And they’re not all doing the same thing when they arrive.

“Behavior modeling comes into play,” Foss says. “You need to take into account the many different types of users on the Internet. Usually people are only using a few pages on a site, there are a range of pause times, and different page types with different elements — all these things and more need to be factored in.”


You can test on a QA system and get some insight and make some adjustments. But ultimately, the live, production system itself has to be tested, and that can be a scary prospect for retailers depending on the site for revenue.

“We’ve tested production systems at just five percent of the projected peak load and crashed them,” Foss recounts. “Most of our clients do full testing on their production systems and allow outages to occur. It’s the only way to know how the live site will actually perform. Of course, we try to target low traffic times — we run a lot of tests in the middle of the night. We can also target some level of performance degradation and then stop the test. If a site does go down, Keynote can cancel a load test in 60 seconds. In most cases, users have no idea that the test is even taking place.”


All is for naught if the shopper doesn’t end up buying what’s in their basket. And this is a real and costly problem for online retailers. As Claire Cain Miller writes in the New York Times, “Shoppers rarely drive to the mall, load up their carts and then abandon them in the middle of the store. On the Web, though, it happens all the time. 11The New York Times, “Closing the Deal at the Virtual Checkout Counter,” by Claire Cain Miller, 10/12/09 And with more and more shoppers hitting more sites to comparison shop, the problem is only getting worse. comScore reports that in Q2 this year, the visitor-to-buyer ratio dropped some 30 percent for Zappos, and 26 percent for The Gap. 12ibid

“Peak load testing is just part of the equation,” Foss says. “You have to have the ability to do end-to-end transaction testing, including taking the entire order, checking out, and confirming the order.” In order to do that, the retailer’s e-commerce system has to have the ability to process a fake order from start to finish — or the testing partner has to want to buy a lot of the merchant’s product.

Some retailers are trying to incentivize shoppers to complete the process, alerting them to low inventory levels, or putting time limits on special pricing. Neiman Marcus, for example, runs two-hour sales online to push people to checkout. Whether it’s making sure the check-out process is intuitive and fast, or giving shoppers an incentive to press the “complete order” button, the check-out process is a critical focus for testing.


If you haven’t fully tested your site now for this holiday season, it’s time to stop reading this and get to work because time is running out. In any case, now is the time to make a New Year’s resolution to continuously monitor your site performance throughout the coming year, and do systematic load testing well before the holiday peaks, including Valentine’s Day, Easter, and all the traditional retail holidays.

“The true best practice is a continuous improvement process,” Rushlo concludes. “The retailers that are successful every year, not just at the holidays but all year long, have in their culture a continuous improvement process, a culture that cares about the quality and performance of their technology.”

Back to Top