Making a Site, and Checking it Twice
A guide to preparing a Web site for the holiday rush
If any retailer needs proof of the importance of load testing, they need only look to the headlines around the launch of the new Apple iPhone 4. All day on June 15, Apple and AT&T’s online stores went down and out. Tech bloggers spent hours trying to place online orders, and hours more on hold with customer service. An official announcement from AT&T called it “the busiest online sales day in AT&T history.” Yet despite the problems, they managed to sell out their first day’s allotment before the day was out.
Retailers dream of having such problems as rolling out a new blockbuster and taking orders for 600,000 units in one day. But you can count on one hand the number of sellers who command such fanatic loyalty and patience. For the rest, site outages, freezes and checkout failure mean lost revenue that will likely never be made up—and those losses mount minute by minute, hour by hour, especially during the peak holiday season, when most retailers rack up most of their annual business. During those critical four to six weeks at the end of the year, when the site breaks, the bottom line breaks, too.
So: it’s July or later as you’re reading this. The biggest, most successful online retailers have been prepping their sites for the holidays for six or seven months now. The next tier started in June. Retailers who have not started to stress test their sites could be looking at many sleepless nights come the last months of the year, and potentially serious hits to their 2010 revenues.
Online makes the bottom line
There’s no underestimating the importance of e-commerce to overall retail success, particularly during the crucial holiday season. Every year from 2000 through 2007, online sales posted double-digit growth, handily outpacing in-store sales. 1U.S. Department of Commerce And last year, as retail struggled to recover from its worst holiday ever (in 2008), online holiday sales grew at four times the rate of total retail growth—though admittedly modest growth, at 4 percent for online sales, 2comScore press release, “comScore Reports $29.1 Billion in U.S. Retail E-Commerce Spending for Full November-December Holiday Season, UP 4 Percent vs. Year Ago, 1/6/2010 and just 1 percent for total retail. 3Internet Retailer, “Total holiday retail sales rise 1%, trailing e-retail’s 4% increase (For the year, e-commerce sales grew 2 percent, while overall retail sales fell by nearly 3 percent, according to the U.S. Department of Commerce.)
With online playing such a critical role in holiday sales, it is impossible to ignore the real bottom-line costs in lost revenue of site slowdowns and outages. Site metrics and sales data show the revenue value of every site visitor, and site testing can show the actual cash lost when visitors abandon the site because of poor performance. As the accompanying chart shows, using actual results, a traffic spike 25 percent beyond the site’s optimum capacity can cost $100,000 in lost revenue every hour.
Further complicating matters is the huge unknown factor of mobile for 2010. With the tremendous and ever-accelerating proliferation of browser-equipped, app-loaded smartphones, mobile is sure to play a bigger role than ever, and likely to exceed even the most optimistic retailer’s projections. All those smartphones searching for products, comparing prices, and making purchases will be accessing the same back-end databases and applications as all the PC-based browsers, putting even more stress on e-commerce systems.
The first billion-dollar online day?
One likely milestone in the upcoming 2010 holiday season will be the first billion-dollar e-commerce day. Sales broke the $900 million mark on Green Tuesday (December 15 last year), and at least nine shopping days exceeded $800 million. 4op cit A major east coast snow storm kept shoppers home-bound and contributed to brisk online sales. This year, mobile, with shoppers logging on even as they are out in the stores, is likely to have an even greater effect.
First things first: Updates
Getting a site into the near-final form it will have for the holiday season is the first step to making sure it will handle the demand. New features and functionality—like enhanced product presentation, reviews, and personalization options—should be well under way if not complete by now. Sports fashion leader Lacoste is one retail marketer that understands the importance of readying major site revisions early.
“We’ve been working on a huge upgrade of our site and have been doing side-by-side testing with Keynote in order to measure the current experience against the new experience,” Lacoste E-Commerce Director Maryssa Miller toldBenchmark. “Our criteria is to be better in performance than the current existing experience, even with all the new functionality. And we want to make sure that any kinks are worked out prior to the busy holiday season, obviously.”
“We’re adding larger images with greater resolution,” Miller continues. “We’re adding new features to be able to check your tax and shipping or your total costs much earlier in the process. We’re also adding social networking tools—the ability for people to more easily share products through any of the major social networks.”
Best practice: Build pages lean
Performance management starts with how the pages are built, which often presents a dilemma for retailers. With more and more products approaching commodity status and available at multiple online outlets, site experience becomes a key differentiator. Retailers want to create a rich experience for visitors with interactivity, dramatic product presentation, perhaps Flash, personalization or other features to set themselves off from the competition. But a heavy load of features and functionality can drag site performance down, often because of third-party content, and, instead of making visitors sticky, can drive them to leaner, faster competitive sites.
“Ideally, it’s best to focus on limiting your number of third-party content and domains on each page to roughly six domains at most,” says Keynote Consulting Manager Cliff Crocker. “As opposed to what we see for a lot of retailers, where there’s 20 or 30 different domains that are killing performance on their page.”
Crocker also recommends keeping the number of objects on a page to 40 or 50; many retailers are packing their pages with 150 or more elements.
Factor mobile in from the start
The successful retailers this year will have built mobile into their strategy right from the start—not just as an afterthought to the “main” site, but side-by-side with it. Shoppers carrying smartphones are using them to check prices, locate products, find deals, look at reviews and, more and more, to make purchases. (See “Shoppers, Start Your Smartphones!” in Benchmark.) Many retailers were surprised at the amount of mobile traffic they got during the 2009 holiday season. And there will be millions more smartphones in the hands of shoppers this year.
“We were very surprised,” says Lacoste’s Miller. “And I think that’s where we really realized that we need to focus on [mobile] for 2010. We already knew we needed to focus on it, but I think [the 2009 season] was just really additional proof and we knew that it needed to be a huge part of our strategy for this year, and then continue to be part of our strategy going forward.”
Because of the inherent slowness of cellular networks and devices, mobile sites need to be even leaner and meaner than wired Web sites. It takes some hard decision-making and analysis of what is essential for users when they are browsing on the go and what it takes to satisfy them, including their need for speed. Search results can be confined to return four or five results, for example, instead of the 40 or 50 that might be delivered on the wired Web. And perhaps tracking pixels are needed only on the landing page and cart page, instead of every page on the site.
“There are some pretty strict rules around how many elements you want to have on a mobile page,” Crocker says. “Where we say 40 to 50 over the wired Web, we’re looking at 8 to 10 elements on a mobile page, just because of all the latency that’s encountered over the carrier networks. So step one is, you’ve got to build a dedicated mobile site. There’s no way you’re going to port what you’re putting up on the wired Web or pare that down. You’ve really got to build it from scratch.”
Step 1: Establish your baseline and project traffic including mobile
Knowing your site’s baseline—how much traffic it can handle right now—is the logical starting point to begin understanding how much work needs to be done. Then, projections need to be made for how much traffic and sales are expected for the holiday season. Typically the purview of the sales and marketing team, this is where it gets a little tricky; consumer behavior is difficult to project, especially in these times of continued economic turbulence. Whatever the traffic and sales projections, it’s up to IT and the Web team to make sure the site can handle any unexpected surge in site visitors.
“From the technology side, you should always treat whatever the business tells you with a grain of salt,” says Keynote Director of Global Testing Services Donald Foss. “You may say, ‘I’m going to add 50 percent as planning-appropriate, and then I’m going to add another 30 or 50 percent as a technology buffer—an insurance policy,’ and so it ends up roughly double, or maybe 250 percent, of what the marketing department projects.”
Many, if not most, retailers still fail to account for mobile in their traffic projections, and that could be a bigger mistake than ever this year. Visitors logging on from mobile devices are calling on the same resources and databases as visitors coming through PC browsers, and so are adding to overall site load. Making matters worse is that those mobile connections are extremely slow; traffic coming in through mobile actually holds the connections longer, and consumes more than its share of resources.
“In planning for traffic, it’s important to account for the total sum of the traffic from a holistic standpoint,” Foss says. “It’s important to make sure you’re actually testing the mobile site and the regular Web traffic all at the same time, in the correct proportions and in the right demographics, to see how it works.
“Early on, you take a multi-pronged approach. You test the core Web site by itself. You test the mobile site by itself, and make sure they’re both tuned and working correctly. Then you start the holistic testing, with the full sum of traffic from the correct sources that you predict. If 15 to 20 percent of user traffic is coming over mobile, then we will actually have 15 to 20 percent of the test traffic running mobile-type connections, and make mobile scripts with mobile line speed coming into the site.”
For some retail marketers such as Lacoste, the mobile and wired Web experiences are tied tightly together, so the necessity for holistic testing is apparent.
Speaking about their iPhone app, Lacoste’s Maryssa Miller says, “My favorite part is that it’s actually connected to the e-commerce site. I’ve seen other apps where it is a very separate experience, but ours is very much integrated, which I think is great because we’ve found that a lot of people will save items in their shopping cart and, for whatever reason, they don’t purchase them. This way, if they decide they want it, they can access their shopping cart on the e-commerce site or directly from the app. That’s a really usable feature and one of the best things about it.”
Make testing a real reality check
In a very oversimplified nutshell, with load testing you are throwing users at the site until it breaks (which is why live-site load testing is typically done in off-hours). But to be realistic, accurate, and reliable, a load test has to be far more sophisticated than simply simulating a gross volume of visitors showing up, which is the essence of the often-used but imperfect “concurrent user” methodology.
A concurrent user approach delivers a picture of performance that is far from accurate. Basically, it measures a static threshold. Here’s 100 thousand users. Does it break? OK, add another 10 thousand. And another. The methodology assumes incoming users will patiently wait their turn to enter, and that everything will be fine until they do. But in reality, those additional users are trying to log on and being turned away, and the experience for the users already on the site degrades, often quickly. Testing only for concurrent users does not accurately mirror user behaviors and actual site traffic dynamics.
In reality, users are continuously coming and going. Those with slow connections or mobile devices are taking longer. Some who are already familiar with the site are in and out quickly. When the site slows down under the load, more and more users pile on, because it’s taking longer for the previous users to complete their tasks and leave. The load is never static.
And users are not merely numbers going in and out. They are individuals with individual thresholds of patience—and that is the critical factor to know when it comes to converting visitors to sales, and minimizing revenue lost due to poor performance.
A more accurate representation of site load is created using an “arrival rate” methodology, which factors the peaks and valleys of traffic and availability based on likely user behaviors and site slowdowns. Hand-in-hand with this methodology is behavior modeling, which factors, among other things, users who are familiar/unfamiliar with the site, their tolerance for delays, and their tenacity in sticking with the site until they accomplish their tasks.
“There’s a certain amount of slowness that can be tolerated, and a certain amount that won’t be,” says Foss. “That’s latency tolerance. There’s also tenacity, which means essentially, how important is it for a person to finish the transaction they’re on? How likely are they to continue with it or not?” If they’re checking out, they’re more likely to be patient with order and credit card processing. But if they’re elsewhere on the site, tenacity may be significantly lower.
Best practices: Methodology & Modeling
To really understand how your site performance will hold up—or not—under holiday stress, and to understand what the experience will be like for users, use an arrival rate methodology and factor in behavior models for the many, many types of users and tasks your site will serve.
Behavior modeling results in numerous permutations (often thousands) combining these variables:
- Familiarity: experienced users vs. newcomers
- Connection speed: super-fast FIOS vs. super-slow mobile device, and everything in between
- Latency tolerance: patience of users with slow site response
- Interaction speed: complexity of the page to navigate, and attention level of the user
- Tenacity: willingness of users to stick with a task through completion
“We’re not just replaying a transaction multiple times ad nauseam to try to put load on the site,” Foss says of Keynote’s LoadPro testing. “There’s a whole lot more going on behind the scenes. There’s literally hundreds or thousands of different types of users that we emulate during a single test to make sure we’re emulating what a real user population would be.”
Test in the real world—all of it.
To know how your site will perform for users dispersed across the country or the world, testing must be done over the Internet, from the same geographic locations as your users, not from behind the firewall. There’s simply no way to simulate the vagaries of Internet backbones, third-party content feeds, CDN performance, and signal transmission through the critical last mile—unless you are at the end of that mile, with a browser.
With testing agents dispersed where your users are, you get an accurate picture of variations in performance, and overcome the danger of looking at averages. An average page-load time of three or four seconds may seem OK, but that kind of average could mean your page is loading in one second for someone in New York, but taking six or more seconds for someone in Chicago. And that is not likely to be acceptable. The solution is to test from multiple, geographically dispersed locations, look at the data, and address any local or regional bottlenecks.
Staff up and stock up—on caffeine and capacity
Retailers who put off load testing can look forward to many sleepless nights in October and early November as they try to get their sites up to speed. But even the best-in-class retailers can count on a sleepless night or two—particularly Thanksgiving night—as they stay up to make sure all their hard work has paid off.
“One of my biggest clients spent the night at headquarters on Thanksgiving night last year,” Foss says. “He’s up all night making sure everything is running perfectly, and goes home Black Friday morning at 9:00 a.m.”
The holiday shopping season is the culmination of many hard hours of work for the IT/Web department. And no matter how well things are planned, no matter how rigorously everything is tested, there’s always the chance that the unexpected will happen and something will go wrong. So it makes good sense to have technical personnel on hand and on call during all the critical shopping periods to handle any emergencies, and to have extra computing capacity standing by just in case it’s needed.
Don’t wait another minute
If you haven’t started load testing your site yet, there’s no time to lose. “The most successful retailers look at holiday readiness as an ongoing process,” says Crocker. “They start in January and they test and they monitor performance and they benchmark and they trend all the way through until the end of December.”
And December is just a few short months away.