FAQ – Transaction Performance Index for Online Retail Web Sites

For further comments or questions about the measurement methodology used for this index, please email us at public_services@keynote.com.

Topics

1. What is the Keynote Online Retail Transaction Index, and how is it measured?

The Keynote Online Retail Web Transaction Performance Index shows the total execution time and success rate for logging into an account, searching for an item, adding it to the shopping cart, and proceeding to check out but without login on the selected Internet retail sites from the ten largest U.S. metropolitan areas (Boston, Chicago, Dallas, Detroit, Houston, Los Angeles, New York, Philadelphia, San Francisco, and Washington, D.C.) on high-speed links attached to key points on the largest U.S. Internet Service Provider (ISP) backbones.

There are three mini indices in the Online Retail Web Transaction Index suite – Apparel, Books & Music, and Electronics. The sites that appear in each Index were selected based on publicly available market-share information published in The Wall Street Journal and other reliable industry sources.

The published Index is based on measurements performed every hour between the hours of 8am and midnight Eastern Time, Monday through Friday. Keynote subscribers receive 24/7 detailed measurements (DNS lookup time, TCP connect round-trip time, etc.) of each site in the index and aggregated measurements. Subscribers can also investigate performance based on city, backbone, or agent.

The Keynote Online Retail Web Transaction Index is measured by Keynote's Transaction Perspective. That service is used to measure download speed for well-connected major business users (T-1 to T-3 connection speeds) and as a way of measuring ISP peering performance. The measurement agents use standard Microsoft Internet Explorer running on Windows XP to ensure that any difficulties in downloads are not due to congestion on Keynote's connections to the Internet, Keynote's connections are always lightly loaded and low latency.

Measured performance depends on factors such as geographic location, backbone connectivity, and network infrastructure at each measurement location. The performance experienced by an individual user in one of the cities above may be better or worse than our measured average depending on how the user is connected to his ISP and how that ISP connects to the Internet.

2. How does the transaction work?

The standard Keynote Online Retail transaction begins by entering the retail´s Web home page and searching for an item, adding it to the shopping cart, and proceeding to check out without logging on to an account.

3. How are transaction measurements calculated?

Transaction success rates are the ratio of completely successful transactions to attempted transactions. Unsuccessful transactions include those in which any Web page in the transaction fails to download correctly and completely, those in which any error was detected on any page, or those that do not complete within a specified time limit. The time limit for a transaction is calculated by multiplying the number of Web pages in a transaction by 12 seconds. For example, a five-page transaction is allowed 60 seconds to run, while a seven-page transaction is allowed 84 seconds. The transaction times reported do not include any human interaction time. Since each of the indexes measure the ability to complete expected actions, actual number of pages may vary from site to site depending on site design, or check-out process.

Transaction outage hours is calculated by taking the count of any hour during which the measured path is not available more than 30% of the time, indicating a major user impact.

Transaction performance (speed) is calculated as the geometric mean of the total transaction time over the entire period; that ensures that the heavy tailed outliers do not distort the statistics. Measurements from unsuccessful transactions are not included. The median of the geometric means of the individual sites in the index is used as the index value to present the entire index performance.

4. When is the Index updated?

Results are posted each Tuesday and represent the performance of each site during the previous week. The rankings for the sites in success rate and response times are published for the current week as well as noting previous rankings.

5. What happens when a software agent does not report data?

From time to time Web site changes and other factors may cause a site to be temporarily unable to be measured accurately. In this case the site may be temporarily removed from the published weekly reports. This typically indicates that the Keynote Transaction Perspective agents were unable to interact with the Web site for an extended period of time. This is not always due to troubles at the Web site and does not necessarily indicate real performance problems.

6. What changes have occurred to the Index since its inception?

A similar index E-Commerce Index was first launched in March, 2003. It was re-branded and re-launched as Online Retail Transaction Index in September, 2006. It consists of three mini indices – Apparel, Books & Music, and Electronics. For a complete list of the recent changes in the index please click here.

Related Links