Visitor Monitoring The Benefits of Third-Party Content Monitoring | Keynote
White Paper

The Benefits of Third-Party Content Monitoring

1. Executive Summary

Third-party applications are a vital part of almost any successful Web site. The type of content that resides on Web sites today can be as varied as rich media applications, bi-directional social media feeds, images or video delivered by CDNs, and, of course, ads delivered by ad networks. This content is important in driving business and enhancing the user experience (UX) to any online site. However, these enhanced features can also come with risk that can be in direct relationship to the amount of added third-party content.

Many sites attempt to ensure a rich UX with third-party Rich Internet Applications (RIAs), such as Twitter, Facebook, and TypePad, to name just three. These RIAs have become indispensable components to the user experience and to the bottom line of the site owner.

With constant enhancements in available content and the desire of users for more interactivity in their data and social media, the amount of third-party content will only increase. Each widget, plug-in, and script is an opportunity to slow down and negatively impact Web site performance, and so can negatively impact the user experience. A slick-looking Web site is ultimately of limited effectiveness if all of the its bells and whistles are an impediment to system performance. Hence, the monitoring of system performance at the end-user/UI level is extremely important to ensuring a consistently excellent user experience. By using the proper performance tools in a targeted manner, both business managers and developers can effectively monitor the impact of third-party performance both on the individual component level and in the aggregate.

Third-party widgets that can compromise system performance can be broken down into several discrete types. For the purposes of this discussion, these groups include but are not limited to blogs such as TypePad and WordPress, microblogging sites, and external advertisements. Each of these groups can impact system performance to some extent. The issue comes down to two questions: Which components impact the performance negatively, and why?

This paper will examine the following:

  • The impact that third-party content can have on Web site performance.
  • Distinguishing the variance in performance based on the type of widget.
  • How effective performance monitoring can establish accountability for the internal organization and the third-party vendor.
  • How performance tools such as Keynote’s Virtual Pages can be used to monitor specific third-party applications and possible bottlenecks.

2. The Benefits of Third-Party Content Monitoring

The benefits of consistent and focused performance monitoring are fairly well known to savvy IT organizations. Poor performance translates to a poor UX and a resulting loss of “eyeballs” and revenue over both the short and long term. However, drilling down to a more focused level can yield even more rewards that are not readily apparent:

  • Measuring third-party content can accurately determine whose domain is responsible for poor performance.
  • More detailed analysis can allow for management of content that is under your ownership.
  • Easier tracking of ongoing performance of all content, be it internal or third party.
  • Targeted alarms in case of a performance drop-off can lead to faster resolution of potential issues.
  • Targeted testing and the associated alarms can help improve development practices, since some performance issues are due to flawed modifications of previously working code.
  • Targeted performance monitoring of third-party content can establish proof of accountability for third-party providers. This can also establish liability for performance issues that slow down or crash the Web site.

Sean Power of Watching Websites has showed the exceptionally costly consequences of downtime due to performance loss.

downtime costDepending on various factors, a loss of $50,000 per hour of downtime is not unrealistic. In the case of a premiere e-commerce Web site such as Amazon.com, the offline cost could well be $1 million an hour.

Even if your Web site isn’t likely to crash, excessive latency will take a severe, if more subtle, toll. Let’s look at another analysis by Sean Power that relates to conversion rate, which is defined as the number of completed “actions” (such as subscription sales, opt-ins, or any other action that can produce a measureable outcome), divided by the total number of visitors to the site.

 

 

conversionrate over time

As readers can testify from their own experience, slow performance will cause any user to leave a site. Power’s graph shows the impact of latency on the conversion rate of a Web site.

As you can see, there is a clearly inversely proportional relationship between latency and conversion rate. When latency (the dotted line) increases, the conversion rate decreases by as much as 50 percent. Further research shows that there are two waves of abandonment that can occur: the first wave of users who leave in frustration, and then a secondary wave as they tell others about the poor user experience and drive them away from the site.

 

 

Here is the home page of the CNN Web site (www.cnn.com). When a user loads this page, they are loading an array of components:

cnn

The page features at least a couple of external widgets; the dynamic ad in the frame on the upper right features a rotating series of advertisements that updates at least once a minute. Just below that is a Facebook widget that shows CNN stories that your Facebook friends liked and/or shared. This content is sent from an outside entity to appear on the CNN site.

2.1. Types of Third-Party Content

Many Web sites use a wide array of widgets to enhance the user experience and drive business. Many, if not most, of these widgets are third party in nature, meaning that they are external to the Web site being viewed. The increasing presence of social media applications is becoming a central part of this equation. When we look at the various types of third-party content, especially rapidly growing Twitter and Facebook traffic, we find that these applications are used to enhance dissemination of information and leverage increased traffic to the Web site. For example, a user can be on the CNN Web site, see an article or video they fancy, then link that content to Facebook using the Facebook widget. After the content link appears on Facebook, a Facebook user may “like” the article or select the page to view/read the content. In this manner, the third-party widget drives traffic to two different sites.

We can divide third-party content into the following broad groups:

  1. Microblogs – This would include feeds and links to Twitter, Digg, RSS, and similar sites.
  2. Full Blogs – TypePad and WordPress blogs can be linked to other sites via widgets that can be considered third party on other sites when they are loaded.
  3. Social Media – Facebook and Facebook Connect. This category also includes polls that are generated by Facebook applications.
  4. Content Delivery Network (CDN) – Akamai is one example of a CDN, which can improve performance by reducing data hops for the delivery of content to an end user.
  5. Advertising – Ads by companies such as DoubleClick, AdSense, and Tribal Fusion. These can be banner ads, and such ads can be static or dynamic.
  6. Miscellaneous Widgets – A generic category based on widgets from third- party vendors that fit none of the other categories.

Consider the bottom portion of the CNN homepage site below:

At the bottom of the page are three more dynamic ads. Just above that is “CNN TV,” whose widgets link the user to exclusive video content. As you can see, many widgets are being loaded every time this page is accessed. Whether your site is as complex as CNN’s, more complex, or less complex, the challenge is to figure out which components are dragging down overall performance of the Web site.

3. Google Case Study – Re-engineering AdSense

knockout lab

In June 2010, Google’s Michael Kleber and Arvind Jain offered a presentation that examined the impact that third-party additions have on Web site performance. To illustrate their point, they devised what they called a Knockout Lab, where they looked at the Top 100 Web pages (http://www.web100.com) and focused on the impact that certain add-ons had on the Web page. In short, how much do third-party applications slow down the loading of a Web site?

Here is what certain add-ons do to major Web sites.

In each case above, there is a significant impact on page performance due to just ONE thirdparty application. Note that the very popular Digg widget can impact page launch performance by as much as 14 percent, meaning that a Web page without any Digg widgets can load 14 percent faster than a page with one. The impact of some other widgets is even more profound; for example, FriendConnect can impact a site’s performance by as much as 30 percent!

Needless to say, not loading key third-party content is hardly a viable option for most Web sites. So the challenge is not to remove the thirdparty content but to make it more efficient for everyone, with the ideal of course being to make a site load as just as quickly with third-party content as without it.

Kleber and Jain considered two options. Option one is optimization of the main Web site. However, given the wide variety of Web sites, it is hardly realistic to expect thousands of Web sites to be modified to optimize third-party components.

the problem

Hence Option two, optimization of the thirdparty widget itself so that it loads faster under all conditions. For example, Google AdSense (GAS) would, prior to optimization, block a publisher page by running a script. The Web page would not finish loading while the GAS script was being run (see below).

A look at the screenshot at the top of this page shows the original GAS script. The first part of the script sets the size of the ad. The segment in red is the script tag that parses and executes the script. The publisher page will not finish loading until this script completes. The loading or parsing of the script will cause the browser to stop. Google came up with a method to improve the loading time of AdSense (depicted in the middle screenshot). The solution involves recalibrating the script so that there is no need to make modifications to the Web site itself.

The impact of these changes is significant. The loading time of a Web page with AdSense is improved by a factor of 4, leading to real improvement in performance.

In this paper, we won’t go into specific solutions for third-party content. Our focus is on how we can monitor site performance and target the third-party content that could slow down the overall performance of the Web site. With this in mind, let’s look at some effective methods for targeting our testing to find the slower performing widgets with the intention of optimizing them later.

aswift

Hence Option two, optimization of the thirdparty widget itself so that it loads faster under all conditions. For example, Google AdSense (GAS) would, prior to optimization, block a publisher page by running a script. The Web page would not finish loading while the GAS script was being run (see below).

A look at the screenshot at the top of this page shows the original GAS script. The first part of the script sets the size of the ad. The segment in red is the script tag that parses and executes the script. The publisher page will not finish loading until this script completes. The loading or parsing of the script will cause the browser to stop. Google came up with a method to improve the loading time of AdSense (depicted in the middle screenshot). The solution involves recalibrating the script so that there is no need to make modifications to the Web site itself.

 

 

results aswiftThe impact of these changes is significant. The loading time of a Web page with AdSense is improved by a factor of 4, leading to real improvement in performance.

In this paper, we won’t go into specific solutions for third-party content. Our focus is on how we can monitor site performance and target the third-party content that could slow down the overall performance of the Web site. With this in mind, let’s look at some effective methods for targeting our testing to find the slower performing widgets with the intention of optimizing them later.

 

 

 

 

4. Third-Party Monitoring in Action – Three Lessons

Eric Goldsmith, an accomplished operations architect, has used Keynote Virtual Pages and similar tools for performance monitoring. He categorizes data performance into four discrete types: 1) How internally owned and operated (O&O) components are performing; 2) How the content delivery network (CDN) is performing; 3) How the internal advertisements and infrastructure are performing; and 4) All the other external third-party components are performing. This analytical schema can determine who owns what performance issues, be it his advertising infrastructure or a third-party vendor.

 

spike

4.1. Lesson 1: Content Separation Tells a Better Story

Here is an example of the impact of Web site performance monitoring without a category breakdown. A typical performance graph might look like this:

The first graph certainly shows a performance spike (circled in blue), along with a longer, less severe degradation in performance (circled in red), which is still significant. However, knowing that these problems happened doesn’t tell us why they happened.

Now let’s take a look at the same data broken down by Goldsmith’s categories.

 

 

 

 

avg performance category
When we separate the performance by categories, the culprits are now more obvious. The most severe issue (the blue graph, circled in blue) was due to third- party content performance. To a lesser, but still significant extent, O&O (in green) content was responsible for a couple of severe spikes in performance. Meanwhile, the performance of the CDN (in pink) proved to be consistent. Clearly, this approach allows us to see what components are impacting site performance, and when.

Goldsmith used standard Keynote products to develop a baseline for performance, and subsequent performance testing involved checking the deltas from that baseline. If the deltas are too large, they trigger an alarm that is sent out to the organization.

 

4.2. Lesson 2: Third Parties Aren’t the Only Ones Who Become Accountable

This brings us to a primary benefit to targeted performance testing of third-party content: improving accountability. As Goldsmith put it, “Don’t hide behind the SLA [service-level agreement]. Develop your site to not to rely on their compliance, but be able to pinpoint which third-party component is a problem area.” For example, if you have a page that features news content with many ads, customize the site so that the content can be read even if some or all of the ads fail to load, and make sure that your organization knows what failed and why.

The graph below is a sample of the type of performance metrics that can be delivered by component type:

load time domain

Owned and operated content loads fairly consistently, as does CDN content. What may surprise some is that in this case the advertising performance, often a culprit in poor Web site performance, was consistently solid. This example brought a revelation to Goldsmith: Testing with Virtual Pages can lead to the exoneration of certain third-party content that might have been a prime performance suspect before testing.

4.3. Lesson 3: Early Warnings Save Time and Money

The third advantage that Virtual Pages can give you is a faster Mean Time to Identification (MTTI), a metric that defines the speed at which an operations group can identify a problem once reported and be able to triage it successfully. Goldsmith found that while his evidence was anecdotal, the improvement was evident to everyone in his organization. As a Level 2 support person, Level 1 issues never came to him anymore, since the Level 1 people could get the alarms from the start. As a result, diagnosis and triage took place much faster than before. He could then attack Level 2 issues directly and far more quickly without the relative “noise” of lower-level issues impacting him and his team. Overall MTTI was greatly improved over previous levels prior to third-party monitoring.

5. Keynote Virtual Pages – Case Study

In our test case, we set up a Performance Lab of our own. In this example, we turn to Keynote.com using Keynote Virtual Pages to demonstrate how Virtual Pages can highlight the different impact of third-party components. We run Virtual Pages on various parts of www.keynote.com to determine how the third-party widgets on different pages impact load times. Here is a brief description of the third-party content being tested by Virtual Pages.

  • Company Page (MicroBlog) – There are two Twitter buttons, which dynamically show the number of Twitter followers that Keynote has.
  • Contact Us (Chat) – Chat Live and Talk Live are widgets that use Bold Chat to open live online dialog windows with Web site visitors.
  • Keynote Live! (iframe/MicroBlog) – This page features Hootsuite as the third-party application that streams Twitter feeds.
  • Press Room – This page features all three of the above applications on a single page.

6. Test Case 1: The Press Room Page

We’ll look at the results in reverse. The Performance Lab for all measurements ran for three days. The next graph shows the performance with and without third-party applications on the Press Room page. It is clear that at roughly the same time each day both performance and availability suffer:

the pressroom page

The Press Room page with third-party applications takes as much as 9 times longer to load as the same page without them. Availability of the Web page drops as much as 5 percent at about the same time every day.

The graph shows a big difference in performance with the third-party components (the blue line) versus without them (the orange line). But also look at the performance curve over time. In both cases, the response time spikes during the early business hours (approximately 10 am, with the peak declining around noon). There is another surge in Europe during the early afternoon hours. From these few graphs alone, we know that third-party components have a significant impact on the Web site, and the impact is largest early in the day. Which components are having the greatest impact on performance and when? These questions lead to the next step in our research, waterfall analysis.

6.1. The Power of Waterfall Analysis

So far we know the following:

  • We know that there is a significant difference in Web site performance with third-party content as opposed to without.
  • We know when the performance spikes take place.

What we do not know is why. What components are causing the difference in performance? We can use waterfall graph analysis, an option available in Virtual Pages, to break apart the data by component types. Here is a sample of the waterfall view of the Press Room data, with and without third-party content. On the Y axis are the various scripts and files that are requested by the browser in sequential order. At the bottom on the X axis is the time that it takes for the entire page to launch. You can see from this graph where each component of the Web page launched, what transactions took place, and when it completed.

Let’s take a look at the loading of the third-party Press Room page via waterfall analysis. First, look at the performance of the Press Room page without third-party components. The loading of the Web page takes just over 2 seconds. Note that there are no delays (which would be indicated by long green or orange bars).

waterfall analysis

Now let’s take a look at the same page with third-party components. At the beginning of the page load, a couple of style sheets (css files) are being loaded. While not strictly third-party components, these files are loading slower with the other components present. The first file (common.css) takes almost 0.8 seconds to load, while the second file (company.css) takes about the same time to load, but there is a delay in the initial connection time. Looking downward to the next area, there are two Twitter-related components that add a small amount to the download time. At the bottom of the graph things get a bit more interesting. Those three components taking significantly more time are Chat and Talk Live components. The download and response delays for each component are significant.

waterfall analysis1

Take a look below and you can see how scrolling on the component in question can yield potentially important data for the support teams to analyze:

waterfall analysis2

We’ve viewed most of the page, and we would be remiss if we didn’t examine one other area in the waterfall analysis:

waterfall analysis3

Note that the entire page loads in slightly more than 4.4 seconds, which is nearly double the time as the same page without third-party components. There is also a noticeable gap in the page for at least 1 second, or nearly 25 percent of the total download time for the page. It appears that the Webtrends component is causing the delay in writing to the Web page. A team that wants to streamline this Web page’s performance should look at the Chat and Twitter widgets as well as at Webtrends.

Let’s look at one more graph, one that illustrates variance in performance based on geography.

waterfall analysis4

At the bottom left is the legend for the graph, where you can see the page-loading performance over three days by city. A quick look shows that the performance spikes are pretty consistent across geographical regions. If, for example, Dallas or New York showed slower response times and availability than San Francisco or Los Angeles, the lack of a CDN (or problems with one) could be indicated. However, this does not appear to be the case.

In conclusion, Press Room page analysis tells us the following:

  • Third-party components are causing poor performance at consistent times.
  • Certain third-party components (Twitter, Webtrends, and BoldChat) clearly contributed to a slower page loading time and reduced availability.
  • The issues seem to be fairly consistent in all areas. Adding or optimizing CDN is probably not a priority.

7. Test Case 2: The Company Page

In the Keynote Performance Lab we also looked at the Company page, which has several third-party components. Again, the performance of this page with and without third-party components was monitored over a period of time. Let’s take a look at the results:

company page

The orange line indicates the performance of the Web page without third-party components, and it shows a pleasing level of performance with very little variance over the three days. But when we look at the page with widgets, a very different story appears. Between 6 am and noon, the loading time increases by a factor of 8. Yes, it takes 8 times longer for the Company page to load with third-party widgets than it does without them. In addition, there is a 3 to 5 percent drop in availability during those periods. Why? Let’s start by looking at the time and availability graph for the Company page without third-party additions. The graph shows a clean page load with the components loading steadily and quickly.

Company Page1

The geographical analysis shows a different story. Overall times are faster in all cities, but the variance is even greater. Perhaps load balancing or the service providers need to be examined in more depth. Conclusion: This analysis tells us that optimization is possible through load balancing and finding ways to improve the chat components’ connection time.

8. Test Case 3: Keynote Live! (HootSuite)

Among the many advantages continuous performance monitoring can deliver is an understanding that performance can vary widely over time. Performance can degrade, then improve, then fall off again. Note the performance of HootSuite in isolation using Virtual Pages:

hootsuite

In this case, isolated spikes in performance occurred at 4 am, 9 am, and noon with and without HootSuite. Availability stayed consistent throughout the day until dropping off a bit after 9 pm.

hootsuite1

A breakdown by city shows that performance on the East Coast and in the Midwest appears to be slower than on the West Coast.

hootsuite2

Analysis of performance by global region shows that U.S. performance displays a consistent trend of midday performance slowdown (albeit not by much).

hootsuite3

The following graph shows a distribution breakdown with respect to download times. In this particular case, the vast majority of downloads took less than 6 seconds.

hootsuite4

In a small number of cases, the download time is much longer, over 23 seconds. We can examine the detailed listing of the datapoints to find out why those downloads were so slow.

It turns out that in most of the longer downloads there was a content error, and that the download took place in Shanghai, vital information for the operations team.

9. Test Case 4: BoldChat

The analysis of the Bold Chat widget yielded some interesting data for future optimization. Note the clear performance peak starting at 6 am and extending to roughly 2 pm. Web site load times triple when BoldChat is being used. But that is not all:

boldchat

boldchat1

The availability chart shows that the availability of the site takes three significant hits during peak times. Again, Web operations teams can use this information combined with a waterfall analysis to focus optimization efforts.

10. Conclusions

Third-party components are virtually indispensible to many Web sites today, whether because they bring in revenue through advertising, or drive business by bringing in new users from outside sources, or deliver a user experience that creates a buzz. Living without them is simply not a realistic option. Given that sites can use as many as 200 widgets from third parties, the potential impact on performance can be enormous. An e-commerce site can lose anywhere from $40,000 to $1 million per hour in downtime. Even if your site doesn’t crash, latency can impact your conversion rate by as much as 50 percent, again with potentially devastating consequences for your bottom line.

The more complex your Web site is, the more likely it is that its performance – and possibly your profit margin - is in the hands of third parties and their components running on your site. Ideally, we want to improve the performance of these third-party components (aka “widgets”), so that a page loads just as fast with these widgets as without them.

If you are the owner of the Web site, there are at least two ways to improve overall site performance. One is to optimize the Web site itself for each widget so that those widgets running on the page will be more efficient, a method that is not cost-efficient. The other, more cost-effective option is to use continuous and focused performance monitoring of your Website. Breaking down performance by time and by component category allows you to pinpoint the components that adversely impact Web site performance.

The benefits of monitoring of Web pages and third-party components are significant indeed. First, operations can target these issues quickly and efficiently, which can reduce potential downtime and loss of revenue. This metric, known as Mean Time to Identification, can be tracked. Second, business unit managers can track the performance of all content, both internal and external, which can establish SLA accountability with the third-party vendors, saving money on lost downtime or the cost of rebates. Another benefit is the accountability that can also be established internally on components and content that has been developed on your site. Third, development and QA teams can save money by tracking these issues in real time. Modifications to code on the Web site or to the widget have been known to adversely affect a previously well-performing Web site, and monitoring can nip these issues in the bud, saving time and therefore money.

Finally, the user experience can suffer due to bad performance. This can cause a loss of viewers, both because of direct experience and by word of mouth. This impact can be potentially devastating to the bottom line.

One tool that can be used to monitor third-party content performance is Keynote Virtual Pages. Using Keynote Virtual Pages, you can maintain targeted vigilance over time and bring a new level of clarity to performance monitoring. Keynote Virtual Pages can determine which components are performance-intensive and also when and where they are slowing down loading times. By isolating and grouping these third-party components (if in fact a third-party component is responsible for the performance issue), you can target components that are causing the largest drag on loading times in accordance to performance SLAs. Alarms can be set to notify the owners of various components if performance slips below SLA-determined levels. Baseline performance can be established with Keynote Virtual Pages, and it can be a very effective means of determining the quality of the user experience.

11. Recommended Reading

The following papers, presentations, and Web sites were referenced in the development of this white paper and are highly recommended for a deeper, more detailed understanding of the impact that various components have on the performance of Web sites that contain rich media and other Web 2.0 components.

Back to Top