Visitor Monitoring Filtering Out Web Performance Monitoring Traffic From Google Analytics | Keynote
Blog

Filtering Out Web Performance Monitoring Traffic From Google Analytics

By Product Management | August 24, 2012

CATEGORIES: Web Performance

EDITOR'S NOTE: This solution no longer works with Google Analytics. Please see our latest post on Filtering Out Keynote Web Performance Monitoring Traffic From Google Analytics

If you're a webmaster, site owner, or digital business manager, picture this scenario. You open your Google Analytics dashboard and you see a big spike in traffic. At first, you're really excited - wow, look at all those site visitors! You run off to tell your boss that she can give you that bonus she promised if you got that online marketing campaign to work. And then... you learn that someone in the Site Ops team added web performance monitors, and so it's all so-called robots - synthetic traffic generated for the purpose of keeping tabs on your site's response time and availability.

Spike

You think, no problem, I could filter that out from Google Analytics, and then you learn you can't. It's always going to be there, like, forever. You go crazy, emailing the GA team, posting on forums, and then slowly resign yourself to forever dealing with that spike. It will disappear from your default view, maybe after a month, but that's a long month to wait. And heaven forbid if your execs ask you for a site traffic report for the past year, try explaining why you can't filter out that spike - what, you weren't thinking ahead? What kind of guy did we hire to run our online business, anyway?

Don't let this happen to your career. Plan ahead and learn how to use Google Analytics to filter out all web performance monitors from your site analytics reports. Here's the recipe:

STEP 1: Find Out the User Agent String for your Web Performance Monitors

Keynote's monitors, like all other web performance monitors, insert a special marker in the browser, called an user agent string. So all you have to know are the user agent strings that Keynote adds to the browser. Keynote has several performance monitors - using real IE and Firefox browsers, or mobile browsers. Each performance monitor comes with its own special marker, so you have to construct a regular expression to filter all of these out.

Here are the browser markers that you have to use to filter out web performance monitors:

 Keynote Systems - Use "KHTE" (for Application Perspective monitors), "KTXN" (for the real browser Transaction Perspective monitors), and "Keynote" for Test Perspective load testing agents. Compuware Gomez - "GomezAgent" Smartbear AlertSite - "AlertSite" Pingdom - "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" Yottaa - "YottaaMonitor" 

If you are using a web performance monitor not listed above, Google "<insert your monitoring vendor> user agent string" and you will definitely find the user agent string you need to know.

STEP 2: Change Google Analytics Tag to Ignore Web Performance Monitors

What you do next is to put a conditional wrapper around your Google Analytics tag to never log visits from testing companies. This requires you to edit the JavaScript code on your site. An example Google analytics tag is shown below – the actual Google code depends on your implementation. Note that the Google Analytics account ID below is an example – you should replace it with your own account ID.

 <!-- Google Analytics tracking code that eliminates Keynote robots --> <script type="text/javascript"> // These are the strings that Keynote adds to the User Agent string: var gk_KEYNOTE_TXP_MONITOR = "KTXN"; // Transaction Perspective agents var gk_KEYNOTE_APP_MONITOR = "KHTE"; // Application Perspective agents var gk_KEYNOTE_TSP_MONITOR = "Keynote"; // Test Perspective agents // if the User Agent string indicates Keynote monitors, then don't track the visit in Google Analytics if (navigator.userAgent.indexOf(gk_KEYNOTE_TXP_MONITOR, 0) == -1 && navigator.userAgent.indexOf(gk_KEYNOTE_APP_MONITOR, 0) == -1) && navigator.userAgent.indexOf(gk_KEYNOTE_TSP_MONITOR, 0) == -1) { // Ok, this is not a Keynote monitoring agent, so go ahead and log the visit with Google: var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-12345678-9']); _gaq.push(['_setCustomVar',1, 'User Type','Public',2]); _gaq.push(['_trackPageview']); (function () { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); } </script> <!-- Source code formatted using http://codeformatter.blogspot.com --> 

That's it. At this point, Google Analytics will not log your robot traffic (at least for Keynote web monitoring traffic, using the above example) and you can safely keep your job. The code snippet can be easily extended to filter out Gomez, AlertSite, Yottaa, as well.

Next, we will create an Advanced Segment in Google Analytics to filter out any synthetic traffic that has gotten through the cracks.

STEP 3: Create An Advanced Segment to Filter Out Additional Web Performance Monitors

Some monitoring tools don't modify the User Agent string, and instead use other techniques. Ensuring that you are in the MySite tab, click on Advanced Segments.

Segments

In the bottom right area of the Advanced Segments dialog box, click on the button and name it "Real People". We will now create a segment that filters out the synthetic traffic from web performance monitors. Here's the Advanced Segments dialog box:

Step1

Now, here's the tricky part - writing the regular expression that the Advanced Segment requires. Here is the regexp that filters out both the Keynote and Gomez monitors. Be careful to use the string exactly as shown, with the periods and asterisks: .*(KHTE|KTXN|GomezAgent).*

It's critical that you use a regular expression correctly, and getting it wrong is why I suspect many people believe that Google Analytics can't filter out this traffic. Once an advanced segment is created, then all traffic AFTER today will be filtered in the reports, but this will not apply to traffic that was generated prior to your creating this advanced segment. That's what I believe, from trial and error, though Google Analytics help says that you can filter out historical traffic. In any case, it's important you setup these monitors anyway, because you have no control over someone else setting up web performance monitors - even if your company didn't create performance monitors, your competitors could be monitoring your site's performance and creating all this traffic to your site - it is the world wide web, after all.

STEP 4: Select the Real People Advanced Segment When Viewing Data

STEP 4: Select the Real People Advanced Segment When Viewing Data

Drop down the Advanced Segments dialog box, and select "All Visits" on the left hand side, and the advanced segment that you created, "Real People" on the right hand side.

Step5

Click on Apply, and view your data:

Step4

Now, it would be nice if I could choose "Real People" as the default segment to apply to all my reports, but I can't do that in Google Analytics yet. Nevertheless, you now have a handy way to view all the traffic and exclude web performance monitors, including Keynote.

Back to Top