Visitor Monitoring How to Block Web Monitoring Agents (aka Bots) from Your Web Analytics | Keynote
Blog

How to Block Web Monitoring Agents (aka Bots) from Your Web Analytics

By Michael Kopp | June 10, 2015

CATEGORIES: Web Performance

Bots 

This is my second post on how to block web monitoring agent traffic from infiltrating your web analytics data. My last post about blocking IP addresses of Keynote agents is still valid, but I’ve recently come upon a much better way to ensure you block web monitoring traffic from the major web monitoring vendors in one method.

If you are using Google Analytics, Google keeps an ever increasing list of known spiders, bots, and agents. To have these removed from your analytics, simply tick the box on the View Settings page.

Bot Filtering 

However, this is a bit of a black box. To be more certain that you are blocking agent traffic, or if you are using another solution, like Ominture or WebTrends, follow these instructions.

The solution involves using a tag management system. On www.keynote.com, we use Google Tag Manager, mostly because it’s free. While these instructions are specific to Google Tag Manager, I am sure you can do this with other tag management systems, like Tealium.

Step 0: Use Google Tag Manager

Step 1: Create a custom JavaScript macro

Agent Bot - Step 1

This macro will fire to detect the user agent string and then return a value. True=Agent/Bot; False=Not an Agent/Bot. You can name it anything you want, we called ours Agentbot.

Here’s the code:

function() {
var useragent = navigator.userAgent.toLowerCase();
var regex = /(gomezagent|apm synthetic agent|ruxitsynthetic|ktxn|khte|ktht|dynatrace)/g;
if (useragent.match(regex)) {
return true;
} else {
return false;
}
}

Step 2: Create a rule using the macro to return a value

Next, we want to grab the value returned by the macro and use it to create a firing rule.

The macro you created should appear in the drop down menu. In this case we want to know when a visitor is an Agent/Bot, so we tell the rule to be considered matched when the macro returns a true value.

Give your rule a name. We called ours, surprisingly, Agentbot.

Agent Bot - Step 2

Step 3: Use the rule to exclude the traffic from firing your analytics tag

Agent Bot - Step 3

Now, all you need to do is add the rule as a Blocking Rule to any tag you want to NOT fire when an agent hits your site. Keep in mind that to truly measure your end user experience, you want to have as many tags fire for agents as would for an actual site visitor. As always, it’s a tradeoff – do you want clean analytics or a detailed understanding of your customer experience?

One workaround would be to create another analytics tag that would ONLY fire for bot traffic. To do this you would follow the same directions with the new analytics tag, but use the rule as a firing rule, not a blocking rule.

Step 4: Publish your tag manager container

Done! It’s just that easy. Now most of the agent traffic will be gone from your analytics.

Tag managers are an extremely powerful tool, but they are a little more complicated than simply deploying analytics tags on your website.

If you have any thoughts or improvements, please comment away!

Back to Top