Posted by Carlosesal
There is no doubt that Google Analytics is one of the most important tools you could use to understand your users' behavior and measure the performance of your site. There's a reason it's used by millions across the world.
But despite being such an essential part of the decision-making process for many businesses and blogs, I often find sites (of all sizes) that do little or no data filtering after installing the tracking code, which is a huge mistake.
Think of a Google Analytics property without filtered data as one of those styrofoam cakes with edible parts. It may seem genuine from the top, and it may even feel right when you cut a slice, but as you go deeper and deeper you find that much of it is artificial.
If you're one of those that haven’t properly configured their Google Analytics and you only pay attention to the summary reports, you probably won't notice that there's all sorts of bogus information mixed in with your real user data.
And as a consequence, you won't realize that your efforts are being wasted on analyzing data that doesn't represent the actual performance of your site.
To make sure you're getting only the real ingredients and prevent you from eating that slice of styrofoam, I'll show you how to use the tools that GA provides to eliminate all the artificial excess that inflates your reports and corrupts your data.
Common Google Analytics threats
As most of the people I've worked with know, I’ve always been obsessed with the accuracy of data, mainly because as a marketer/analyst there's nothing worse than realizing that you’ve made a wrong decision because your data wasn’t accurate. That’s why I’m continually exploring new ways of improving it.
As a result of that research, I wrote my first Moz post about the importance of filtering in Analytics, specifically about ghost spam, which was a significant problem at that time and still is (although to a lesser extent).
While the methods described there are still quite useful, I’ve since been researching solutions for other types of Google Analytics spam and a few other threats that might not be as annoying, but that are equally or even more harmful to your Analytics.
Let’s review, one by one.
Ghosts, crawlers, and other types of spam
The GA team has done a pretty good job handling ghost spam. The amount of it has been dramatically reduced over the last year, compared to the outbreak in 2015/2017.
However, the millions of current users and the thousands of new, unaware users that join every day, plus the majority's curiosity to discover why someone is linking to their site, make Google Analytics too attractive a target for the spammers to just leave it alone.
The same logic can be applied to any widely used tool: no matter what security measures it has, there will always be people trying to abuse its reach for their own interest. Thus, it's wise to add an extra security layer.
Take, for example, the most popular CMS: Wordpress. Despite having some built-in security measures, if you don't take additional steps to protect it (like setting a strong username and password or installing a security plugin), you run the risk of being hacked.
The same happens to Google Analytics, but instead of plugins, you use filters to protect it.
In which reports can you look for spam?
Spam traffic will usually show as a Referral, but it can appear in any part of your reports, even in unsuspecting places like a language or page title.
Sometimes spammers will try to fool by using misleading URLs that are very similar to known websites, or they may try to get your attention by using unusual characters and emojis in the source name.
Independently of the type of spam, there are 3 things you always should do when you think you found one in your reports:
- Never visit the suspicious URL. Most of the time they'll try to sell you something or promote their service, but some spammers might have some malicious scripts on their site.
- This goes without saying, but never install scripts from unknown sites; if for some reason you did, remove it immediately and scan your site for malware.
- Filter out the spam in your Google Analytics to keep your data clean (more on that below).
If you're not sure whether an entry on your report is real, try searching for the URL in quotes (“example.com”). Your browser won’t open the site, but instead will show you the search results; if it is spam, you'll usually see posts or forums complaining about it.
If you still can’t find information about that particular entry, give me a shout — I might have some knowledge for you.
Bot traffic
A bot is a piece of software that runs automated scripts over the Internet for different purposes.
There are all kinds of bots. Some have good intentions, like the bots used to check copyrighted content or the ones that index your site for search engines, and others not so much, like the ones scraping your content to clone it.
2016 bot traffic report. Source: Incapsula
In either case, this type of traffic is not useful for your reporting and might be even more damaging than spam both because of the amount and because it's harder to identify (and therefore to filter it out).
It's worth mentioning that bots can be blocked from your server to stop them from accessing your site completely, but this usually involves editing sensible files that require high technical knowledge, and as I said before, there are good bots too.
So, unless you're receiving a direct attack that's skewing your resources, I recommend you just filter them in Google Analytics.
In which reports can you look for bot traffic?
Bots will usually show as Direct traffic in Google Analytics, so you'll need to look for patterns in other dimensions to be able to filter it out. For example, large companies that use bots to navigate the Internet will usually have a unique service provider.
I’ll go into more detail on this below.
Internal traffic
Most users get worried and anxious about spam, which is normal — nobody likes weird URLs showing up in their reports. However, spam isn't the biggest threat to your Google Analytics.
You are!
The traffic generated by people (and bots) working on the site is often overlooked despite the huge negative impact it has. The main reason it's so damaging is that in contrast to spam, internal traffic is difficult to identify once it hits your Analytics, and it can easily get mixed in with your real user data.
There are different types of internal traffic and different ways of dealing with it.
Direct internal traffic
Testers, developers, marketing team, support, outsourcing... the list goes on. Any member of the team that visits the company website or blog for any purpose could be contributing.
In which reports can you look for direct internal traffic?
Unless your company uses a private ISP domain, this traffic is tough to identify once it hits you, and will usually show as Direct in Google Analytics.
Third-party sites/tools
This type of internal traffic includes traffic generated directly by you or your team when using tools to work on the site; for example, management tools like Trello or Asana,
It also considers traffic coming from bots doing automatic work for you; for example, services used to monitor the performance of your site, like Pingdom or GTmetrix.
Some types of tools you should consider:
- Project management
- Social media management
- Performance/uptime monitoring services
- SEO tools
In which reports can you look for internal third-party tools traffic?
This traffic will usually show as Referral in Google Analytics.
Development/staging environments
Some websites use a test environment to make changes before applying them to the main site. Normally, these staging environments have the same tracking code as the production site, so if you don’t filter it out, all the testing will be recorded in Google Analytics.
In which reports can you look for development/staging environments?
This traffic will usually show as Direct in Google Analytics, but you can find it under its own hostname (more on this later).
Web archive sites and cache services
Archive sites like the Wayback Machine offer historical views of websites. The reason you can see those visits on your Analytics — even if they are not hosted on your site — is that the tracking code was installed on your site when the Wayback Machine bot copied your content to its archive.
One thing is for certain: when someone goes to check how your site looked in 2015, they don't have any intention of buying anything from your site — they're simply doing it out of curiosity, so this traffic is not useful.
In which reports can you look for traffic from web archive sites and cache services?
You can also identify this traffic on the hostname report.
A basic understanding of filters
The solutions described below use Google Analytics filters, so to avoid problems and confusion, you'll need some basic understanding of how they work and check some prerequisites.
Things to consider before using filters:
1. Create an unfiltered view.
Before you do anything, it's highly recommendable to make an unfiltered view; it will help you track the efficacy of your filters. Plus, it works as a backup in case something goes wrong.
2. Make sure you have the correct permissions.
You will need edit permissions at the account level to create filters; edit permissions at view or property level won’t work.
3. Filters don’t work retroactively.
In GA, aggregated historical data can’t be deleted, at least not permanently. That's why the sooner you apply the filters to your data, the better.
4. The changes made by filters are permanent!
If your filter is not correctly configured because you didn’t enter the correct expression (missing relevant entries, a typo, an extra space, etc.), you run the risk of losing valuable data FOREVER; there is no way of recovering filtered data.
But don’t worry — if you follow the recommendations below, you shouldn’t have a problem.
5. Wait for it.
Most of the time you can see the effect of the filter within minutes or even seconds after applying it; however, officially it can take up to twenty-four hours, so be patient.
Types of filters
There are two main types of filters: predefined and custom.
Predefined filters are very limited, so I rarely use them. I prefer to use the custom ones because they allow regular expressions, which makes them a lot more flexible.
Within the custom filters, there are five types: exclude, include, lowercase/uppercase, search and replace, and advanced.
Here we will use the first two: exclude and include. We'll save the rest for another occasion.
Essentials of regular expressions
If you already know how to work with regular expressions, you can jump to the next section.
REGEX (short for regular expressions) are text strings prepared to match patterns with the use of some special characters. These characters help match multiple entries in a single filter.
Don’t worry if you don’t know anything about them. We will use only the basics, and for some filters, you will just have to COPY-PASTE the expressions I pre-built.
REGEX special characters
There are many special characters in REGEX, but for basic GA expressions we can focus on three:
- ^ The caret: used to indicate the beginning of a pattern,
- $ The dollar sign: used to indicate the end of a pattern,
- | The pipe or bar: means "OR," and it is used to indicate that you are starting a new pattern.
When using the pipe character, you should never ever:
- Put it at the beginning of the expression,
- Put it at the end of the expression,
- Put 2 or more together.
Any of those will mess up your filter and probably your Analytics.
A simple example of REGEX usage
Let's say I go to a restaurant that has an automatic machine that makes fruit salad, and to choose the fruit, you should use regular xxpressions.
This super machine has the following fruits to choose from: strawberry, orange, blueberry, apple, pineapple, and watermelon.
To make a salad with my favorite fruits (strawberry, blueberry, apple, and watermelon), I have to create a REGEX that matches all of them. Easy! Since the pipe character “|” means OR I could do this:
- REGEX 1: strawberry|blueberry|apple|watermelon
The problem with that expression is that REGEX also considers partial matches, and since pineapple also contains “apple,” it would be selected as well... and I don’t like pineapple!
To avoid that, I can use the other two special characters I mentioned before to make an exact match for apple. The caret “^” (begins here) and the dollar sign “$” (ends here). It will look like this:
- REGEX 2: strawberry|blueberry|^apple$|watermelon
The expression will select precisely the fruits I want.
But let’s say for demonstration's sake that the fewer characters you use, the cheaper the salad will be. To optimize the expression, I can use the ability for partial matches in REGEX.
Since strawberry and blueberry both contain "berry," and no other fruit in the list does, I can rewrite my expression like this:
- Optimized REGEX: berry|^apple$|watermelon
That’s it — now I can get my fruit salad with the right ingredients, and at a lower price.
3 ways of testing your filter expression
As I mentioned before, filter changes are permanent, so you have to make sure your filters and REGEX are correct. There are 3 ways of testing them:
- Right from the filter window; just click on “Verify this filter,” quick and easy. However, it's not the most accurate since it only takes a small sample of data.
- Using an online REGEX tester; very accurate and colorful, you can also learn a lot from these, since they show you exactly the matching parts and give you a brief explanation of why.
- Using an in-table temporary filter in GA; you can test your filter against all your historical data. This is the most precise way of making sure you don’t miss anything.
If you're doing a simple filter or you have plenty of experience, you can use the built-in filter verification. However, if you want to be 100% sure that your REGEX is ok, I recommend you build the expression on the online tester and then recheck it using an in-table filter.
Quick REGEX challenge
Here's a small exercise to get you started. Go to this premade example with the optimized expression from the fruit salad case and test the first 2 REGEX I made. You'll see live how the expressions impact the list.
Now make your own expression to pay as little as possible for the salad.
Remember:
- We only want strawberry, blueberry, apple, and watermelon;
- The fewer characters you use, the less you pay;
- You can do small partial matches, as long as they don’t include the forbidden fruits.
Tip: You can do it with as few as 6 characters.
Now that you know the basics of REGEX, we can continue with the filters below. But I encourage you to put “learn more about REGEX” on your to-do list — they can be incredibly useful not only for GA, but for many tools that allow them.
How to create filters to stop spam, bots, and internal traffic in Google Analytics
Back to our main event: the filters!
Where to start: To avoid being repetitive when describing the filters below, here are the standard steps you need to follow to create them:
- Go to the admin section in your Google Analytics (the gear icon at the bottom left corner),
- Under the View column (master view), click the button “Filters” (don’t click on “All filters“ in the Account column):
https://moz.com/blog/how-to-filter-junk-traffic-google-analytics
No comments:
Post a Comment