When talking with clients about how WP Engine counts visitors, we often get asked questions like this: “who are these ‘bots’ you speak of?” and “why do their visits not show up in other analytics programs, like Google Analytics?”
The most common reason this subject comes up is that a client’s Google Analytics will show less traffic than their user dashboard does. This happens because Google Analytics does not count non-human traffic (bots) as unique visits to a website, but WP Engine shows all traffic human and non.
It’s good to know what these bots, or “crawlers,” are, since they can represent 50% or more of the actual traffic to your website—traffic normally completely hidden by Google Analytics and other JavaScript-based web analytics. To understand why, we will go over a few things: what bots / crawlers are, what they do, and why we want them. Lastly, we will explain why they don’t show up on your (other) analytics, and why WP Engine chooses choose to show them.
Growing traffic is hard.
I know. Over the course of my time helping companies in the online marketing space, there has always been one constant goal—to achieve more traffic. Have any of you site owners, marketers, or technical folks ever come across anyone who says, “I’d like less traffic please”? My guess is no.
One of my friends checks his traffic like a compulsion. I’ll be sitting with him at lunch as he pulls out his phone, just to tell me about his 3% increase this week. So I started playing the table phone stack game, which he of course loses constantly. Hey, nothing beats a free lunch!
But where does all this traffic come from?
Did you know that you have a private army that helps you grow your traffic every day? They go by many names online: Robots, Crawlers, Spiders… (Wait, is this starting to sound like a horror movie?) But these non-human helpers are not scary. In fact, they are what drive the matrix of the web.
Crawlers can come in all different forms and styles. But lets start with the most basic and well known crawler, Googlebot. Googlebot is the crawler that Google uses to create and index for their search engine.
From Wikipedia:
“Googlebot discovers pages by harvesting all of the links on every page it finds. It then follows these links to other web pages. New web pages must be linked to from other known pages on the web in order to be crawled and indexed or manually submitted by the webmaster.
A problem which webmasters have often noted with the Googlebot is that it takes up an enormous amount of bandwidth. This can cause websites to exceed their bandwidth limit and be taken down temporarily. This is especially troublesome for mirror sites which host many gigabytes of data. Google provides “Webmaster Tools” that allow website owners to throttle the crawl rate.”
Now the first thing to note here is that Googlebot is very smart and can even run some of the code of your website. The other takeaway is that it can be a hungry, data-eating beast when it comes to a site’s bandwidth. You can, however, change the settings on your webmasters tool to lower the amount of bandwidth Google eats, but that’s just a side note.
The real question is this: would you ever tell the Googlebot to go away (on purpose) just because it eats bandwidth? No, because of all the benefits a Googlebot brings to your site. Most site owners would say, “No, Googlebot, please stay. In fact let me buy you a steak dinner if you’ll increase my ranking! Here, why not have some champagne while I convince you how awesome my content is…” Also, if you turn off Googlebot, your site would be much less likely to pop up in search, and why would anyone want that?
Now that’s just one type of crawler we’re talking about. And like I said, you have an army of them working for you. One of the robot databases shows 302 active well-known crawlers searching the web right now! Imagine how much data all those crawlers eat on a daily basis. They’re hungry little monsters that feed off the data and turn it into traffic. Some of them can even pull gigabytes a month if your site is big enough. Imagine the traffic that’s generating!
This army of bots, or crawlers, searching through your site does three very important things. It indexes your site, sees your site’s progression over time, and, last but not least, helps the world find your site.
So why don’t they show up in Google Analytics?
Google Analytics is a well written and smart tool that knows the difference between a human and a crawler (most of the time). So when you’re obsessively checking how many people went to your site every day, it does not list the hits from crawlers in its numbers.
You open Google Analytics and see that 4000 unique people viewed your website today. You’re excited because yesterday only 3500 people viewed it. You can rest assured that 500 more people in fact visited your site because as a marketing tool, Google Analytics is only supposed to show you real human beings that are capable of reading or buying things. This is helpful information to have, but it serves a different purpose than the information in your WP Engine analytics, which includes your visits from traffic-boosting crawlers.
At WP Engine, our analytics are a more accurate portrayal of who (or what) is visiting your site. Just because crawlers aren’t capable of buying anything, doesn’t mean they aren’t beneficial. They take up space and do what they do, just like people, when visiting a site. It only makes sense that they pay an entrance fee just like everyone else.
Now the good news is that crawlers boost traffic. Just because they are not laughing at your jokes or ordering your merchandise does not mean they aren’t hard at work. They are working relentlessly, indexing and chomping up data…all to be turned into increased traffic, which is what every site owner wants, right?
What does this all really mean?
Crawlers eat data, and in turn, supply your site with traffic. They don’t show up on Google Analytics because they aren’t real humans. But the point is, real humans won’t be able to find your website without the crawlers visiting and indexing you. It is a symbiotic relationship in which humans and machines feed off each other in a wonderful information cycle.
So welcome these crawlers into to your site with open arms. Feed them well, feed them quickly, and then watch your traffic and business grow.
The visitor we all want – the Googlebot.
Never thought about it not showing up in Google Analytics.
“At WP Engine, our analytics are a more accurate portrayal of who (or what) is visiting your site.”
Interesting!
A good over view. From reading it on our end we don’t really need a continuous crawl then. And while googlebot is essential a lot of the bots we don’t need at all. Like the pharmacy spam bots. Our business need could get by with just googlebot,bing, and yahoo.
There are bad bots and other automated undesirable traffic, too.
When I see a dozen concurrent visitors from the same IP subnet belonging to an overseas country for which I receive negligible legitimate traffic per Google Analytics, it’s either spambots or threats. Blocking the bad traffic with CloudFlare reduced my daily WP Engine visitors by ~40% with no discernible impact on Google Analytics visitors or revenue.
Thanks for the post Thomas…we noticed that when we blocked IP’s from a couple of addresses in Russia and China we lost ranking but seemed safer. Not sure if they were human or bots, but seemed malicious. I think even with bots monitoring helps
This forced me to change my plan from personal to professional. But I still can’t understand…how my real traffic overaged GA statistic in 5-7 times…WPE Support told me that now, with CDN I’ll see better results and less traffic, but I still can’t understand how it’ll works with CDN?!?!
Hi Alex,
I’m not familiar with your specific site and its traffic patterns that our support has advised you about. However the abstract point here is that with CDN most, if not all, of your hits to images will be routed to the CDN and not hit our WP Engine servers where we count that as a visit. This makes sense for sites which might get a lot of hot linking directly to images – which we would count as a visit.
If the CDN is not working as a remedy, or you still have questions please open a ticket or a live chat with our support team, and we’ll be very happy to assist further.
Thanks so much,
Austin
Hi Alex,
Thanks for the comment and I apologize for any frustration here. I’d love to chat with you directly about this to make sure we can get your hosting all set up the way you need it to. Without digging into your site, I can’t comment on what your traffic would be. I’d love to chat with you via email about this. If you could email me, [email protected], I can make sure we answer all your questions.
Thanks!
-Austin
I empathize with Alex. I’m getting 1200 unique visitors per day in real life, but WPEngine calculates every script and css file load on a page as a visit too. That gave me 143k additional visits, just when people were loading the css files that supported a page!
A professional plan isn’t even large enough, I’d have to go to a $250 per month business plan, and even then, I couldn’t ‘grow’ my site as I simply couldn’t afford getting charged for all of those extra ‘visits’ running at 9-20x the real visits.
When I read
At WP Engine, our analytics are a more accurate portrayal of who (or what) is visiting your site.
I think, that’s all fine and dandy, but I (the paying customer) can’t see or review any of those analytics!
I can only see the top 10 from the Overage report, if I go over.
How are those better analytics supposed to help me if I can drill into them?
Similarly, those analytics in the Overage report do not reconcile with each other!
In essence, those better analytics are all just a black box of data I don’t have access too, but result in me getting charged 6x my monthly hosting bill. 🙁
I know I’m not alone, nor is Alex as blog articles about this same problem are popping up elsewhere too.
This concept has some severe flaws in it. It needs to be fixed.
WPEngine imho is a managed WordPress host provider. WPEngine is supposed to help manage my WordPress installation, helping me avoid the problems and hassles that more generic hosting can create.
This concept of visits (dating back to 2012) combined with this new overage concept from August when put together is a recipe for trouble.
I agree with Brett. Our visits as measured by WPEngine are TEN times the GA measure. Twice as much I could handle but ten times, when you are paying overage is too much.
In fact most of the traffic seems to be comment spam – bots that just hit a single comment page so not exactly chewing through bandwidth. But it counts as a visit, so you pay..
Not very good and I wonder how many other users are affected by this?
I’m feeling very confused.
I’m getting ready to return to WP Engine and launch my website. I’m anticipating low visits as I grow my brand and build an audience. But it sounds like the potential exists for me to exceed the bandwidth restrictions of the $29.99/month plan and end up getting charged overage fees.
I’m on a tight budget and need to think about this. But like I said, I’m kind of confused. Doesn’t WP Engine screen and filter bots as a part of your safety protocols? It never occurred to me that I could end up paying for overages due to non-human activity on my website.
Hi Carla,
Small world that we’re chatting here now. I can elaborate a little bit about your bot traffic concerns. I’ll reply to the Support conversation you and I currently have. In the meantime, briefly…
The crux of this article is that in most cases you don’t want us to block bots. You need those bots to index your site, for the benefit of your search rankings. And the fact that we count bots as unique IPs/day (same as humans) means those visits are actually *less* expensive relative to human visits than they would be with (for example) a host that charges based on bandwidth.
In most cases, bot visits will still be only a small percentage of your WPEngine-billed visits. But if you do have higher proportions of bot visits, our Support Team will be eager to help you with cost-saving and/or performance-enhancing recommendations.
Ultimately, we’ve found that it’s a very rare occasion that a customer need worry about how bot visits are affecting their traffic and charges. I hope that helps.
Thanks for responding to my comment.
I’ve read about those robot critters and how they feed on blogs and swell the numbers. But I think I’m being too literal (and blasé) in my understanding of how they actually function and the impact on a website’s stats/bandwidth/etc.
I’m now starting to wonder if I can somehow gather them up and bombard the things with some supped up sugar, get them to grow by the thousands, and sell them to bloggers in cute little boxes.
Okay, that’s enough.
One more note . . .
I think it’s important that I don’t get carried away by tech stuff that’s over my head. I love learning, but I also completely trust WP Engine.
This WPE blog post was really insightful and should prove helpful to anyone browsing the comments and wanting to learn more:
http://wpengine.com/2012/04/09/visitors/
:o))
Thanks for your trust Carla, we appreciate your engagement with the blog and will keep doing our best! Kirby