How bots inflate traffic numbers: from the raw log to the real audience

A site we watch had, one day, almost twice as many “visitors” as on the days around it. The reflex is to be pleased and to look for which article “took off”. When we looked closely, almost nothing in that spike was human. It is a good lesson in how easily traffic numbers lie to you, if you do not ask where they come from.

Why “unique IPs plus User-Agent” overestimates

The most common way to count visitors from server logs is: add up the unique IPs, exclude the ones with a bot User-Agent. The problem is that this method was good ten years ago. Today’s bots send a perfectly valid Chrome or Safari User-Agent, exactly like a real browser. A filter that looks at the User-Agent no longer sees them, so every scanner, proxy or probe ends up counted as a “human”.

The result is a number that looks good on a chart and means nothing.

What the “spike” actually was

When we broke the day down by source, here is what we found in place of an audience:

• Health checks from our own infrastructure. A load balancer or a probe checking whether the site responds, hitting the homepage every few seconds, was being counted as a visitor. Infrastructure does not read blogs.
• Cloud scanners that lie about their User-Agent. IPs from data centers, with a browser User-Agent, that were actually requesting /admin.php, WordPress files and configuration panels. Automated recon, all answered with 404 on a site that has none of that.
• A swarm of IPs that hit the homepage once and left, without looking at anything else.
• Over 80% “direct”, that is, with no referrer at all. A real audience comes, in large part, from search and social networks, with a referrer. Massive traffic with no provenance is a signal, not a success.

No article had “taken off”. No one was reading anything. Just the internet knocking on your door.

What a real reader looks like

The difference between a human and a bot is not in the User-Agent, but in behavior:

• a real reader has provenance: they come from a search, a LinkedIn link, a newsletter;
• they move around: they open the article, then another page, maybe the contact page; they leave a trail;
• they come back sometimes, at another time, from another device.

A bot usually appears from nowhere, touches a single page and disappears. When your “spike” is made of thousands of one-page visits, with no referrer, concentrated on the homepage, you are not looking at an audience, you are looking at noise.

How to count more honestly

You cannot clean everything, but you can remove what is clearly not human:

• exclude internal infrastructure from the counts (health checks, probes, internal addresses);
• reclassify as bots the scanners that generate many 404s, regardless of the User-Agent they send;
• cut the flood from a single IP making dozens of requests, because a real reader does not behave like that.

But be honest about the limit: bots that imitate a browser and touch real pages only once cannot be told apart from a human without also cutting legitimate readers behind a shared IP. That is why, for audience, a source that recognizes bots (for example a CDN that filters them at the edge) is closer to the truth than the server’s raw log.

Why it matters, beyond vanity

An inflated number is not just vanity. You make wrong decisions based on it: you think a topic worked and repeat it, though no one read it; you estimate a conversion that does not exist; you justify a budget from a phantom audience. And on the security side, the same noise that inflates “visitors” also hides the real scans: if you do not separate the bots, you do not see who is seriously probing you either. Honest measurement is, here too, a matter of being able to show what is real, not just what looks good, exactly the idea in the control that passes the test.

What we take away

“Big” traffic that leaves no trace of human behavior is not an audience. Before you celebrate a spike, ask two things: where did it come from, and what did it read. If the answer is “nowhere” and “nothing”, you measured bots, not people. A number with no provenance is a vanity metric.

Want to know how much of the traffic and the “security events” you see are real and how much is automated noise? Contact us and we start from an honest look at the numbers.