Traffic that is not an audience

One of our sites was serving a request count that looked, on a dashboard, like a product with an audience. It was not. Almost none of it was people.

The giveaway is uniformity, not volume

Real traffic has texture. Varied clients, referrers that make sense, bursts that follow time zones, people who load a page and then load something related to it.

Automated traffic has none of that. Requests arrive in a steady sweep, from a narrow range of addresses, with an identical client signature and no session structure at all. Once you have seen the pattern it is unmistakable.

What makes it worth writing about is that the client signature was forged to look like an ordinary browser. A crawler that identifies itself is easy to exclude, and most well-behaved ones do. One pretending to be a person is counted as a person by anything that believes what the client says about itself — which is most tools.

How to check

• Group requests by client signature. One value accounting for the overwhelming majority is not a popular browser; it is one client.
• Group by address range. Traffic concentrated in a few hosting ranges is not an audience.
• Look for sessions. A person loads a page, then its assets, then usually another page. A scraper requests pages and nothing else.
• Compare against anything that requires a human — signups, purchases, comments. A wide gap answers the question.

Why it changes decisions

The case for keeping that site running rested on traffic figures that turned out to describe a scraper. Once the number meant what we thought it meant, the decision to close it made itself.

It also changes what you improve. We had spent real effort on page performance for an audience that was not experiencing it.

Before you optimise a number, find out who is generating it.