Technology

The mystery of dark traffic to websites

Can you solve the mystery of the web traffic that comes from nowhere?

“We have the whole history of the web wrong” was the headline gracing an article published by The Atlantic in 2012. Two years on Alexis Madrigal’s piece remains a seminal piece of insight for understanding the internet.

Madrigal almost gave us a new phrase too, ‘dark social’. The ‘almost’ caveat slips in because ‘dark’, with its overtones of mysterious knowledge and secret forces, is used to label multiple different phenomena, often interlinked but not all the same.

‘Dark’ can mean parts of the web beyond the reach of search engines. It can mean more deeply hidden networks for dodgy dealings (and valiant campaigners – secrecy protects freedom of speech campaigners in a dictatorship as much as it protects drug dealers in a democracy). But for Madrigal ‘dark’ had a simpler, even less dark, meaning: traffic that comes to a website without the website owner knowing where it has come from.

If, say, you visit a webpage by following a link from a story on Blue Rubicon’s website, then that website manager can crunch through its web analytics and see that visit came via that link. Aggregate the data and you can see over time how much traffic was generated by the Blue Rubicon site and to which pages.

Moreover, such referral data can also distinguish traffic from search engines or paid-for adverts. As a result, it gives website owners valuable information. What stories generate traffic to their website? Are links from new sites generating more or less traffic? How much traffic is coming from search engines or via the web address for a social network? Is paid-for advertising working well?

Or at least, it used to give website owners valuable information. But more and more traffic is coming through with no referral data. Long ago in internet time (a few years back) there was very little traffic with no referral data. A few people would type in your web address directly or use a bookmark in their web browser, so generating a visit without any referral data – but not a huge amount beyond that. The main source of such dark traffic was (most likely) emails – because a link in an email that generates a visit also leaves no referral trace unless it’s a specially constructed tracking link.

The dark is rising

But as Madrigal and others have pointed out since, the levels of dark traffic are on the rise. One part of this is, arguably, not dark traffic. There has been a move from search engines to increasingly hold back the exact search terms people used to get to a website. Increasingly, a website owner just knows the traffic came from a search engine, not that it came from people using keywords X, Y and Z.

In that case, at least you still know it came via search. What is more troublesome is the rise of traffic where you have no good answer to where it came from. The cause? Most likely it is traffic from apps on tablets and mobiles, which do not supply referral data.

Receive a message on your phone in an instant messaging app from a friend, click the link on it, visit a website and most likely you will have added to the dark social traffic count for that site. No matter how unmysterious you feel at the time and how banal the message, it’ll add to the puzzle for the website owner in not knowing where their traffic is coming from.

The mystery is not completely watertight; skilled analysis of traffic data can still tease out clues. For example, if copious volumes of traffic come from Android phones during and just after certain TV shows, then the most likely cause is the sharing of stories about the show in real-time as people watch TV with a mobile or tablet in hand.

But if you don’t make use of such advanced analysis, trying to understand where your web traffic is coming from is just guessing in the dark.

UPDATE
Alexis Madrigal has now written a follow-up to his classic piece:

In the original story, I relied on data from Chartbeat, the web app that media people use to obsessively track their traffic in real-time. Since it came out, I’ve done research that’s caused me to rethink dark social—and so has Chartbeat. And they released new research to me this week showing that a good chunk of what we might have called dark social visits are actually Facebook mobile app visitors in disguise.

The takeaway is this: if you’re a media company, you are almost certainly underestimating your Facebook traffic. The only question is how much Facebook traffic you’re not counting.

Leave a Reply

Your email address will not be published. Required fields are marked *

All comments and data you submit with them will be handled in line with the privacy and moderation policies.