It’s an irony that is keenly felt by virtually every media entity, whether it is a tiny web-native outlet or a giant traditional publisher: the Internet, one of the most easily — and frequently — measured media in the history of humanity, is also one of the most difficult when it comes to getting a straight answer about who is consuming your content and when. That conundrum is at the center of a new report from the recently launched Tow Center for Digital Journalism at Columbia University, entitled “Confusion Online: Faulty Metrics and the Future of Digital Journalism.”
The report notes that even among the leaders in web measurement such as comScore and Nielsen, there can be widespread disagreement about how many visitors a site gets, how long they spend there, and plenty of other important metrics that might help media websites do their jobs better. In May, for example, comScore said that the Washington Post’s website had 17 million unique visitors, while Nielsen said it had less than 10 million — a difference of more than 40 percent. Estimates from the two companies of Yahoo’s traffic differed by 34 million, or approximately the population of Canada.
Although the report doesn’t go into the details, these gaps stem from differences in the way that each measurement firm pulls in the data it uses. Some companies (such as comScore) have software that users download and install, while others use information that comes directly from Internet providers. Some actually still sample web users by calling them on the telephone and asking them what sites they visit regularly. And all of the numbers being used that come from the servers themselves are polluted to some extent by the effect of “click-bots” and other scams that can inflate traffic counts, in some cases by huge percentages.
[inline-pro-content] And it’s not just comScore and Nielsen — there’s also Omniture, which many larger media companies use, as well as Google Analytics and other services that measure traffic directly from a site’s servers, and services like Hitwise and Compete that take data from ISPs and other sources. And then there are newer analytics tools that can give publishers an almost real-time look at the activity on their site, such as Chartbeat. While some news sites are excited by the potential of all this data, others say there is so much conflicting information that it is actually making their jobs harder rather than easier — particularly when it comes time to talk to advertisers.
Tom Heslin, senior vice president and executive editor of the Providence Journal, calls this the “irony of expectations”: neither publishers nor advertisers have been able to keep up with the flood of data. “Our biggest challenge is to simplify solutions for our clients, even for national advertisers,” he explains. “The development of metrics has far outstripped knowledge of ad buyers and sellers.”
As this comment makes clear, the biggest issue with metrics isn’t that newspapers or websites can’t figure out what their readers want to read — although that is part of the problem. The big issue, in financial terms at least, is that most publishers rely on advertising for the bulk of their revenue, and there is no simple standard for showing traffic or readership online the way there is in the dead-tree world, where NADBank and other measurements are accepted (although they also have their flaws). That leaves media outlets in a quandary, says the Tow report.
Uncertainty about audience measurement hinders online ad spending, with buyers and sellers of media favoring incompatible metrics [and] increasingly, the decisive information resides not with the publisher but in the databases of intermediaries such as ad networks or profile brokers.
So what is the future of online measurement? The report doesn’t come out and say this, but it probably looks a lot like the present — multiple competing measurement sources, each of them flawed in a slightly different way, so that most publishers have to throw as many different numbers as they can into a hat and then average them all out and hope for the best. The report’s authors say there is a chance that Nielsen and comScore might merge and become the dominant player and the de facto standard, or that publishers and advertisers will settle on Google Analytics as the main arbiter of numbers, but both of those seem relatively far-fetched.
In all likelihood, digital media will have to continue muddling along as best it can, for better or worse — tormented by the fact that in a medium where measurement is so easy, measurement by itself can mean very little.