So yesterday was the Super Bowl, which means this morning USA Today released their annual Ad Meter ratings. That also means today I engage in my annual rant about how the Ad Meter Ratings terribly represent data journalism, use of data, and measurement.
According to their ratings, people like the “100-Year Game commercial” for the NFL the most.
This annual story, while a great deal of fun, is a terrible piece of data journalism. It gives a false sense of implied accuracy, doesn’t give us the margin of error or a sense of who the participants were, doesn’t show what’s significant or meaningful, and probably measures the wrong thing.
The Ad Meter story doesn’t seem to indicate anywhere what the margin of error might be (if it’s even relevant). We can roughly estimate it though, given the size of the population of viewers and the number of participants in the Ad Ratings.
Close to 100 million people watched the Super Bowl this year, so let’s call that our population. Based on their FAQ, USA Today expected to have “thousands” of participants. If 10,000 people participated in the ratings, the margin of error would be about ±.98, or a tenth of point on a 10-point scale.
That assumes that those participating in the ratings are representative of the larger population we’re interested in (i.e., people purchasing the products being advertised). Without knowing a lot more about those that take part in the ratings, it’s impossible to tell this.
What are they measuring?
Most importantly, I doubt the Ad Meter measures anything relevant to the actual performance of the ads. Presumably companies advertise during the Super Bowl to increase awareness of their product, to create a buzz, and mostly to sell more stuff. And we don’t know how well the commercials do at this.
For example, I must have been one of the few people who quite enjoyed the Andy Warhol Burger King commercial, which the Ad Meter ratings pegged as the lowest rated commercial. It certainly got my wife and I talking about it. “Is that really him?” “If not, it’s a really good fake!”
The commercial sent down a rabbit hole of Google searches, to find out there’s a four-minute clip of Andy Warhol eating this burger. It’s part of a longer film from 1982, 66 Scenes of America, by Danish poet and filmmaker Jørgen Leth. Just about the entire film appears available on YouTube!
But most importantly, I’m not going to forget about Burger King. Not that this will change my behavior – I largely avoid fast food, and even when I did eat a lot more of it, I never liked Burger King. So, for me, this ends up being very effective, but expensive advertisement for Jørgen Leth.
Most importantly, does any of this actually work? Jeri Smith of advertising research firm Communicus conducts research on the effectiveness of Super Bowl ads. Their research suggests that companies just waste their money with Super Bowl ads. According to their work, just 10% of viewers remember the ad and who it was for.
While the USA Today Ad Meter story appears to be the most popular version of this story, other groups have different measures. iSpot.TV looked at “Digital Share of Voice,” essentially what percentage of online activity that ad gets in relation to the other commercials, including views on YouTube, tweets, Facebook likes, and so on. According to their measure, the Avengers: End Game commercial scored in first place, in comparison to 32nd place finish in the USA Today Ad Meter.
You may be saying that this just a fun story…that it doesn’t matter if has any meaning. But it would be relatively easy for USA Today to add in some of this information, to actually inform the public about how what the data means.
I don’t mind a fun story, but when they put this much effort into a project, they might as well make it good data journalism, rather than bad.
End of rant