So Sunday was the Super Bowl, which means yesterday USA Today released the Ad Meter ratings, the rankings of all the ads that were shown during the Super Bowl. Which means it’s also time for me to rant about data and measurement.
According to the USA Today story, Amazon won, with a score of 7.18 on a 10-point scale. Their “Alexa Loses Her Voice” ad beat the NFL’s “Touchdown Celebrations to Come” commercial, which had a mere 7.18.
Huh?!
This annual story, while a great deal of fun, is a terrible piece of data journalism. It gives a false sense of implied accuracy, doesn’t give us the margin of error or a sense of who the participants were, doesn’t show what’s significant or meaningful, and probably measures the wrong thing.
The USA Today story shows an exact tie, but other stories show it down to the 10,000th place — Amazon “won” by .0059 points on a 10-point scale (7.1836 vs. 7.1836). I suspect that this difference isn’t statistically significant, and it certainly isn’t meaningful.
The Ad Meter story doesn’t seem to indicate anywhere what the margin of error might be (if it’s even relevant). You can roughly estimate it though. Based on their FAQ, they expected to to have “thousands” of participants.
Roughly 100 million people watched the Super Bowl this year, so let’s call that our population. If 10,000 people participated in the ratings, the margin of error would be about ±.98, or a tenth of point on a 10-point scale. So the 0.0059 difference would not be anywhere near statistically significant.
But we don’t even have to worry about the margin of error if those doing the rating aren’t representative of the larger population. The raters are those who chose to participate — “Yes, US citizens 18 years of age and older can participate. Interested panelists must register in advance at admeter.usatoday.com. Participants will be asked for basic demographic information— age, gender, income, zip code.” But we don’t know if these participants are skewed in one direction or another or, if they are, if USA Today weights the results to be more representative.
What are they measuring?
Most importantly, I doubt the Ad Meter measures anything relevant to the actual performance of the ads. Presumably companies advertise during the Super Bowl to increase awareness of their product, to create a buzz, and mostly to sell more stuff.
According to their data , the worst performing ad is actually one that sticks in my head — Diet Coke’s “Grove,” starring Australian actress Hayley Magnus. As Justin Peters of Slate says, “I am now fully aware that Diet Coke comes in mango. Mission accomplished!”
I didn’t know that before, and now I do — and I’m unlikely to forget it. Of course, I’m also unlikely to drink it, but perhaps that’s just my own peculiar taste in beverages.
But does any of this actually work? Jeri Smith of advertising research firm Communicus conducts research on the effectiveness of Super Bowl ads. Their research suggests that companies just waste their money with Super Bowl ads. According to their work, just 10% of viewers remember the ad and who it was for.
Other Measures
While the USA Today Ad Meter story appears to be the most popular version of this story, other groups have different measures. iSpot.TV looked at “Digital Share of Voice,” essentially what percentage of online activity that ad gets in relation to the other commercials, including views on YouTube, tweets, Facebook likes, and so on. As of today, the Doritos Mountain Dew commercial, “Doritos Blaze vs. MTN Dew Ice” was in first place, in comparison to its fourth place finish in the USA Today Ad Meter.
Particularly noteworthy to me was the Westworld Season 2 ad, which finished in 50th place in the Ad Meter, but in 9th place in iSpot.TV’s measure. From a quick glance, TV and movie ads do relatively poorly in the Ad Meter, but much better in online activity. The 2017 results particularly reflect this, with Netflix’s Stranger Things ad only in 49th place in the USA Today poll, but scoring in second place of iSpot.TV’s study.
You may be saying that this just a fun story…that it doesn’t matter if has any meaning. But it would be relatively easy for USA Today to add in some of this information, to actually inform the public about how what the data means. I don’t mind a fun story, but when they put this much effort into a project, that might as well make it informative too.
End of rant.