Polls, Statistics, and Lies

A few recent commercials I've seen started me thinking about the statistics they cite.

All three of these commercials were for auto insurance products. In one of the commercials, the commercial finishes with the statement, "Save up to 15% or more!"

The second commercial improved on that statistic by pointing out a consumer "could save up to 20% or more!" However, if a rational viewer were to dissect that phrase for a moment instead of immediately being assaulted by the next commercial, he would see that the statement really means nothing. A consumer may save 20%, which is what the advertiser hopes to suggest. Finally because the phrasing is so loose, the consumer may save less than 20%. Or nothing. Or could lose money by switching.

The real "source" of these numbers (15% or 20%) is the company or its advertising agency held a focus group to determine what percentage savings do they need to suggest to consumers that encourages the greatest number of people to consider switching, while still sounding believable.

A third commercial for auto insurance at first seemed to have a more solid claim: Virginia (the state the commercial aired in) consumers saved on average $189 per year.

Unfortunately, this statistic only considers consumers who switched to the company. Anyone who would have paid more to the company advertised would not have switched. Further, anyone who would only save a little in many cases would keep their current insurance as well.

Polls can be similarly misleading. Further, the results are often read so definitively, even though there may only be a few percentage points difference. These polls can often influence opinion and therefore perhaps even policy, instead of serving solely as the barometer they should be.

Internet polls shouldn't ever be quoted for anything. They are self-selecting (meaning people choose to answer the poll). People who aren't passionate about an issue will not participate. In other cases, someone may post a link to the poll on a group's web site encouraging others will a similar opinion to vote. Finally, they are still very susceptible to some people voting early and often. Some technical safeguards can diminish this, but some people can still get around the defenses.

Another problem with polls is that the specific phrasing of the questions may influence the results.

One specific example of a poll question gone awry is from August 1998. In it people were asked whether they had a favorable or unfavorable opinion of former president Bill Clinton "as a person." The editor in chief of the poll estimated this phrasing caused the results to be 10% off from what they would have been if the question was phrased identically to the phrasing on previous nights. Previously, people were simply asked if they had favorable or unfavorable opinions of several people, including the president.

Another example from a poll around the same time period is the question, "Based on what you know at his point, do you think that Bill Clinton should or should not be impeached and removed from office?" How would the response vary without the first phrase of the question? Or even just changing the questioner's inflection on the word "know" could vary the responses. The media, policy makers, and the American people must be more critical in evaluating the information they receive. It is very, very rare to find an organization without an agenda publishing a poll or "citing" a statistic. The next time you run across a poll or statistic, take a moment to question it, question the organization presenting it, question how it was measured. After all, 85% of statistics are made up.