Jan 1

Jan 1 Why I don’t trust political polls -- and why you shouldn’t, either

Data, Analytics, Politics, Policy, Surveys, Propaganda

By Dennis D. McDonald

The December 31, 2022 New York Times article The ‘Red Wave’ Washout: How Skewed Polls Fed a False Election Narrative provides much food for thought for anyone interested in understanding what political polls tell us about voting behavior.

The word “skewed“ in the articles title suggests there are nefarious forces at work when polls are conducted by those with political axes to grind, especially since important messaging and spending campaign decisions are being made based on the results of these polls.

As someone who spent years designing and managing statistically-based surveys, research projects, questionnaires, and data analysis efforts, I am surprised by none of what is reported in this article.

I understand what can go right and wrong in conducting polls. Much of this can be boiled down to the old saying, “There’s many a slip between the cup and the lip.” Events that can influence survey based political poll include:

Selection of the survey respondents
How questions are worded
Assuming people tell the truth when polled
Failure to control for non-response
Biased survey management
Cherry picking survey results for public consumption
Asking people to predict future behaviors

Any of these factors can influence the numbers that will be reported and analyzed. The New York Times article doesn’t get into the weeds about such issues. Instead it focuses primarily on the trustworthiness and professionalism of those managing the polls.

This lack of detail in the article is understandable. As a former number cruncher, I do understand how these bullet point items can impact survey results. Their cascading effects can be very complex to analyze, even if one is relatively sophisticated about survey data collection and analysis. Consumers of polling information frequently lack analytical sophistication. When survey results are presented publicly the details of how the survey was conducted are often glossed over or left out completely.

There was a time when I was willing to take the time to respond to polling questions posed via telephone, but no longer. Survey staff usually refuse to identify the actual sponsor of the survey and biased wording of the questions is easy to detect for anyone who has designed as many questionnaires as I have.

Also, email surveys run directly by political candidates are usually part of a fundraising effort and have little statistical validity for a general population of voters given the lack of control over the responding population.

A fundamental question not addressed in the New York Times article is, “What is the relationship between someone’s political opinions now and how one votes in the future?“ Even if we accurately assess via polling one’s views on the range of political questions, what guarantee do we have that these views won’t change when one stands alone in the polling booth?

Knowledgeable analysts will understand the ins and outs of analyzing sentiment trends over time, They will also understand the challenges, logistical and cost wise, of conducting a series of polls that are similar enough to support trend analysis.

Don’t get me wrong; I still love conducting and analyzing surveys. I also know when political polling results are published and disseminated via traditional and social media that insufficient detail on survey methods will be provided to enable an assessment of the data’s validity. That’s a big reason why I no longer pay much attention to political polls.