No More Surveys!
Back when the earth was still cooling I designed and managed surveys for a living. The survey topics tended to be esoteric (for example, database copyright policy, electronic publishing, knowledge sharing by teachers, astrophysicists, and cancer researchers, etc.) but the methods we employed were standard at the time including careful questionnaire wording and testing, detailed sample design, stratified random sampling, follow-up of non-response, projecting survey results statistically to population totals, etc.
I’ve maintained an interest in survey techniques especially regarding how you ask questions so as not to bias the response one way or another. When I get an email or telephone request to take a survey I’ve tended to respond positively; I used to hire and manage the folks on the front lines of survey administration so I’ve usually tried to be polite and cooperative.
I’m now thinking of refusing to participate in any more surveys.
The reasoning is simple: there are too many fake surveys floating around. A couple of weeks ago, for example, I responded to a phone survey request. About halfway through a fairly complex set of questions I realized, based on question wording, that results were being tilted towards an increased use of natural gas as part of our national usage mix. Responses were being limited to forced choices between clearly identified positive or negative (no neutral or middle ground). Energy options were being listed without mentioning nuclear, solar, or wind. Generally, any potential alternative to increased exploration for and use of natural gas was being characterized as tantamount to destroying the US economy, mom, and apple pie.
I patiently answered most of the questions and explained to the survey taker – politely — at the end that the results of this survey would be worthless, but I knew that would do no good.
I resolved to discontinue responding to any more surveys.
This morning my resolve weakened. I received a request to respond to an online survey that was described as ranking the policy issues I feel are most critical here in the Commonwealth of Virginia. The sender was a public official for whom I have voted before and whose public policy positions I tend to support. But a few questions into the “survey” it switched to an explicit fundraising appeal. Granted I tend to support the causes being discussed but I discontinued the survey at that point figuring that the results would be worthless, just as worthless as the above described “energy survey.”
So, no more surveys, even if they are sponsored by a “friendly” source. What’s the point? The results are predetermined, the statistics will be quoted out of context, and “unfriendly” results will be buried.
Here’s my new policy: if you want me to respond to a survey either online, in person, or via telephone, please tell me in advance:
- Who is paying for the survey?
- Who is conducting the survey?
- What can I get a copy of the results (not just cherry picked results).
- Will I have an opportunity to provide open-ended comments or suggestions especially when the question wording is confusing or biased?
- Where can I get data on response and completion rates?
In other words, if you make the people, organizations, and processes surrounding your survey more open and transparent, I’ll be more likely to respond. I don’t have to know who is responding, I just want to trust that the time I spend with your survey will generate some insight into reality.
Know also that I intend to ignore polling numbers as the election season ramps up. For all the reasons suggested above I believe that polling numbers released to the public without serious methodological detail will be worthless and not to be trusted.
But I’m not holding my breath.
Copyright (c) 2016 by Dennis D. McDonald
