unsurprisingly, i have never been polled

A (not so) recent post by Thirdpower reminded me of something I had been meaning to post about for a while now.

Consider every “poll” you see suspect until you can see the source data.

Yes, that is an unquestionably bold statement, but bear with me a moment. To begin with, only 9% of those people called actually comply with the pollsters. Nine percent. This problem compounds itself with the majority of the non-responders typically self-identifying as “Republican” or “Independent”, but the fact remains that those who respond to polls are those who want to take them, which creates a certain level of bias in the results.

To be certain, pollsters have gotten tremendously good at “compensating” for that bias, but once you start normalizing numbers, the bias of the person running – or paying for – the poll rears its ugly little head as well.

However, that was not the real reason I wanted to write this post; instead, there is this small detail that far too many people forget, or never actually knew: pollsters cannot robocall cell phones. Or, rather, they can, but it is illegal for them to do so.

As Pew Research explains, or, really, rationalizes, this legal block drastically increases the costs of polling cell phone owners, because numbers have to be dialed manually, the pollster company might have to reimburse the cell phone user for airtime (less so these days, though), not everyone who has a cell phone is old enough to be polled, people tend to check their caller IDs better on their cell phones, and various other reasons. In the end, the vast majority of pollsters simply do not bother.

How does this affect things? Well, take a look at this report from the CDC (*.pdf warning) regarding how many households tend to be cell-phone-only (i.e. do not have a landline at all) and how that skews the demographics. Do note that this survey was conducted in person, so it it is arguably more accurate than robocall surveys.

In short, 35.8% of households are “wireless-only” as of JUN12, with the overwhelming majority of 18-34 year-olds falling under that heading. Men were slightly (3%) more likely to be wireless-only than women, the Northeast is significantly less likely to be wireless-only, and Hispanics were more likely than non-Hispanics. In all cases, the trend has been steadily increasing since 2003 (when the total wireless-only households were around 3%), and if the current growth continues, it would not surprise me to see the majority of all American households being wireless-only by 2020.

Now, with the younger generations being markedly underrepresented, the Northeast being overrepresented, and various other hiccups and speedbumps along the way, how do you think that will affect the outcome of polls, and specifically polls relating to firearms? The ranks of young recreational shooters are growing every year, and getting more and more vociferous about their hobbies and rights, while the Northeast is notoriously a bastion of authoritarianism and anti-firearms sentiment – one need only look at NYC for a perfect example of both.

Sure, now that the specific error percentages are known, pollsters can account for it – to some extent – but will they, and how will they? The farther and farther you get from the source data, the less accurate information can become; it all depends on who or what is separating you from the originals.

Which brings us back to my original point – if someone is unwilling to share their source data with you, and then show you how they got from point A to point B, I would strongly advise against believing a word they have to say. They might be correct, for all we know, but that is rather the problem – we do not know, and unless someone can demonstrably prove their case to me through a logical chain of facts and reasoning, I simply cannot bring myself to care.

(And, as always, our individual rights – as observed and protected by the Constitution – are not subject to popular opinion or poll approval. Even if the Second Amendment itself were amended out of existence, I, as a living, breathing, thinking human being, would still retain the rights to self-defense and ownership of private property, and from those rights the right to own firearms logically extends.)

9 thoughts on “unsurprisingly, i have never been polled”

  1. The “wireless only” thing is something I bring up virtually every time someone shoves poll results in my face. I’m 36 years old, and I have a half dozen or so close friends, and a couple dozen more distant ones that I’m still in contact with (real contact, not Facebook bullshit), all of whom I’ve been friends with since high school or earlier. Among that group of… let’s say 50 people, I can count on one hand the number of them who have a landline in their home. And I’m not even sure I’d fill up that hand.

    That’s 45 or more people in my age bracket who will never be included on any poll, regardless of who does it or how accurate that polling company may historically be. Quinnipac, Pew, Harris Interacive, USA Today, it doesn’t matter. We’re not on it.

  2. We have been wireless only for 2 years and don’t miss either having to check the recorder, nor answer the sales calls.

    I wonder if any of the polling Masters like CNN have tried to explain the shortcomings of their polling now. Or if they care

  3. We’ve been wireless-only for nearly a decade. I can literally say, “We hated our landline and dumped it before it was cool.” I’m with “Matt in FL”; out of all the people in/near my demographic I know, the vast majority don’t have landlines.

    Now, to poll results: I’m generally suspicious of any “research” where data “normalization” is required. In my experience, “normalization” is researcher code for “the data didn’t match what we expected, so we changed it.” An honest researcher will adjust the data against known biases so that it (hopefully) more-closely matches reality, but a less-scrupulous researcher will adjust the data to more closely match his/her preconceived opinion (read: agenda). Generally, the former will be happy to show and publish his/her work and the reasoning behind every adjustment, but the latter rarely will.

    My statistics teacher was fond of saying, “The numbers never lie; they just get misrepresented.” I’d add that they can also get “normalized” into inaccuracy, if not outright falsehood. Not that it matters when it comes to fundamental, Constitutionally-guaranteed rights; as you and others say, they are not for sale, not up for a popular vote, and in a perfect world would not be up for discussion. Period.

  4. I was landline free in 1999, although in 2004 i moved somewhere that had no internet but dial up for a few years, so i had one then. But i dont answer numbers i dont recognize anyway.

  5. We’re still landline bound because we live in an area w/ crappy reception but I’m part of that 91% as I don’t answer any number I don’t recognize and regularly screen calls.

  6. I always enjoy being polled, and will gleefully add more data that the Leftists desperately want to not be there(and then wish the caller a nice day afterwards. Last guy I told to enjoy the Superbowl game.). I also will not answer my cell phone if I don’t recognize the number.

    Much better, however, is my brother, who will gleefully hand in junk data, keep track of every exaggeration he has spouted, and then confirm them when called upon to verify that he’s not simply trolling the poller. Which he is.

    He usually takes the phone into another room while we listen to a second phone on speaker and try desperately not to let our laughter be heard.

  7. Count me in with the people who don’t have a landline. The only people who I know who still have them are the parents of my friends. Everyone I know sub 40 does not have one, and I mean everyone. Of course I never trusted polls anyway, since the biggest problem is they are optional.

  8. The wireless only issue is actually a relatively small problem. It takes surprisingly few people to be reached in person, by mail, etc to get estimates for that demographic. These estimates can then be resized to the population of wireless only people. Stratified Random Sampling is an easy fix.

    Even lying isn’t that big of a deal most times as someone else is likely lying in the opposite direction and outliers have much less effect than most people think.

    The massively bigger issue is the 91% non-response rate. Depending on the topic the non-responses may be completely at random and ignorable. I see no reason why chocolate ice cream lovers would be more or less likely to answer than vanilla ice cream lovers.

    Politics is a completely different question. A non-response there could mean simply “I wasn’t home”, “I don’t answer from numbers I don’t know”, to “Fuck you, it’s none of your damn business”. All these groups would need to be treated differently. You’d need to know how likely each group is and know something about their demographics that you could possibly correlate their answers to (a libertarian is much more likely to be in the latter group, for instance).

    Lots of room, for even the well meaning, to be extremely honestly wrong, much less those of ill-will.

  9. @ Matt in FL: In fairness, some poll companies do have methods for calling cell phone customers, including human dialers and reimbursement systems, and some of them specifically mention doing so in the notes for their respective polls… but from my observation, such examples are fairly rare and unusual.

    I think the only people I know who have a land-line at all, much less who still use it, are my parents. That said, they screen their calls even more fiercely than I do.

    @ Fill Yer Hands: I have been wireless only since I left the house, and have never looked back. I would imagine a significant portion, if not outright majority, of my generation is the same.

    @ Archer: I would observe that “normalization” is not, in and of itself, a necessarily bad thing; in general, if an opinion poll does not follow something approximating a normal distribution, you probably did something wrong (asked the wrong question, polled the wrong people, etc.). In general. The problem, such as it is, is that there are so very many ways to get data to fit a normal curve that statisticians can… craft… the data to show what they want it to.

    This is largely why, when I do data-related things, I generally stick to just showing the actual, raw data, and let people draw their own conclusions.

    @ dave w: Whee, dialup… Do not miss that.

    @ Thirdpower: And, seriously, given how many robocalls a landline gets (we “had” one for about three months as part of a television service deal, and finally just unplugged the phone entirely), “filtering” is the name of the game.

    @ Volfram: As Yu-Ain says, tossing out outliers is not only hard, but generally one of the first things pollsters / statisticians do; that said, wasting those pollsters’ time is a worthy goal indeed.

    @ LucusLoC: Ayup. Much like voting, you only get to hear the opinion of those who wish to express it.

    @ Yu-Ain Gonnano: The problem is, yes, almost all of he shortcomings for polls can be corrected, but do all pollsters actually bother to make those corrections? In these days and ages where polls are executed over a few days across a seemingly smaller and smaller sample size each time, just to be on the bleeding edge of news and so forth, even stratified sampling starts looking suspect when your sample population is measured in hundreds.

    Hell, when you look at the questions some of these pollsters ask, it is plainly apparent they are trying to make the public opinion, rather than report on it…

Comments are closed.