Yu-Ain Gonnano: You will have to forgive me – moving around way too many pounds of gos-se (literally, today, too) over the past few days has made me a little stupid – but I will probably have to get back to your comment in the next couple of days.

Let me retract my complaint about weighting, and instead convert it inot a complaint about them weighting solely based off the ages of the respondents – as you say, there are at least two, and probably more, more-interesting aspects of the situation to consider, if not more statistically-relevant ones.

Not exposing the questions is probably my biggest grief with them, though – like I said in the post, I am willing to wager that the fact that drinking alcohol while carrying is still illegal did not come up in any of the questions, and when people hear “in bars”, their first thoughts of course gravitate to alcohol.

Would that we lived in a society where designated drivers were also designated carriers.

1) About 700 people actually is fairly close to a sufficient sample size for a survey of this type. The size of the entire population is completely irrelevent since the variance of the mean decreases as the sample size increases completely independently of population size.

In fact, once you get much above a sample of a couple of thousand the human error rates tend to become larger than the sample error rates as 1s get misread as 7s etc.

2) Weighting does not introduce bias, it eliminates it. In fact, one of the most egregious cases of sampling error (the Dewey v/s Truman polls) was caused exactly because weighting was *not* done. (Strickly speaking, there is no such thing as “unweighted”. The arithmetic mean uses equal weights and “unweighted” samples simply uses the sample weights as the population weights)

Ex: 10,000 people total: 1,000 Type A = 10% in favor and 9,000 Type B = 50% in favor for a total population result of 46% in favor.
I sample 100 of each with the following results Type A = 12% in favor and Type B 53% in favor.
Unweighted Average = (.12 + .53)/2 = 32.5%
Weighted Average = .12*10% + .53*90% = 48.9%

It is the *unweighted* average that is biased because it applies a 50/50 weighting scheme when the actual population weights are 90/10.

The problem with the survey are that

1) They left off two really obvious predictive variables: Political affiliation and Rural/Urban residence. You can’t possibly tell me that Repubs and Dems have similar views on Guns. Or that the downtown highrise apartment dweller’s views will be the same as the person whose closest neighbor is a mile down the road.

2) How were non-responses handled? Left as null values? Bad idea unless you have reason to assume people that refuse to answer are no different than people that do. Highly unlikely that. Deleted? Even worse. Now you’re assuming that people who hang up the phone on you don’t exist. Mean replacement? A completely worthless idea because they have already decided that they need to correct for demographic differences. Nearest Neighbor? Closer, but do they only account for age, race and sex? As before, party affiliation and geographic distribution will be important. Urban democrats and a rural democrats tend to disagree on guns.

3) (You did mention this) They don’t show you the questions. Interviewer bias is a *GREAT BIG FRAKING HUGE* issue and cannot be overlooked. The question “Would you like to be rich?” and “Would you like to be rich if it meant having to kill someone?” will get wildly different answers even though both purport to determine whether people support the accumulation of wealth.

This is what most of the “people overwhelmingly support ObamaCare” polls do. “Do you want the gov’t to give people cheaper healthcare” will get a very different answer than “Do you want the gov’t to give people cheaper healthcare if it mean higher taxes and premiums and longer lines to see a doctor for yourself”. Everyone wants free stuff, but in the real world things have costs.