OK, here’s my first take on what a poll’s “margin of error” really means. (Title of this post shamelessly cribbed from the Car Talk guys, of course!)
First, you don’t actually get enough information just by knowing what the margin of error is. You also need to know the “confidence interval.” Apparently, a confidence interval of 95% is common in polls, but other levels can be used (99% is also commonly used in some kinds of studies), so you need to know. Fortunately, Survey USA tells us: 95% it is. (Aside to statistics geeks: a 95% confidence interval corresponds to two standard deviations away from the mean; a 99% interval corresponds to three, all assuming a normal distribution.)
So: a result that 55% of the people in an appropriate random sample support candidate X, with a 5% MOE at 95% confidence, means that we can be 95% certain that the actual support for candidate X in the relevant community is somewhere between 50% and 60%. (95% certain? What does that mean? As I understand it, it means that if you did this survey 100 times, you’d get an answer in the range you expect 95 times.)
Applying this to the results from the last two Survey USA Dem primary polls, here’s what we actually know.
- We can be 95% certain that the support for these candidates among likely Democratic primary voters falls somewhere in the following ranges (previous poll results in parentheses):
- Patrick: 30.4-39.6 (32.1-41.9)
Gabrieli: 25.4-34.6 (22.1-31.9)
Reilly: 22.4-31.6 (21.1-30.9)
We also know, from a spiffy little calculator that Kevin Drum set up, that in the last poll there was a probability of 99.5% that Patrick was actually ahead of Gabrieli, a probability of 99.8% that he was actually ahead of Reilly, and a probability of 61% that Gabrieli was actually ahead of Reilly. (Note that this is just the probability that candidate X is ahead of candidate Y by some unspecified amount – as little as 1 vote would do it.) In the current poll, those numbers have changed a bit: the probability that Patrick is “really” ahead of Gabrieli is now down to 91.3%; the probability that Patrick is ahead of Reilly is down to 98.7%; and the probability that Gabrieli is ahead of Reilly has increased to 80.8%.
Note how the probability calculations reflect the margins of error. Since the margin of error means that we can say, with 95% certainty, that the candidate’s “real” support is within the expected range, one would expect that, if one candidate’s range has no overlap with another candidate’s range, the probability of that candidate being ahead would exceed 95%. And that is exactly what we see. Looking just at Patrick and Gabrieli, for example, the most recent poll shows an overlap in the expected (i.e., estimated with 95% confidence) “real” ranges of support – that is, the lower limit of Patrick’s expected “real” support is 30.4%, and the upper limit of Gabrieli’s is 34.6% – though neither of those extremes is all that likely. Accordingly, the probability that Patrick is “really” ahead of Gabrieli is 91.3% – pretty good, but not 95%. In the previous poll, on the other hand, there is no overlap in the expected ranges, and accordingly the likelihood that Patrick was “really” ahead of Gabrieli at that time was 99.5%.
So I was wrong to say that there’s no meaningful difference between the last poll and this one. There is: the likelihood that Patrick is “really” ahead of the closest of his rivals is down from over 99% to just over 90%. He’s still looking good, but the difference is meaningful.
However, it is also wrong to describe the current poll as a “statistical dead heat,” as is commonly done when the margin of error allows overlap between two candidates. As the discussion above shows, all an overlap between the expected ranges for the two candidates really means is that the “real” likelihood that one is ahead of the other is less than 95%. How much less? Use the spiffy calculator referenced above to find out – all you need to know are the percentages reflected in the poll and the sample size.
Another fun fact: when comparing the lead of candidate X over candidate Y, you cannot simply add the two margins of error. The actual calculation is complicated, but a good approximation is to multiply the poll’s reported margin of error by 1.7 if you want to figure the margin of error for one candidate’s lead over another. So the current SurveyUSA poll’s margin of error of +/- 4.9% becomes an 8.3% margin of error when comparing two candidates.
That’s all for now. If I’ve got some of this wrong (and I wouldn’t be surprised if I have), feel free to correct me. And more to come, no doubt!
It’s always a pleasure to see someone in the media promote better statistical understanding!
Compare this with Jon Keller’s contention that you only need to get over 33% of the vote to win a three-way race.
<
p>
Thanks for shedding some light, David, though iirc a 99% confidence interval is rarely seen in political polling. It’s too much extra effort for too little extra payoff. Polling is expensive, after all.
the stuff I looked at suggested that a 99% confidence level (3 standard deviations) was commonly used for some kinds of studies, but that 95% was much more common in political polls. I’ve updated accordingly.
Do you KNOW how many people lie to pollsters? Say they are voters when they are not? LIE about who they support? Just to ‘goof up’ poll numbers? There are people who think election returns are wrong because poll numbers say so! (Me, I think Ted Kennedy hasn’t been reelected in years based on his approval ratings…)
<
p>
Unenrolleds are the worst, with GOP’s a close second. IMHO, both feel that they are being ‘massaged’ towards a particular answer (and they are often right) and get truculent. And a Suffolk poll of 400 people – over 50% Democrat registered – is as useful in deterimining eventual election results as chicken entrails.
<
p>
The ONLY people who willingly and truthfully answer poll questions are Kool-aid drinkers like Me, Charlie, EB3, et al.
<
p>
Say it with me – the only poll that matters is one held in November every other year.
’cause they probably lie equally and cancel each other out.
<
p>
I’ve seen papers that suggest that early polling over-counts support from the left and accurately counts support from the middle and right.
<
p>
Maybe that’s why Dukakis and Gore and Kerry polled so well early. Their supporter gave ’em false hope and left them at the alter. Dunno. It’d be an interesting analysis.
<
p>
The holy grail: the youth vote.
<
p>
One reason being is that younger people are more liberal and don’t vote (but do answer polls); older people are more conservative and do. You’d think poller would be able to use the ‘likely voter’ adjustment for that, and I’m not sure.
<
p>
If true, it seems to bode ill for Mr. Patrick considering his strong reliance on those who identify their ideology as liberal. Compound that factoid with the apparent success from Mr. Gabrielli’s tv campaign. If true, too bad, it’d be great to see Patrick on the ballot in November.
and taking the oath in January.
David, excellent post. Thanks for helping us to understand the statistical aspect better, but there’s one thing that I’m still unclear on- the Undecided vote.
<
p>
According to this analysis, we can be 95% certain that between 3.4-13.6% of likely voters are undecided?!?!? Does anyone actually believe that’s even close?
<
p>
This poll encourages the respondents to give an answer, which for a great many voters may very well mean “flip a coin” at this juncture. Therefore, I am not certain that the 95% confidence is realistic.
<
p>
As they note in their methodology:
<
p>
“There are other possible sources of error in all surveys that may be more serious than theoretical calculations of sampling error. These include refusals to be interviewed, question wording and question order, weighting by demographic control data and the manner in which respondents are filtered (such as, determining who is a likely voter). It is difficult to quantify the errors that may result from these factors.”
<
p>
The bottom line is that I’d rather be up than down (and my guy’s up) but with six weeks to go none of us should get too worked up about this one way or the other.
i’d be amazed if more than 10% of likely primary voters are still undecided…
Only 10%? I was thinking more like 20-30%. I think those of us involved in the campaigns tend to think that everyone is following this as closely as we are, when in reality, there’s an awful lot of people out there who still don’t even know who these guys are.
1) Built-in likely voter bias. People who have opinions about the candidates are more likely to vote in a primary than those who are undecided. It’s probable that the SUSA likely-voter model takes this into account.
<
p>
2) Leading questions. Again, the question says “imagine yourself in the voting booth” (or similar). This implies that a choice is necessary to answer the question. I haven’t decided on who I’m going to vote for for Lieutenant Governor yet, but if someone said to me “You’re in the voting booth now, who’s it going to be?” I’d have an answer. Doesn’t mean that I’ll end up in the same place next month, though.
There’s little doubt that the non-sampling sources of error (as also noted by PP upthread) may be far more significant than the sampling error, and of course margin of error only measures the latter. The thing to remember is that with all polls, they represent at best only a snapshot in time – the “if the election were held today” kind of thing. Of course, the election isn’t today, and things can and will change between now and September 19.
<
p>
And I think your bottom line is basically right. Better to be ahead than not be ahead, but don’t get too comfortable.
Thank you David for the research, know that you’ve given me a math headache–I think you missed a major point, although you being reality-based, it’s understandable.
<
p>
Except for the pollsters and series wonks, no one is going to want to have this level of understanding about MOEs. Media folks will be as superficial as always and write that Gabs cut Patrick’s lead in half. And the campaign folks are so hypersentitive about everything that emotionally the poll changes matter. As I said yesterday, there were lots of smiles in Gabns and Rielly’s campaigns, and while I’m sure Patricks people shrugged it off as MOE, their stomachs turned when they heard the numbers.
<
p>
So beyond the math is the perception
But we do what we can to bring reality back into the discussion!
I really appreciate the effort you put into this and the careful references. I think it went a long way to clearing up some of the confusion around polling numbers and what they mean.
<
p>
That said, I don’t think we’re quite there yet. I will assign myself the task of coming up with an additional response (one that won’t give “Frank” a headache!) in future, but right not I just don’t have time.
<
p>
My only complaint about your otherwise excellent post is this statement:
<
p>
Actually, Patrick’s “real” support (to use your terms) could be as low as near zero (maybe the pollsters happened to call the only Patrick supporters in the whole state!), and Gab’s could be as high as nearly 100%. Anything is possible, however unlikely, but the purpose of a poll is to measure the mood of the general population based on a small sample.
<
p>
When you start arbitrarily adding and subtracting the MOE, as you did, I’m afraid you encourage the impression that these numbers are producing the mythical “statistical dead heat” which you deny with your statement that “neither of those extremes is all that likely.” In fact — taking your spreadsheet outputs as legit — there is only about a 9% chance that this is true.
<
p>
Also, when you say, “the probability that Patrick is ‘really’ ahead of Gabrieli is 91.3% – pretty good, but not 95%” where does the 95% come from? If my candidate has a 91% chance of winning, I don’t think I’m going to rue that he doesn’t have a 95% chance! Now, do I really believe that this represents the situation on the ground is another question. No, I don’t. I think Patrick’s chances are much lower than that. Don’t forget, this is a 3-way race, and the dynamics (not to mention the mathematics) are very complex. You hinted at that.
<
p>
In any case, you’ve made a terrific contribution to the discussion of poll results, and I hope we can build on it going forward. Thanks again for your efforts.
your first point: you’re right – I didn’t state it correctly, and I will update accordingly. What I should have said – and what I believe to be correct – is that there is a 95% chance that “Patrick’s ‘real’ support could be as low as 30.4%, and Gabrieli’s could be as high as 34.6% – though neither of those extremes is all that likely.” Since the MOE on this poll is based on a 95% confidence interval, that’s the right way to say it. Thanks for the note.
<
p>
To your second point, again, the 95% is with respect to the confidence interval reflected in this poll. It’s not exactly arbitrary, since it corresponds to two standard deviations, but it’s a useful benchmark for this sort of discussion.
<
p>
As for the “chances” of winning on Sept. 19, of course all the MOE and statistical discussion in the world can’t account for what happens over the next few weeks. The very best this poll can do is accurately measure what a candidate’s support in the relevant community is NOW, not on Sept. 19 when it matters.
Your last point is key. No matter how accurate a poll is, it only represents voters’ feelings on the day it was taken. Lots can happen over the next 45 days to change people’s minds!
<
p>
As for the 2 sigmas (sigma is the Greek letter usually used by mathematicians in equations to indicate standard deviation), your interpretation needs a little elaboration. I think your first attempt was clear, but your latest statement is less so.
<
p>
Let’s take Deval’s 35%, for example, and your MOE of 4.6%. (You said 4.9% one place, but your ranges were calculated with 4.6%, so I’ll use that.)
<
p>
As you pointed out earlier, you need to know the “confidence interval” in order to translate MOE into sigmas. Here’s an article that isn’t perfect in terms of its explanation, but it does have a nice table showing how the MOE varies by sample size. Where the article doesn’t get it quite right is in talking about the MOE predicting the results of future surveys. That’s out in left field, mathematically speaking. The sample result that we see (e.g. 35% for Patrick) is the “sample mean” or average, and the question is how representative is that of the “population mean” or true average.
<
p>
Without getting into the subtleties of standard error versus standard deviation and the application of all these concepts to differences, etc. (gotta save some fun for another day!), let me try to reinterpret what I think you were trying to get at.
<
p>
You had it right, originally, when you said
but you stumbled when you subsequently said
If you think about it, I think you’ll see why that is wrong. If there is a 95% chance of being within that range, there is only a 5% chance of being outside it. Assuming a normal distribution (as you pointed out, and the Law of Large Numbers allows us to do that), then there is only a 2.5% chance the “true” result could be below 30.4%. In fact, since the MOE represents 2 sigmas, one sigma below 35% is 32.7%, and there is only a 16% chance the “true” result could be below that. You’re beginning to get an insight into how the Kevin Drum calculator works, I hope. You have to figure how much of a probability there is that the two distributions overlap.
<
p>
I think I better stop here before I totally confuse everyone, including myself, but I like this discussion. I think between us (or among us) we’ll come up with a useful way to present and explain these data that is far more sophisticated than the MSM does yet is easy for people to understand. Of course, I also expect to see peace on earth and universal healthcare, so don’t hold your breath!
Tough to state these things accurately.
(but not my earlier comment, since those aren’t updatable). I hope it’s accurate now.
the MOEs for the two Survey USA polls (a couple of weeks ago and a couple of days ago) are different.
Love the statistics, and this was a great explaination. However, this only covers half the problem.
<
p>
IF the people polled are a perfect sample size of the people who vote, then the results and analysis you covered are outstanding.
<
p>
However, it ain’t. How do you choose how to contact? By phone? What about those who don’t have a phone, screen all calls, or only own a cellphone? What about those who are never home during the time you are calling?
<
p>
If these people don’t have a skewing demographic, that’s fine. But, for example, young people are far more likely to only own a cell phone — and more likely to vote Dem — but less likely to vote at all. Do people who work nights (low wage, blue collar, often cleaning or hands-on jobs) tend to vote a particular way?
<
p>
Even exit polling (asking people how they voted as they leave the polling location) data isn’t perfect, because the rate of refusal may not be equal on each side, rate of absentees may not be equal on each side, language barries not equal, rate of errors and spoiled ballots not equal, etc.
<
p>
Sure, the bias might only be a few votes — but if its a 2 or 3 per cent bias, that renders huge structural errors (not random errors) in the poll.
<
p>
Polling is hard. Even if voting was compulsary, you’d still have a tough time measuring what a statistically unbiased sample actually did. Will do is, of course, even tougher.
This discussion is fun, but let’s work out the kinks in “private” and then have the press conference once we have arrived at eternal truth and beauty. We ain’t that far away!
<
p>
Your points are well taken. The best mathematics in the world can’t correct for sloppy polling techniques, and as you point out, even the best attempts to interview a “random” sample are doomed to failure in one degree or another. Pollsters know this, and adjust their numbers accordingly. Thus triggering another dispute: are they skewing their numbers according to their biases (a charge long hurled at the gooper-leaning Gallup organization, e.g.), or is it a “scientific” adjustment? There’s as much art as science, and in the end we can’t even tell which polls are predictive because of the fact that some significant percentage of people don’t make up their minds until the last hours or even not until they enter the voting booth!
<
p>
Still, there’s no excuse for the MSM misunderabusing the numbers that are out there, no matter how suspect they might be.