Outlier AP Poll?
The Associated Press Poll today gave Obama a 1% lead over McCain. This was substantially different from other polls released about the same time. And one might think it is merely an outlier.
Polls often report the margins of error, say plus or minus 3% that is typical of a sample size of around 1000 observations. This means that the real value is likely to be within the range of the poll, but we don't know precisely where. And note that here the range is 6 points. So if Obama has a score of 44%, the real value is presumed to be between 41% and 47%. Most political junkies understand this now.
But there is another element of the calculations that is usually not stated in reporting the results. That is the confidence level. All of these reported levels of support for a candidate have a measure of likelihood that the true value lies within the plus or minus range reported. And the most common confidence level is 95%. Thus the proper way to read a poll that gives Obama a 44% rating is that "we can be 95% confident that the true rating is between 41% and 47%." Notice that 95% is 19 times out of 20. Thus one time out of twenty will not even be within the confidence interval: it will be outside the confidence interval.
Next question, how do we know which polls are these (extreme) outliers? Answer: we don't. And that is precisely why measuring a large number of polls is an important exercise. Because some of them will be outliers. Averaging out a number helps to control for this, which is exactly what TPM does.
Finally, there have been dozens and dozens of polls; hundreds in fact. When there are so many polls it would be surprising if there were no (extreme) outliers. And there should also be a few polls that overstate Obama's lead. Again, it would be surprising if there were not.
Polls often report the margins of error, say plus or minus 3% that is typical of a sample size of around 1000 observations. This means that the real value is likely to be within the range of the poll, but we don't know precisely where. And note that here the range is 6 points. So if Obama has a score of 44%, the real value is presumed to be between 41% and 47%. Most political junkies understand this now.
But there is another element of the calculations that is usually not stated in reporting the results. That is the confidence level. All of these reported levels of support for a candidate have a measure of likelihood that the true value lies within the plus or minus range reported. And the most common confidence level is 95%. Thus the proper way to read a poll that gives Obama a 44% rating is that "we can be 95% confident that the true rating is between 41% and 47%." Notice that 95% is 19 times out of 20. Thus one time out of twenty will not even be within the confidence interval: it will be outside the confidence interval.
Next question, how do we know which polls are these (extreme) outliers? Answer: we don't. And that is precisely why measuring a large number of polls is an important exercise. Because some of them will be outliers. Averaging out a number helps to control for this, which is exactly what TPM does.
Finally, there have been dozens and dozens of polls; hundreds in fact. When there are so many polls it would be surprising if there were no (extreme) outliers. And there should also be a few polls that overstate Obama's lead. Again, it would be surprising if there were not.
