a Ron Paul supporter, you probably have read a few comments, blog
posts and articles attempting to justify Ron’s poor performance
in polls. Some make sense and others are plainly excuses. I’m not
certain theories about cell-phone-only households are good enough
to explain Dr. Paul’s poor showing in these polls though it may
be valid for a certain demographic (18–24 year olds).
One of the
explanations I have read is that pollsters are selecting responses
which only include loyal republicans and democrats who are likely
to vote. Because they are ruling out other possibilities, they are
not able to measure what is really happening amongst the general
For the most
part, poll respondents are discovered by dialing numbers in a phone
directory (listed numbers). The pollsters continue calling until
they get enough respondents who match the criteria they’ve selected
(x republicans who answer they are likely to vote –
x democrats who answer they are likely to vote).
Out of curiosity,
I decided to do some googling in order to get a better idea of the
mechanics of polling. I’m certainly not going to become an expert
on polling using google but there are things that I’d like to know.
standard directory listings, pollsters also use a method called
Random Digit Dialing. This attempts to reach people who have unlisted
numbers and won’t be found in the directory listings. Some people
have theorized that people with unlisted phone numbers might offer
significantly different results but according to a Zogby
study on RDD, advances in technology (CallerID) and changing demographics
(unlisted numbers aren’t just for the affluent anymore), people
with unlisted phone numbers are less likely to participate then
they were 20 years ago.
I started in 1984, response rates averaged 65%,” Zogby says.
“You were twice as likely to get someone to answer the telephone.
Today they don’t answer the phone and they are much more
likely to refuse to participate in a survey.”
Zogby has concluded
that RDD doesn’t offer a significantly better sample and so doesn’t
use RDD. However, the Zogby study conclusions talk more about costs
and "interviewer fatigue" as a rationale then it does
the actual polling results. Not being an expert, I merely note this
because it is interesting. Zogby admits that academic studies haven’t
found RDD any more or less reliable then the old phone book method.
In the academic
world, no consensus has been reached among public opinion researchers
on the superiority of RDD over listed samples. However, the results
from a number of studies are consistent with Zogby’s assessment
that listed samples are more efficient, less costly to administer,
and produce results similar to those of RDD.
does say that in measuring differences between the two polling techniques,
there are moderate differences in responses 35% of the time and
significant differences 2–3% of the time. Given the margin
of error, one might conclude that RDD may produce more accurate
results. But at what cost? Zogby thinks it’s too expensive for the
A poll cannot
predict anything by definition. It only attempts to get the current
opinions of the people contacted. It can’t read the minds of those
contacted and it cannot contact the same people intentionally (to
track trends) lest it invalidate the randomness of the samples used.
alleged randomness. For some reason, the media and politicians seem
quite obsessed with the opinions of very small numbers of Americans
extrapolated to conclude that therefore the general population would
react similarly by percentage. I’m not convinced. As much as the
media is liable to promote a point of view using polls as "evidence,"
I am suspicious that such reliance is because polls are so easily
who follows the gun debate remembers Arthur Kellermann and Donald
Reay’s infamous epidemiological farce which concluded (on the first
pass) that people who keep guns in the home are 43 times more likely
to have a family member killed by a gun than those who do not keep
a gun in the home. This number kept getting revised downward as
time went on and as more and more reputable reviewers poked holes
in the data until the study was altogether discredited. The basis
of the study, which they claimed properly ruled out bias, was homes
of people where a gun homicide had occurred. Apparently it didn’t
occur to either Kellerman or Reay that study "participants"
were more likely to be endangered due to their criminal behavior
then they were by idle, inanimate objects. Because the sample used
was weighted so heavily with people involved in crime (prior felonies)
the sample was biased in the extreme.
So why isn’t
it considered bias when respondents are limited to those who are
likely democrat or republican registered voters? There are certainly
states which allow crossover primary voting though not all states
do. So, in those states where it is allowed, why not include responses
from members of "third" parties?
not an expert in polling, but I am familiar with testing methodologies.
I’ve been testing software for over 20 years. I have seen some very
interesting and brilliant uses for random sampling that proved extremely
useful. However, people are not like software or hardware. When
applying random sampling techniques in simulations, the data being
sampled is finite and well-defined in its characteristics. Humans
are far more complex and infinitely less predictable than the opinion
makers would have you believe.
results we are seeing also doesn’t account for claims made by RP
supporters and noted by observers. These fall into several distinct
- Young voters
who were not old enough to have voted in the prior election cycle.
- Young voters
who could have registered in the last cycle but did not find a
candidate attractive enough to bother.
- Older voters
who had given up on the electoral process.
voters who aren’t always loyal to a specific "third party"
voters who have shown loyalty to a particular third party but
will not be loyal to that party this election cycle.
who plan on switching parties or will not be voting for a Democratic
candidate for President.
Newsweek, Bloomberg et al., do not include these categories.
If they show up, they eliminate these from their reported results
in order to keep their samples "pure." For "second
tier" candidates including Ron Paul, Tom Tancredo, Mike Gravel
and Denis Kucinich – who offer at least some significant variations
on the issues from their "first tier" counterparts –
the omission of this data could be significant. It may even be significant
enough to skew the numbers unrealistically toward the "top
tier" candidates since substantial percentages of respondents
may not be considered.
that may be significant is the formulation of the questions. The
questions do not attempt to discover any rationale for respondents
choices but only the "seriousness" of the respondent (are
you not sure, somewhat sure, pretty sure, extremely sure, super
Are those sampled
choosing an answer based on name recognition alone or based upon
a specific issue which they agree with the candidate’s view? Can
the respondent even list the positions of the candidate they have
chosen? Since polling questions as to motive are saved for another
random sample, we don’t have the answers to such questions by the
original respondents. It is pure conjecture to attempt any linking
of motives to the original responses. (e.g., people are angry with
Bush because the war isn’t going well) I am reminded of daily news
reports which attempt to tell us why the market went up or down
in spite of the fact that there are often news reports which contradict
those reasons. Yesterday, Yahoo claimed that the market went up
on retail sales reports. On the same page, there were two other
news headlines, one which said that Wal-Mart reported a 9% increase
in sales last quarter, the other claimed that there was a general
retail sales slump for the past quarter. While not equivalent, the
claim by Yahoo that markets went up on reports of stronger retails
sales is as reliable as claiming Ron Paul is getting 0–3% "real
It is also
extremely early in the cycle to put any stock in polls that have
been conducted thus far. As mentioned before, polls cannot be used
to offer predictions. The only reliable political polls are those
which include openly counted paper ballots.
One other important
factor is the intentional omission of certain candidates. In spite
of the fact that Ron Paul has surpassed the one-time front-runner
McCain in cash on hand, neither Newsweek nor LA Times/Bloomberg
include Paul in its poll.
poll results as a means to prove or disprove a candidate’s chances
next fall is revealing an agenda. The polling data that you aren’t
seeing would prove interesting I think. Candidates often hire polling
firms to help them measure the results of certain policy statements.
Clinton was notorious for commissioning polls to see which way the
wind is blowing.
If polls were
so good at predicting voter opinions, what happened with the recent
fiasco over immigration? Both parties had to run for cover and the
amnesty bill was abandoned. Wouldn’t it be interesting to see privately
commissioned polling data immediately after "The Decider"
got decisive and demanded an amnesty bill?
I don’t intend
to completely dismiss polling data. However, the growth of third-party
membership and the decline of those participating in both telephone
surveys and voting in elections suggest that the pickings are getting
slimmer and slimmer for pollsters. To put any faith in them 16 months
prior to the general election is unrealistic. To use them as a means
of predicting the outcome of the general election? Completely absurd.
Fisk [send him email] is
a 44-year-old software developer and entrepreneur. He is married,
has 3 children and resides in Austin, TX.