How can pollsters in the field at the same time, asking similar questions, get answers that vary by this much?
The latest batch of national polls is a mixed lot, as the table below reveals.
There is a six-point spread between Essential's estimate of Labor's primary vote and Newspoll's: 40% to 34%, respectively. The spread in the Coalition's primary vote is smaller, just three points (Essential's 44% to Newpoll's 47%).
The variation in the two-party preferred estimates – once smaller parties' preferences have been redistributed– range from Labor on 46% with Newspoll to 50% with Essential and AMR, both of the latter polls conducted online.
The Essential numbers are an average of two weeks of polling. Essential's previous two-week average (based on surveys fielded between 9-12 August and 1-4 August) had Labor on 49% TPP. This means that Essential's polling conducted in the August 14-18 period must have had Labor on something like 51% TPP.
Morgan's large "multi-mode" survey (spanning face-to-face interviewing, self-completion via the internet and SMS) estimates TPP using respondents' own reports of how their preferences will be allocated. If preferences are allocated as in the 2010 election (the method used by almost every other pollster) then the estimate of Labor's TPP is one point lower, 48% versus 49%, a shift back in the direction of Newspoll and Galaxy, making the Essential estimate look even more anomalous.
What's going on here? How can pollsters in the field at the same time, asking similar questions, get answers that vary by this much?
A certain amount of variation across pollsters is to be expected, due to the uncertainty of random sampling. Suppose pollster A and pollster B field surveys using identical methods. Let's even assume that the method is unbiased, such that both pollsters get the "right answer", averaging over many polls.
Any given poll will deviate from the correct, long-run average, due to randomness inherent in sampling and the size of the sample. For instance, if the ALP's TPP was 50%, then any given poll with a sample size of, say, 1,500, would have a 95% credible interval or "margin of error" (MOE) of +/- 2.5 points.
But what about the differences between these polls? The 95% confidence interval on the difference between the two polls, each with 1,500 respondents, is +/- 3.6 points, considerably larger than the MOE on any single poll. There is a 50% chance that the polls will differ by more than 1.2 percentage points. The probability that the two polls will differ by a percentage point or less is just 42%.
These calculations assume the pollsters are using identical methods. But pollsters vary in their sampling methods (eg random-digit dial v face-to-face v recruitment to internet panels) and interviewing procedures (live interviewer v self-complete internet). There are also subtle differences in question wording and question order, and in the statistical corrections used to minimise non-response bias. All of this makes the polls exhibit more variation relative to one another than that we would expect given their stated sample sizes.
So the spread in this last batch of national polls fact is interesting, but not completely surprising. My poll averaging model helps us deal with a disparate set of polls like this, directly addressing sources of variability over and above that from random sampling.
My poll average has Labor's TPP at 46.8%, +/- 1.4. This estimate is lower than just about all the stated poll results (save for Newspoll's 46%), after correcting for over-estimates of Labor's TPP by the pollsters in the 2010 election (detailed in a previous post).
The graph below shows the recent trajectory of my model's estimates of ALP TPP. There is a hint that perhaps the fall in Labor TPP has ceased, driven in part by the Essential and AMR estimates (50% ALP TPP) and the large, nominal sample size of the Morgan multi-mode poll (49% ALP TPP). But I would be wary of extrapolating trends from the last couple of polls going into my model.
If we wipe the slate clean, forgetting the pollsters' 2010 over-estimates of ALP TPP, we get a slightly more favorable picture for Labor, 48.3% TPP, +/- 1.1.
Even under this latter set of assumptions — giving the polling industry the benefit of the doubt — the Coalition remains in an election-winning position.
