Nick Sparrow, former head of polling for ICM and one of the country’s best pollsters, has some real doubts:
Over the last 3 years the British Population Survey has been monitoring people who respond to online surveys and comparing them to the population as a whole, in terms of detailed demographics and attitudinal variables. It is a massive survey involving 6-8,000 face-to-face in home interviews per month.
In an article published on the Research-Live web site Steve Abbott describes some of the important findings. Analysis suggests that online survey respondents are more active in the broadest sense, than others, more likely to vote, have stronger opinions than others, are more optimistic and more volatile.
Such respondents are just what any journalist commissioning a poll would want, giving results that show people have strong opinions, suggest big movements in public opinion, producing surprising and therefore newsworthy results. And they are cheap as chips. As a result, they are everywhere. So much so that when we talk about “the polls” we mean, substantially, “Online polls”. But all online responders account for no more than 10% of the population, perhaps the most influential minority in Britain today suggests Abbott.
That’s important for political polling, but even more important for much non-political polling because with political polling there are at least the regular real election results which help (and force) good pollsters to calibrate their results against reality.This calibration does not solve everything, for getting it right on polling day may still mean you were getting it wrong earlier (such as by being far more volatile than reality). And having a system that would have got it right at the last election is no guarantee that it is still calibrated right for the next.
But at least there is the occasional check-in with reality.
The big problem for much non-political polling is that it doesn’t come with such reality check-ins. Given how much adjustment to raw data is need to make political polls match reality, that should be a big worry for both online and offline polling. Add in the doubts that Nick Sparrow highlights and it means great caution should be taken over how to use and present online polling data, especially where the margins it shows are slim.