Media & PR

Can you trust the opinion polls?

In an earlier blog I highlighted the dangers of focusing on a single data point – as shown by voting intention polls for the general election, where a swing one way is usually followed by a swing back the other.

But there is a more fundamental problem: can you really trust poll results at all? Take the ITV Leaders’ debate; according to the polling, twice as many people said they watched all the debate as really did.

That is an embarrassingly large error for something that should be easy to poll. This wasn’t a poll about whether people watched porn 25 years ago, but a poll about something done recently and with little personal embarrassment in answering either way. Despite these polling advantages, the poll missed the methodological equivalent of an open goal.

Yet, for all the brickbats thrown at political polls, they have one big advantage. Election results provide a regular (if not that frequent) test of their methodology. This testing against reality (including the notorious polling failures in the 1992 general election) has led to the countless adjustments pollsters now make to turn their raw figures into a plausible attempt to reflect the real world accurately.

Political pollsters have found they need to do far more than simply weight for basic demographics in order to ensure their poll results match up with reality. Despite these lessons, relying on canvassing opinion of people from the right sample and weighting by demography is still the bread and butter of much other polling.

Which puts the question “How accurate is digital listening?” – the tapping into the views expressed on social media – into a very different light. Yes, social media is not a carefully assembled representative sample of the population, but when you consider credible pollsters who take care to assemble the right quality of sample often end up wide of the mark, is that question the right one to ask?

Better instead to acknowledge with open eyes that you have to be careful about how representative online views are. Apply that same scepticism to polling. In both cases ensure that the results are benchmarked against alternative sources of information so the right levels of adjustment can be made. Alternatively, where such benchmarks are not available, look at trends and their causes, rather than obsessing about current levels.

Your sample may not be fully accurate, but if interest in health implications of a product shoots up by 15 percentage points, that is something useful to know and wise to act on.

Leave a Reply

Your email address will not be published. Required fields are marked *

All comments and data you submit with them will be handled in line with the privacy and moderation policies.