The Observer shows the media’s problem with reporting voting intention opinion polls
A headline in The Observer claims:
Success of vaccine rollout pushes Tories ahead of Labour in the polls
In that one short, confident assertion, the headline both gets the truth wrong and shows how the media struggles to report voting intention polls properly.
Before dissecting the headline, it is worth noting (again) that headlines are not usually written by the journalist in the by-line.
So what’s the problem with the headline?
It asserts that the Conservatives have moved ahead of Labour in the polls, and that this is a new thing. Hence “pushes Tories ahead”. Yet the story is based on only one opinion poll. Every guide I have ever seen for journalists on how to report polls or for the public on how to understand polls hammers home the same point: don’t draw definite conclusions from just one polls. That’s because the nature of the sampling used by polls means there’s a bit of random variation or noise in the figures. Polling figures will bounce up and down even when the underlying reality is unchanged.
Why does the story rest on just the one poll? Because it’s the one poll The Observer has just paid for. That commercial reality – media outlet pays for poll and therefore wants to make a news story out of it – repeatedly lures media outlets into poor reporting of polls by producing reports based on just their own latest poll.
What is more, if The Observer had done the sensible thing – look at more than one poll – it would have found a different story. We’ve had six different polling firms conduct polls between 21 and 29 January. The changes they found in Conservative support from their previous poll were +4, +2, +2, +1, -2, -2. That’s on average an increase of less than one point in the Conservative support. Given the errors involved with polls, that’s far too small a number to be able to say Conservative support has gone up.
Ah, you may be thinking, but the Conservatives could be moving ahead of Labour by staying still while Labour falls back. Here are the six Labour figures: +3, +1, 0, 0, -1, -3. Dead on zero average.
The reality is consistent across all these polls: Labour and the Conservative support is both roughly steady, with Conservatives on average a point or two ahead of Labour. Individual polls bounce around, producing apparent headline-worthy shifts… but only if you skip past the other polls when writing that headline.
There is a an additional possible mistake that headline makes, which is to assume that even when there is a real movement in the polls it is down to whatever political journalists were writing headlines about in the meantime. What more in-depth research repeatedly shows is that public opinion usually sails past stories, barely noticing them and not being moved by them. Even stories that have dominated the headlines for days.
In this case it is, to be fair, plausible that the vaccine rollout may be causing a change in political views. It is after all something that is different affecting several million people each week. But to be confident that it is changing political views, you need rather more evidence that the one poll or even guessing from the headline figures on a series of polls. This is where focus groups are so useful. In this case, there is a question cited about how well or badly the government is seen as handling the vaccine roll out so there’s some circumstantial evidence for the guess of the cause of a (non-existent) trend, but it’s still a guess based on thin evidence.
All of which means the more accurate headline would have been:
Tories push ahead of Labour in our opinion poll although multiple other polls show a different story suggesting that the shifts shown in our poll is a bit of an outlier and even if our poll is really the start of a new trend we’ve not done other research to tell the cause so we’re going to take a punt that it’s to do with vaccines and cross our fingers.
A bit long, I’ll grant you.
Which in its slightly cumbersome way illustrates the problem at the heart of reporting of voting intention polls. The norms of journalism and the commercial pressures on media outlets mean individual polls get treated as credible sources from which firm conclusions can be made. Yet the moment anyone steps back from writing the next story about their own poll, everyone knows that you should no more rely in one opinion poll being right than you should rely on one uncorroborated source for a non-polling story.
Good journalists know this, which is why The Observer‘s piece, after telling a story about how something new has supposedly happened (Conservatives moving ahead of Labour) has this buried deep down, giving a clue that the whole premise of the headline is wrong:
By August the two main parties were neck and neck. Since then the lead has changed regularly between the two parties.
Which means the headline could have been:
New poll shows same pattern as for last few months, but please read on despite that sounding boring
Because of course the other factor driving this reporting problem for polls is all the rest of us. Over-egging what an individual poll says doesn’t harm media outlets, it boosts their audience.
We give them the wrong incentives. That’s on us, not them.
To lean how to spot problems with how opinion polls are reported, see my books, Bad News: what the headlines don’t tell us and Polling UnPacked.
What is the purpose of a headline, even “The Observer shows the media’s problem with reporting voting intention opinion polls”? It is to persuade you to read the article, as opposed to simply turning the page. If we keep turning the page, then next time we don’t buy the newspaper.
So, how is “We give them the wrong incentives. That’s on us, not them.” true? The headline is just advertising, but it is advertising by the (sub-)editor rather than an external company that has paid for space. This is why whenever I read, say, The Guardian I look first for who has written the article: if I see it is by John Harris then I know that it will be well-researched and informative, even when I disagree with his conclusions; on the other hand I know that anything from Simon Jenkins will be ill-informed and bilious and which will give me nothing new even if, coincidentally, his postion supports mine. Unfortunately, when buying The Guardian, I am forced to take a job-lot of the good and bad mixed together and this would be true of any other newspaper.
I have argued for many years that what we need is for newspapers to act as middle-men for journalists’ output, with a principal function of enabling journalists to be paid for each article they write according to how popular they are with their readers. Micropayments work well for mobile phones, why should they not work work just as well for newspaper articles. Even a system that allowed the reader to read, say, the first ten lines for free and then opt to pay for the whole article would be an improvement on the present system and remove the need for click-bait headlines. Yet this is entirely within the power of newspaper owners to do already as an alternative to subscriptions and paywalls.
I think most people see the discrepancy between the headline and the story, even if it is only subconsciously, and that is just one reason newspapers are disbelieved.
TV is not immune I recall a BBC reporter saying people are running scared here with bombs all over – behind him a couple were nonchalantly walking hand in hand.
Perhaps a more accurate (and more depressing) headline would be: “Despite everything, the voters still trust the Tories”.
That also nicely encapsulates the problem for us. With perhaps the worst Prime Minister since Lord North, how do we shift public opinion to realise that the Tories have nothing to offer the ordinary person, whilst Labour have yet to come up with anything new?
How come the YouGov poll announcement gave Labour a 4 point lead over the Conservatives for Jan 29th, yet not mentioned in these analysis?
Michael – it’s in the list of recent polls for which I give vote share changes. As you say, it’s a good example of how there isn’t a simple pattern of ‘Conservatives move ahead of Labour’ in the polls.
1) if my memory serves me, previous Observer polls have tended to show Labour in the lead and certainly higher than in other outlet’s polls. So, taking this sequence as the basis, a Conservative lead of 4 is definitely pushing ahead.
2) On the other hand, the Observer, even more than its weekday sister, is very keen across its coverage to show how Starmer is a huge improvement on the disastrous Corbyn and that Corbyn’s personality and policies were the main cause of the election defeat. It fits in that it should seek to explain away a Starmer/Labour setback as being a special phenomenon outside his control.
Jim: you’re right that Opinium has slightly higher figures for Labour typically. Just over 1 point higher on average in the last three months. But there’s nothing particularly out of kilter with previous Opinium polls in giving the Conservatives a four point lead. It is a little on the high side, yes, but the typical fluctuation we’d expect between polls will throw up leads a little on the high side now and again by random chance. We can’t conclude with any certainty that this is anything other than random chance signifying nothing. The headline, however, paints a picture of absolutely certainty that therefore goes well beyond what the evidence supports.