Last month a story did the rounds about how a poll shows that if Labour, Greens and the Liberal Democrats did a deal to stand only one candidate against the Conservatives, Boris Johnson would be out of power.
But how good was that poll and what conclusions can really be drawn from it? Answering that question is important in its own right, but I’m also going to show the steps to the answer – showing how to approach evaluating an opinion poll finding that is in the news. (More on all that in my book, Bad News: what the headlines don’t tell us.)
The poll was conducted Find Out Now. They are a member of the British Polling Council, the self-regulatory body for British political poling and the full data tables are available online. So far, so good.
The right methodology for the task
The poll was an MRP poll, which allows constituency level results to be extrapolated from large national samples. That’s a good methodology to use to answer questions like ‘how would a seat pact work?’ as you need to get into how results might vary between different types of seats.
But MRP polls should come with a big caveat. Basic opinion polling is a fairly widespread skill, with plenty of experts and long track records. MRP is not. MRP methodology is complicated, with only a small number of real experts and with a limited track record. It is also much more of a ‘black box’ than more traditional polling.
All of this means the only real judgement of how good a particular team’s MRP model is comes when you have actual results to compare them against.
Pollster track record
Find Out Now teamed up with Electoral Calculus for this MRP poll and there isn’t much of an MRP track record for them. The 2019 general election prediction from Electoral Calculus was respectable, but it wasn’t a pure MRP prediction. Rather, it included other data, such as polling averages.
There was a MRP poll for the 2021 local elections but its accuracy is far from clear. The seat predictions were way off, e.g. predicting Labour to gain 280 seats when in reality Labour lost 326. However, the seat numbers were all rather strange and didn’t seem to even add up to the right total number of seats (e.g. see the replies in this Twitter thread). Electoral Calculus’s own published assessment of this prediction simply leaves out the seat numbers and only talks about council control and vote share. On those two measures, it did well. But on seat numbers, something went wrong.
The actual questions asked
Turning to the poll specifically, it was based on working out a possible set of seat deals, picking one of Labour, Green and Lib Dem to stand in each seat and asking people in England and Wales (though not Scotland) how they would vote. People were given the choice of candidates appropriate for their own seat:
Q2. Suppose at the next general election that all the usual political parties are standing in your seat except that the [OTHER PARTY1] and [OTHER PARTY2] have agreed not to stand and are asking their supporters to vote [SELECTED PARTY]. Which party, if any, would you vote for, in this general election?
[SELECTED PARTY, OTHER PARTY1, OTHER PARTY2 are “Labour”, “Liberal Democrats”, “Green party” depending on respondent’s seat]
The results were then fed into an MRP calculation to provide seat numbers. These were compared with seat numbers without a pact, apparently showing a big boost for having such a pact with Labour wining 36 seats more with a pact, the Lib Dems winning 14 more and the Greens 8 more. Those 58 gains remove the Conservative majority.
But the with/without pact comparison was not a like for like one.
The poll didn’t ask a normal voting intention question and then a pact question. Rather, it just asked the pact question and compared the results from that with the results from a poll of polls, looking at standard national voting intentions.
So although the MRP pact poll finds a difference from traditional voting intention polls, we don’t know how much of that difference is due to the former being an MRP poll and how much due to the theoretical pact. If both MRP and other polls are perfect (or equally in error!) then that doesn’t matter. But we’ve no way of knowing if that is the case, which means the supposed impact of the seat pact is left uncertain.
It’s a real shame that the MRP poll didn’t ask both voting intention questions and so give a much better like-for-like comparison.
Theory and reality
However, even if it had, there’s still a problem about the question asking voters a hypothetical: if X happens, what will you do? There’s a risk that people may not predict their behaviour accurately. There’s also a risk that by just asking about X a poll question focuses people’s attention on it much more than you would get in reality.
Take the YouGov poll from 2016 which I reported as ‘New poll shows Lib Dems could overtake Labour, Ukip if politics polarises over Brexit’. We got an election in 2019 that was rather like some of the scenarios in that poll but a result that was very much not like its findings. The poll was right in the sense of showing that Brexit might fuel a powerful Lib Dem performance. Might rather than would because other things can – and did – happen.
So even if the MRP poll on seat pacts was perfect, it would still suffer from this forecasting problem.
Those seat deals in detail
Before concluding what we can learn from this poll, there’s another detail that needs examining. The poll was based on a theoretical set of seat deals. Some of those deals (see Appendix 2 in the report) were very unusual.
Take Lewes, where the result in 2019 was:
Liberal Democrat 43%
The deal the poll was based on? Labour and the Liberal Democrats standing down for the Greens in this seat (!). Which highlights a problem: there are no Conservative-held seats in which the Greens are in a strong second place as the obvious challenger. So any sort of seat deal would be very difficult to agree, as it either involves other parties standing aside where they can legitimately think they have a better chance than the Greens, or it involves the Greens standing aside without getting much in return.
The Conservatives won’t do nothing
The Lewes example also shows one other risk. At 48% the Conservatives were not far short of an absolute majority in the seat anyway.
In practice, what would happen with a seat deal? The Conservatives would use it as a political weapon to win over support – e.g. by squeezing party X’s old support by saying a vote for X is really a vote for Y, and also be using it as a motivator for its own supporters – ‘you’ve got to go vote to stop the other parties conspiring against us’.
A poll about a theory can’t capture well the reality of the counter-reaction if theory were to become reality.
What can be concluded?
Where does this all leave us? The polls gives a clue that Green, Labour and Liberal Democrat supporters may be willing to switch to defeat the Conservatives.
It’s a tentative clue as it doesn’t give a clear like-for-like comparison between seat deal and no seat deal. It’s also a tentative clue as it doesn’t capture the political dynamics and difficulties that would be involved in any such deal. Nor did it include Scotland.
So it’s a clue that suggest there’s a willingness to switch. But it doesn’t go so far as to tell us the best way to make that sort of switching happen, i.e. it doesn’t tell us enough to judge the relative merits of formal seat deals versus a more informal targeting of different campaigning and encouragement of tactical voting, 1990s-style.
A piece of evidence, then, rather than an argument clincher.