The other thing that went wrong with Liberal Democrat polling

In my piece What went wrong with the Liberal Democrat polling and key seat intelligence? (now slightly updated) I said:

I suspect … that the Lib Dem polls accurately caught how people would have voted had the election result been clear in advance (as with 1997, 2001 and 2005) and had voters marked their ballot papers thinking about who they wanted as their MP rather than who they wanted as PM.

This touches on a wider issue not only with Liberal Democrat polling, or political polling, but polling in general.

It’s the question of how good people really are at knowing how they are going to behave or giving accurate answers about what influences their behaviour.

As I wrote in It’s not just voting intention polls which should be in the spotlight:

The problems are best exemplified by the good looks of candidates. Experimental tests show that attractive candidates get more support (such as when people are giving mocked up information about candidates with just the photo changed). Yet do people go round telling pollsters they vote for a party depending on how attractive its leader is? They don’t.

Maybe they secretly know they’re not quite telling the truth. Or maybe they just aren’t that good at understanding exactly what motivates them to make their decisions.

The Financial Times has extended this point in a very good look back at the general election polling by quoting one of market research’s regular critics:

In his book Consumer.ology: The Truth about Consumers and the Psychology of Shopping, Graves writes: “The fundamental tenet of market research is that you can ask people questions and that what they tell you in response will be true. And yet . . . this is a largely baseless belief.” He says there is a simple, clear, obvious reason for this. It is that: “The unconscious mind is the real driver of consumer behaviour.” In other words, there’s no point in asking people what they want, or what they’re going to do, because they don’t know themselves…

“We are lousy witnesses to ourselves,” he says. “If you look at people’s inability to be empirical . . . they’ll go out and buy another exercise machine. They’ll join another gym. They’ll buy another smoothie maker. They won’t look in their cupboard and think, ‘I’ve got 16 household appliances that I don’t use.’ And then there are short-term contradictions. I watch people walk out of a pound store and go into a Starbucks. Clearly they’re meeting different needs, satisfying different psychological drives. But it makes no sense in any kind of narrative about people doing things for good reasons.”

Or in the context of the 2015 general election, when asked how people were going to vote in their constituency, voters turned out to think they’d worry much more about who their MP was going to be than turned out to be the case.

The failure – with hindsight* – was not to factor in the risk that one of the pieces of the basic framework for the polling would be nullified by events and no longer be helpful.

No amount of refinements of weighting, improvements in sampling or tinkering with question order, in other words, is going to remove the risk of a future similar polling debacle where the framework within which voters are thinking about an election changes and the results therefore no longer reflect reality.

Or in other words, high quality polling always needs regularly testing against its set of assumptions especially when you’re trying to predict the future.

 

* Despite the many and varied commentary on constituency polling at the time, I can’t recall anyone who criticised it before polling day on the basis that with a close election it was testing the wrong thing. Do correct me if some sage has slipped from my memory.