What the evidence says: did 38 Degrees really find a way to boost turnout at elections?
Welcome to the latest in my occasional series highlighting interesting findings from academic research. Today – how did the efforts of 38 Degrees to raise turnout at the general election fare?
I have rather a love-hate relationship with 38 Degrees, normally settling on ‘meh’, partly because of their general silence on issues of civil and political reform. They’ve been largely silent on topics from same-sex marriage through to House of Lords reform. For me at least that sits oddly with their founding idea of marshalling a progressive majority of people in the UK. ‘Progressive’ is open to many definitions, but to do so in a way that excludes such a broad range of civil and political reforms is, well, disappointing.
The other reason is doubts over its effectiveness, especially the apparent prioritisation of quantity at the expense of impact and the way it often refers to campaigns as if it was the only participant, pushing aside the contributions of others (note the absence of reference to the work of anyone else in this email for example).
But the general idea of giving large numbers of people a way to influence political outcomes is definitely a good one. So my initial reaction to a recent email from 38 Degrees was very positive:
I’ve found out something exciting, and want to tell you straight away. A leading academic has just sent me a report proving that 38 Degrees-ers made a difference at the General Election. It’s been scientifically-verified that together we made thousands more people vote…
When the snap election was announced back in April, over 200,000 of us voted on what to do about it. Together, we decided to launch the most ambitious voter turnout experiment ever – to persuade everyone to use their vote at the election.
With just four weeks to pull it off, we had to move fast. Once we’d decided to run our campaign in two marginal constituencies (Bath and Hove), 38 Degrees-ers sprung into action to sign up hundreds of volunteers. We also brought onboard leading academic, Professor Peter John, to run the experiment and measure whether or not it worked.
Everyday in the run-up to the election, hundreds of 38 Degrees-ers pounded the pavements. In total, together we knocked on 31,600 doors, delivered 63,185 leaflets and called 7,546 voters. All to persuade people that no matter who they vote for, their vote matters…
Since the election, the Professor has been working hard to crunch the data. The results have just come in. In Hove, we increased voter turnout by 1-2%!
Professor Peter John says: ‘Here’s what it means. Elections can be decided by just a handful of seats. We’ve done the maths and 2% of a constituency is roughly two thousand people. Right now, there are 52 constituencies across the UK whose MP was decided by less than a 2% margin – that’s powerful.’
Mark, that’s democracy at its best – knowing that your vote can make a difference, and going out and using it. And now we have scientific proof that 38 Degrees members helped make democracy in the UK stronger.
I couldn’t be prouder of what we did together. Having the report sitting on my desk as I write this email reminds me of the power of 38 Degrees…
Pretty impressive, hey?
Except, what happened to Bath? This story of unalloyed success starts off saying 38 Degrees was active in both Bath and Hove yet the email then only talks about Hove.
Well, the full report does tell the full story, not only of Bath but also of Hove. It’s also a rather different story, as Julian Huppert has pointed out:
I think facts & evidence matter – so was disappointed by an email from @38_degrees @davidbabbs yesterday. It claimed their report meant 'It's been scientifically-verified that together we made thousands more people vote'. This is not true. (Report at https://t.co/AipQZcu84g (1/9)
— Julian Huppert (@julianhuppert) December 22, 2017
I welcome the approach – a Randomised Control Trial (RCT) of campaign effectiveness is really welcome. I applaud them for that. They had 5 different treatment groups and a control, randomly allocated, in two (arbitrarily chosen – a concern) constituencies. So far so good. (2/9)
— Julian Huppert (@julianhuppert) December 22, 2017
In Hove, with proper corrections for multiple hypothesis testing, only one of the 5 treatment groups has an increase in turn out. Interestingly, it is phone calling, nor door knocking or leaflets, that increases turnout. The others had no statistically significant effect (3/9)
— Julian Huppert (@julianhuppert) December 22, 2017
In Bath, using exactly the same statistical approach, they also found that only phone calling had a statistically significant effect – other approaches didn't. However, it apparently DECREASED turnout – by slightly more than it increased it in Hove. (4/9)
— Julian Huppert (@julianhuppert) December 22, 2017
So it is certainly not the case that 'It's been scientifically-verified that together we (@38_degrees) made thousands more people vote' as claimed. In fact, the efforts led to a slightly higher degree of voter suppression in one constituency, than it increased it in another (5/9)
— Julian Huppert (@julianhuppert) December 22, 2017
Indeed, the technical bit in the report says 'Even in Hove it is not possible to categorically reject the hypothesis that the campaign overall had no effect, except for telephoning.' It says of Bath 'T2 (telephone only-JH) has a statistically signicant negative effect.' (6/9)
— Julian Huppert (@julianhuppert) December 22, 2017
I can understand @38_degrees wanted the outcome to be positive, and so felt pressurised to spin the results. This is a shame – especially given that it started with such a good approach. The best conclusion is that phoning mattered, and could be either positive or negative. (7/9)
— Julian Huppert (@julianhuppert) December 22, 2017
This matters. It's really important we decide to do or not do things based on whether they work, not by the need of people or organisations to tell their supporters they are great and to raise money. I hope @38_degrees will do more RCTs – and report the outcomes accurately. (8/9)
— Julian Huppert (@julianhuppert) December 22, 2017
I want things like this to work. I like the idea that citizens simply encouraging people to vote, via door knocking, leaflets, or phone calls increases turnout. I'd like high turnouts. But we should ensure we do the things that work, not the ones that, surprisingly. don't. (9/9)
— Julian Huppert (@julianhuppert) December 22, 2017
As the research report itself says of Bath, “The table [of data for Bath] shows no effect of treatments [i.e. 38 Degrees campaign activity] compared to control [i.e. those voters who were not subject to 38 Degrees activity], and they may even have had a negative impact. In some cases, voters can receive information that causes them not to be engaged or conflicts with other prior-held views. However, the datasets in Bath here are more consistent with a view that the campaign had no effect.”
At best, therefore, the results really are a score draw: a non-trivial but modest result in one case and no impact in the other case. If you go back and read that 30 Degrees email again, do you think it fairly reports back to members on what the impact of their time, money and support really had?
Here’s the full report, which to their credit 38 Degrees published:
You can read the other posts in the Evidence-based campaigning: what the academic research says series here.
The really significant sentence in that email is this: “Elections can be decided by just a handful of seats.” Trying to increase voter turnout when in most cases the voters know in advance who is going to win is putting the cart before the horse.
There was one group that clearly had considerable success in increasing voter turnout at the last election – and that group was Momentum. Of course, Momentum was only seeking to increase turnout for the Labour Party, but I wonder if they could be applied to increasing voter turnout in a neutral way.
For example, for almost every transaction that you make on the internet nowadays, you are invited to share it with your friends through Facebook, Twitter etc. Perhaps this feature could be added to the Government website for registering to vote.
This underlines the concern I have about the fashion for saying (see various Paddy Ashdown remarks) that traditional political parties are rigid and outdated and we all need to organise like Avaaz, 38 Degrees or Macron’s lot (they could mention the Arab Spring movements, but, er, they lacked staying power). I can sign a 38 Degrees petition or donate to an Avaaz campaign, but I have no idea how these set-ups are organised, who takes decisions or how I could find out. The typical activist is told what news the organisers want to tell him/her and checking it is much harder than with a traditional “hierarchical” (= democratic and responsible) organisation.