Welcome to the latest in my occasional series highlighting interesting findings from academic research. Today – the problem of ‘uninteresting’ trial data not being published.
This has become a high profile issue in medical research, with the concerns that trial data where a treatment is found to have no impact is much less likely to be published than trial data which does show an impact. The result? The published data may give a very lopsided view of what is really happening.
This is not just an issue for medicine, however, as Donald Green, Mary McGrath and Peter Aronow have pointed out in Field Experiments and the Study of Voter Turnout (Journal of Elections, Public Opinion and Parties, 2013, Vol.23, No.1).
Gathering data on political science field experiments designed to test the impact of different ways of raising turnout (e.g. sending letters, making phone calls, and so on), they found that:
The main challenge is the ‘file-drawer problem’ whereby experiments with null findings go unreported. We have made a concerted effort to compile a comprehensive set of studies, including unpublished studies … As expected, unpublished studies tend to have smaller effect sizes than published studies.