The IFS answers… Is the data accurate enough for its calculations?

Yesterday I blogged the first of a set of answers from the IFS about its calculations and the data behind them. Those answers nicely lead on to the next question:

In looking at the impact of the Budget or the Spending Review, the IFS typically quotes the impact on different groups of people to one decimal place. What’s your reason for thinking that your calculations are based on sufficiently accurate data and assumptions to justify reporting them to the nearest tenth of one percent (rather than, for example, the nearest percent)?

Our policy is to publish the exact figures produced by our model, either to the nearest penny (if we are reporting cash amounts) or to the nearest tenth of a percentage point. The Treasury takes a similar approach in its distributional analysis. Obviously our calculations are unlikely to be exactly right all of the time, as they are estimates based on a particular sample of households, and often using some assumptions, but it is not clear what sort of margin of error we should allow.

Two points strike my about this answer from the IFS. First, is a numerical stylistic one. In other fields of study it is the normal practice to only report results to as many decimal places (or significant figures) as the precision of the calculations and data suggests is reliable. That’s one reason why, for example, although opinion poll companies could report their polling data to one or even more decimal places, they round them off to the nearest whole number. Given precision beyond what the data and calculations justify risks misleading and the IFS reports I have seen look to me to be rather light on warnings about reading too much into the apparent accuracy of the calculations.

Second, and more importantly, when added together with yesterday’s answers the gist of what the IFS is saying is ‘Our data and methodology is not perfect and has room for error and interpretation. But it is as good as it can be and anyway uses the same data and methodology as the government’. The latter is a good political debating point – it’s a bit rich for someone to criticise the IFS if other times they are using the same approach etc. – but it doesn’t really help us understand what the actual impact of government policies is likely to be.

Because one answer could be “Sorry, the balance of impact of the government’s policy between different groups just isn’t big enough for us to be able to say with any real certainty whether or not the policy is progressive”. Saying “I don’t know” may be about as politically fashionable as saying “I’ve changed my mind” but that is not a good reason to avoid it.

One organisation in the public eye half-avoids this trap at the moment, and that is the Bank of England. For its forecasts it produces a central projection but also a range of likely outcomes either side of that central projection because, quite rightly, it recognises that making economic predictions is a rather inaccurate mix of art and science. Perhaps this approach of up-front acknowledgement of bands of error and uncertainty is one the IFS could adapt rather more in future?

Check back tomorrow for the next in this series.