Thread by @nklsaltrmrk

I figured I should write something in English about what I presented in the Swedish parliament today: the Swedish ’guesstimates’ of the level of benefit frauds in disability services. This should also be relevant to ppl following disability politics/social policy elsewhere.

As is shown by lots of research, cutbacks on disability services, and welfare more generally, are often accompanied by a rhetoric focusing on benefit fraud and welfare criminality.

In Sweden, the last few years have seen a clear discursive focus on welfare criminality, used to legitimate efforts to reduce the number of assistance users and assistance hours. Of course, the cutbacks are not designed to target welfare criminality..

But the discourse on welfare criminality nevertheless frame the debate. Central here has been the proposition that 10% of all personal assistance payments are incorrect. This has been repeated to the point where it today is a taken-for-granted fact.

This estimate originates in three separate public commissions – from 2008, 2011, and 2012, which are still often referred to in government decisions and public reports. All of these use the ‘expert elicitation method’ (EEM) to estimate the level of incorrect payments.

Together with @nilssonhampus , I conducted a litt. review to assess the suitability and application of the method, based on a material of 1283 separate publications using or discussing the method.

The EEM was developed to estimate security and risk provided hard to interpret background data. Like the size of evacuation zones in the eventuality of nuclear disaster. The idea is that estimates of a group of independent experts can provide policy makers with important input.

Our first finding is that the method never has been used to estimate level of benefit frauds, questions related to welfare, or even in social science. Hence, there is no scientific evidence of the suitability of the method. Still, it is branded ‘scientific’ in the public reports.

Estimating risk provided natural scientific data is fundamentally different from questions of social science, involving human behavior. The use of EEM here is like using a weighing machine to measure the length of a person.

Our second finding is that the applications of the method to measure incorrect payments breaches more or less all central methodological recommendations as concerns the EEM. Thus, the instructional manual of the weighing machine is blatantly ignored. I’ll give three examples:

1.The literature agree that the participating experts should be independent and not affected by results. The expert group should be as heterogeneous as possible. In the Swedish applications, all ‘experts’ are public officials working at the same public authority.

Hence, the experts are very likely to be affected by the results, they represent the same organisational perspective and the group must be considered extremely homogenous. This is a basic methodological requirement ignored.

2.The literature agrees that estimates should be based on solid and indisputable data: this is not a method developed to produce good speculations, but to make estimates when there is an abundance of data that is hard to interpret.

In the Swedish applications, there is virtually no solid background data at all, except from actual re-payments. Provided the low levels of these (we are talking parts per thousand), it is not clear how the commissions arrive at the estimate that 10% of all payments are incorrect

3. In the literature, a lot of energy goes into discussing how results should be interpreted, emphasising the insecurity of the results and the caution required as concerns how they should be interpreted.

The Swedish applications present solid percentage results, with decimals, which are picked up on by media and politicians. The wide range of the uncertainty interval is quickly lost.

Here, the public commission of 2012 is of special importance. Its description of how results are interpreted reveals a shocking lack of understanding of the basics of the EEM. Fundamental statistical rules are blatantly ignored, reasoning is self-contradictory.

Compared to the scientific literature on the EEM, the final report is absurd reading – and tragic, considering the consequences for how disability policy has evolved. It is mindblowing that the results of this commission has set the tone of the debate on personal assistance.

In conclusion, it should be absolutely clear that cutbacks on personal assistance are motivated by estimates of benefit frauds that use a method developed to answer fundamentally different questions.

The applications ignores/misunderstands basic recommendations for how the method should be used. However, the results continue to be repeated, over and over again.

And, finally, right now a new public commission has decided to make use of the EEM to estimate the total volume of incorrect payments in Swedish welfare systems. Austerity needs knowledge production making austerity look a reasonable response.