"Reformulating the algorithm so it no longer uses costs as proxy for needs eliminates the racial bias in predicting who needs extra care."
Only a fricking INSURANCE COMPANY would think actual NEED shows up in past spending. Damn. continued -
cc @MathBabeDotOrg
Only a fricking INSURANCE COMPANY would think actual NEED shows up in past spending. Damn. continued -
cc @MathBabeDotOrg
If I understand correctly, this appears to be exactly the kind of self-reinforcing algorithmic death spiral Cathy O'Neil @MathBabeDotOrg described in the excellent Weapons of Math Destruction. Alg fed by biased histories presumes your current state is a reliable predictor. /2
e.g. You're trying to climb out of a bad neighborhood but resume-filtering algorithms don't highlight you for job interview bc you live in a bad neighborhood.(!!) Or prison sentencing software (yes it exists) recommends stiff sentence for similarly circular reasons. /3
This study seems to say that United (the insurer) equated past spending, which turned out to be biased against blacks, with accurately representing actual need. Circular logic. Weapons of Math Destruction does a great job of describing how blindly-run algs can destroy. /4
The key, @MathBabeDotOrg describes, is that the alg runners have to check the results and fix where it fails. My great hope is that this is exactly what's happening here: having discovered the unintended outcome, they're going to fix it - /5
As the article's abstract says, "Reformulating the alg so it no longer uses costs as a proxy for needs eliminates racial bias in predicting who needs extra care."
We all (esp those who base predictions on history) need to understand this kind of error.
TYVM @MathBabeDotOrg
/end
We all (esp those who base predictions on history) need to understand this kind of error.
TYVM @MathBabeDotOrg
/end
@mathbabedotorg @threadreaderapp unroll
