Machine learning for decision making often results in discriminatory systems. The reason for this isn't specific to ML. It applies to many quantitative models—models that start by assuming a fixed, exogenously specified distribution of decision subjects' characteristics. 🧵
But the distribution of people's attributes (skills, risk levels, preferences…) isn't fixed. In fact, it is usually produced largely by the past effects of the types of decision systems that these models are used to justify. So the logic underlying these models is tautological.
Unless a model can endogenize the risk distribution, it has zero normative content. At best it has some descriptive value to explain observed differences. But these lazy models are so often used to make policy and thus become excellent tools for reinforcing the status quo.
But math doesn't have to be used this way. We can learn to build more sophisticated models, use a diversity of models instead of pretending there's one right answer, and learn to recognize situations where our understanding is so hazy that quantitative models would be premature.
All this will take painstaking structural changes to the way we teach and apply quantitative skills. Until then, the bulk of quantitative modeling will continue to lead to research that explains away systemic injustice and decision making systems that perpetuate it.
For example, datasets that are snapshots of a single system with many subjects at a single point in time tend to hide systemic discrimination whereas datasets that follow people over time as they are successively judged by many systems tend to reveal systemic discrimination.
Yet almost all available datasets belong to the former category as they are much cheaper to collect and serve the needs of the decision makers who fund this work. This is another way in which politics and ideology are encoded into seemingly inert and objective activities.
• • •
Missing some Tweet in this thread? You can try to
force a refresh
