With methods research, the dataset is secondary. This focus is misaligned with broader goals of studying risk assessments (eg COMPAS). A paper can be high quality in a pure AI/ML methods sense, but irrelevant for criminal justice impact or worse. 1/
from arxiv.org/abs/2106.05498
"Placing data in subservience to optimization goals decontextualizes it – the objective is beating a measure of performance instead of gleaning new insights from the data." 2/
"Risk assessment in criminal justice is not a modular pipeline in which each component can be replaced with a fairer version the way you would replace a sorting algorithm with a more efficient implementation. It is a tangled mess drenched in an ongoing history of inequity." 3/
"The conference publication workflow & community norms in CS make appropriate use of risk assessment/criminal justice datasets difficult... Researchers are incentivized to continuously replicate even flawed experiments." 4/
above quotes from "It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks" by @TheMichelleBao @angelamczhou @samzottola @BrianBrubach @DrSLDesmarais @Aaron_Horowitz @KLdivergence @geomblog
Read it in full: arxiv.org/abs/2106.05498
Share this Scrolly Tale with your friends.
A Scrolly Tale is a new way to read Twitter threads with a more visually immersive experience.
Discover more beautiful Scrolly Tales like this.