Thread by @random_walker

To better understand the ethics of machine learning datasets, we picked three controversial face recognition / person recognition datasets—DukeMTMC, MS-Celeb-1M, and Labeled Faces in the Wild—and analyzed ~1,000 papers that cite them.
Paper: arxiv.org/pdf/2108.02922…

Thread ⬇️

First, congrats to lead author Kenny Peng who worked on it for over a year. Kenny was a sophomore here at Princeton when he began this project, and he did this in addition to his coursework and several other research projects. The other authors are @aruneshmathur and me.

Finding 1: despite retraction, ​​DukeMTMC and MS-Celeb-1M are available through copies, derivatives, or pre-trained models. They are still widely used in papers. The creators (especially MS) simply took down the websites instead of making the ethical reasons for retraction clear.

The point isn’t that ethically problematic datasets shouldn’t be retracted. The critical work that led to retractions is invaluable. The point is that the creators could have handled the retractions better and we need other approaches going forward so retractions aren’t needed.

Finding 2: there are many derived datasets that include the original data. There’s no systematic way to even find them all. In most cases they create new ethical concerns by enabling new applications, releasing pre-trained models, adding new annotations, or other post-processing.

Finding 3a: tech change shapes dataset ethics. Benchmark datasets are often introduced for tasks where the state of the art isn’t ready for practice, which seems less ethically serious. But the benchmark enables research progress that leads to production use of the same dataset.

Finding 3b: social change shapes dataset ethics. When LFW was introduced, the diversity of the dataset was a _selling point_.A decade later it became one of the main points of criticism.

(Note: we use the term “social change” in a broad sense.)

Finding 4: the licenses of these datasets are a mess. LFW was released with no license (!). Many datasets are released under non-commercial licenses. If the intent is to stop production use, the effectiveness is limited since some of the most problematic uses are by governments.

Derived datasets often violate licenses. 4 of 7 MS-Celeb-1M derivatives failed to include the non-commercial designation. All 7 violate MS-Celeb-1M’s license which prohibits derivative distribution in the first place. Only 3 of 21 pre-trained models included the designation.

Perhaps the most significant legal issue is the commercial use of models trained on non-commercial data. Our analysis of online forum posts suggests there’s a tremendous amount of confusion about this.

Finding 5: identifying datasets through paper citations is just terrible. Often we couldn't figure out which dataset is being referred to and where to locate it. This causes problems for documentation, transparency and accountability, and research efforts such as ours.

There’s a much-needed and active line of work on mitigating ML dataset harms. The main implication of our findings for this work is the difficulty of anticipating ethical impacts at dataset creation time. We advocate that datasets should be “stewarded” throughout their lifecycle.

We outline a model for what dataset stewarding could look like. There’s lots more in the paper than this thread so check it out: arxiv.org/pdf/2108.02922…
We will share a methodological appendix and replication materials in a few weeks so others can check and build on our work.