Thread by @kristyechoi

Excited to share our #ICML2020 paper on fair generative modeling! We present a scalable approach for mitigating dataset bias in models learned on various datasets without explicit annotations. 👇

w/ @adityagrover_ @_smileyball Trisha Singh @StefanoErmon
arxiv.org/abs/1910.12008

Generative models can be trained on large, unlabeled data sources.

If we naively mix all datasets, a trained model will propagate or amplify the bias in this mixture. On the other hand, labeling all attributes of interest may be impossible or super expensive. (2/7)

We use one dataset as a reference (from external prior knowledge) and let all other datasets be biased w.r.t. this reference. Our idea is to construct an *importance weighted* dataset for learning. Here, weights are the density ratio between the biased and reference distributions

To estimate this ratio, we train a binary classifier to distinguish examples from the reference and biased datasets.

The classifier’s odds ratio is a consistent estimator for the density ratio. This requires *no labels* and makes *no assumption* about the bias source. (4/7)

Empirically, the importance weighting method works well even when the size of the reference dataset is small compared to the biased dataset. There’s little compromise in resulting sample quality as measured by FID. (5/7)

There’s still a long way to go for generative modeling that can be reliably deployed in the real world!

We hope our work is a first step towards solutions that take a more holistic view of bias and fairness in generative modeling, esp. in light of everything going on lately.

Come virtually say hi at the conference during our poster sessions on Tuesday, July 14th: (1) 7:00-7:45am PST and (2) 8:00-8:45pm PST. (7/7)