Remember when Trevor shared (on Instagram) a photo with @michaelkosta at a tennis game? (2/8)
Using cutting-edge image generation models like #dalle2 and #stablediffusion, someone can easily manipulate the above photo to get this (fake) one: (3/8)
Could Trevor have done anything to prevent this? My students @hadisalmanX@Alaa_Khaddaj@gpoleclerc@andrew_ilyas spent an enjoyable weekend hacking together a potential answer: adding small (imperceptible) noise to the original photo can make it “immune” to such edits! (4/8)
After such “immunization”, the same edit of this photo looks much worse.
So, Trevor could have applied such “immunization” to his photo before posting it to protect it against this kind of malicious edits. (5/8)
And it is not only about Trevor’s and Michael’s photo. In fact, the lead student on this project @hadisalmanX has a selfie with Trevor too. Now, Hadi is attempting to “deepen” his (imaginary) friendship with @Trevornoah by manipulating this selfie (and he succeeds!) (6/8)
However, again, had this selfie been “immunized”, this would not have been possible! Indeed, images generated from an immunized version of Hadi’s photo with Trevor are totally unrealistic. (7/8)
This works for other edits too (although, for now, might be specific to the photo-editing engine we had on our hands)! Check out our blog post gradientscience.org/photoguard/ for more examples and more details. And stay tuned for the paper! (8/8)
We trained *hundreds of thousands* of models on random subsets of computer vision datasets using our library FFCV (ffcv.io). We then used this data to fit *linear* models that can successfully predict model outputs. (2/6)
We then use datamodels to: (1) Predict data counterfactuals (i.e., what if I remove subset R from the train set?) and find that you can flip model predictions for *over 50%* of test examples on CIFAR-10 by removing only 200 (target-specific) training images (0.4% of total) (3/6)
ImageNet is the new CIFAR! My students made FFCV (ffcv.io), a drop-in data loading library for training models *fast* (e.g., ImageNet in half an hour on 1 GPU, CIFAR in half a minute).
FFCV speeds up ~any existing training code (no training tricks needed) (1/3)
FFCV is easy to use, minimally invasive, fast, and flexible: github.com/MadryLab/ffcv#…. We're really excited to both release FFCV today, and start unveiling (soon!) some of the large-scale empirical work it has enabled us to perform on an academic budget. (2/3)