Thread by @ch402

“Adversarial Examples Are Not Bugs, They Are Features” by Ilyas et al is pretty interesting.

📝Paper: arxiv.org/pdf/1905.02175…
💻Blog: gradientscience.org/adv/

Some quick notes below.

(1) The most striking thing to me is actually a new method in their paper: creating new datasets by projecting examples into a network's representation and then doing feature inversion.

They’re basically using this to try and filter the dataset through the model’s features. It almost feels a bit like model distillation, but to a dataset.

(The method is vaguely similar in technique to Gerhos et al’s “Stylized ImageNet”, but feels pretty different.)

(2) The claim which will seems to me really remarkable if it holds up is that you can use this process to turn robust models into robust datasets, for which normal training creates robust models.

I say claim because I’ve learned to not believe any robustness claim until people like Nicholas Carlini have taken a shot at breaking it!

Note that this isn’t a robustness free lunch -- you still need to create the original robust model through adversarial training.

(3) The other interesting result is that you can create a different dataset of adversarial attacks, where you try to predict the attack class.

They find this model - trained on adversarial attacks - generalizes to clean data, which I probably wouldn’t have predicted in advance.

This does seem like non-trivial evidence for adversarial examples being genuine features of the data, rather than properties of our models or training set quirks.

Caveat to all of this: Adversarial examples are not my domain of expertise and I haven't read this super carefully!

... On more consideration, I'm a little less persuaded that (3) is strong evidence for adversarial examples being intrinsic to the data.

Clearly the adversarial directions aren't a linear feature. The model is interpolating some more complicated feature from each adversarial direction.

This suggests an alternative interpretation: Perhaps the adversarial directions just strongly correlated in parameter space with a true useful feature. When you fit to them, you get the useful feature, and vice versa.