Thread by @ch402

Our most recent paper on visualizing neural networks is one of the best thing I've ever done. distill.pub/2018/building-…

It qualitatively changes the questions we can ask. One basic example:

neuron 426 fired ~= useless
[floppy ear] fired = very interesting!

We can also use these techniques to see how the entire input image was understood at different layers of the network. First edges, then texture and 3d, then high level ideas like floppy ear and snout:

Making hidden layers meaningful like this means we can do attribution from and to them, instead of the input:

But we can also do attribution to things other than spatial activations. For example, which detectors were important in deciding whether this is an image of a cat or a dog?

(Turns out, floppy ear is super important. The network has a whole vocabulary of ears that play a critical role in classification -- how could we have known that before hand?)

Of course, there's too many channels to reason about. But for a given example, groups of neurons mostly work together. We can extract these neuron groups and explain things in terms of them:

I think techniques like this may have an essential role in getting systems to do the right thing *for the right reason*.

In turn, this seems to me to be at the core of building safe, aligned, fair AI systems that robustly reflect our values in the future.