Thread by @enjalot

Beyond excited to share the latest @distillpub article: Building Blocks of Interpretability
distill.pub/2018/building-…
An epic collaboration with @ch402 @arvindsatya1 @shancarter @ludwigschubert @hypotext and @zzznah introducing exciting techniques for looking deeper into ML models

The amazing @TwoMinutePapers released a great summary video of the article:


You can also get a high level overview of the concepts from the @nytimes
nytimes.com/2018/03/06/tec…

The article takes the Feature visualization technique from distill.pub/2017/feature-v… and uses it to turn "anonymous" activation vectors into "semantic dictionaries": a data structure where the keys are meaningful icons. This opens the door for very interesting interfaces...

Each of the hidden layers of the network can be thought of as a cube that can be sliced in different ways. Any of those ways can use Feature Visualization to create a foothold for a visual interface. So rather than trying to stare at array indices you look at little icons

For example you can visualize activations for each spatial position you get a sense of what the network is taking away from that part of the image. It's interesting to see a cross between a tennis ball and an apple in the dogs mouth (even tho its just dog lips in the photo)

Again, the feature visualizations give us something to look at, and once we have that we can start to investigate deeper, like showing how much particular spatial positions were activated. This helps us see what parts of the image were significant to the network at each layer

A fascinating technique we can add on is Attribution, previously people mostly looked at how the input image impacts the final classes, but we can actually see how spatial activations in one layer impact the next (or any other layer, including the output!)