I love Cynthia Rudin's work, but I don't think she is entirely fair to machine learning in this presentation 1/
The goal of interpretable ML is important, and I am a huge fan of decision tree (CART) and rule-learning methods. However, interpretability can be very hard to achieve, whereas things like random forests and deep nets can be easily trained and give high performance 2/
This means that when you take into account the cost of the modeling work that the data scientist must do, there is definitely a tradeoff between performance and interpretability. 3/
In many engineering applications, we need ML models that we can trust, but this doesn't require them to be interpretable. Methods for formal verification (e.g., reachability analysis), calibrated error bars, and so on can assure trust without being interpretable. 4/
Dr. Rudin admits that the high performance of DNNs in computer vision, speech, and language cannot be matched by existing interpretable methods. 5/
In our XAI work, we are finding that the most practical workflow may be to first train an uninterpretable DNN using existing methods, then create an interpretable model that replicates it, and then discard the original model 6/
This is a perfectly reasonable strategy for achieving interpretable ML, and it side-steps the computational challenges of creating an interpretable model from scratch. It also allows us to use representation learning methods 7/
Rudin's critique of saliency is also unfair. A saliency map may not tell you how the network discriminates between dogs and cats, but it does tell you whether the network is looking at the dog or the cat rather than the background. This is important debugging information. 8/
Finally, as many folks have pointed out, the suitability of an explanation or interpretation depends on what task we are supporting (e.g., testing, debugging, forensics, etc.). There is no such thing as a universally interpretable model. 9/
Summary: Interpretable ML is one important way that ML could be improved, and Dr. Rudin is doing outstanding work in this area. But her path is not the only one to follow to achieve trustable and fair AI systems. end/
I should add that there are interesting connections to be explored between distilling a trained net and creating an interpretable version. Alan Fern and his students have been able to distill RNN policies for some Atari games down to simple finite state machines, for example.
UPDATE: Cynthia Rudin is not on Twitter, so she asked me to post the following responses to this thread. 1/
She agrees that there is sometimes a tradeoff between model performance and analyst effort to create interpretable models but emphasizes that for high-stakes decisions, the need for interpretability far outweighs the cost of the analyst's time 2/
For cases such as bail and parole decisions where someone's life depends on the correctness of the ML system, applying formal methods to prove a fixed set of properties does not suffice. 3/
In these domains, the set of properties a court might want to know about a black box model is unbounded. Interpretable models provide a much more robust way to troubleshoot machine learning. 4/
In computer vision, while saliency may be useful, we need much more. She has a method for computer vision ("This looks like that" arxiv.org/abs/1806.10574) that goes beyond saliency to give partial interpretability. 5/
Finally, she agrees that a workflow that first learns an uninterpretable model and then transforms it into an interpretable one is sensible. 6/
Summary: Cynthia and I are in agreement on all points. However, the topic is more complex than can be fully explained in a 10-minute video (or a twitter thread) 7/
Here is a link to her latest paper "Optimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice" doi.org/10.1287/inte.2… end/
