Thread by @ChristophMolnar

Most common arguments against interpretable ML:
- Humans can't explain their actions either
- Performance > Interpretability
- Slows down ML adoption
- Linear model also not interpretable
- Might create illusion of understanding the model

Thread:

Btw. critique also comes from leading researchers:

Interpretability has been overblown." Y. Bengio
"How important is the interpretability of your taxi driver?" @ylecun
@goodfellow_ia: it might make people think they understand ML when they don't.

Let's dissect the arguments:

Humans can't explain their actions either

Depends.
Humans act and come up with narratives why afterwards. Demonstrated in patients with disconnected brain halves.
I can't really explain why I moved to a certain city or why I like beer better than wine (bc I'm German ofc).

But:

In narrow settings humans can give correct, reliable explanations:
Example: "Why do you think there is a cat in the image?" - "Because in the left upper corner, on the bed, there is an object that looks like a cat: fluffy fur, cat ears, eyes, ..."

Another example:

Taxi takes a detour, customer becomes suspicious:
"Why did you not go straight here?"
"Because I heard on the radio that people are demonstrating on the street and I'm driving around"

We DO require our taxi drivers (and other humans) to explain themselves. On demand at least.

I reject "A does not do X, so B does not have to do X":
"Humans are not deterministic, so computers don't have to be."
"Humans do not come with documentation, so our software does not have to be documented. "

ML's scale, impact and illusion of rationality demand a look inside.

Performance > Interpretability

True, if you can capture 100% of your problem's definition in a single loss function, the training data was collected in controlled setting, and you do not care to learn anything about the problem.

An unlikely situation.

Btw. great work by @FinaleDoshi and @_beenkim on the "When do I need interpretability?"

But I agree that performance is VERY important. What's the use of a model, when its performance sucks? Any interpretation derived from a low performing model is probably wrong.

Drawing conclusions from models with low predictive performance is also my biggest critique of the classic "statistician" culture, which is: Performance << Interpretability.

My suggested solution and big vision: View modeling and interpretation as separated steps.

1. Build whatever machine learning you like and performs best on the task at hand.

2. Interpret the model with model-agnostic tools, draw conclusions. Current approaches are: partial dependence plots, local models for explaining single points (Lime), feature importance, ...

With all the model-agnostic tools we already have - and more that are being published on arxiv every day - there is no excuse not to apply them to any machine learning model.

Detect biases. Find problems in the data. Learn about the data and problem. Audit. Communicate.

"Interpretability slows down ML adoption"

Model-agnostic tools also invalidate this argument. It is also contraire to my experiences from jobs, consulting and dozens of conversations with practitioners and statisticians: LACK of interpretability is slowing down adoption.

"Linear model are also not interpretable"

They are less interpretable then people think. But you can learn the correct interpretation (e.g. interpretation of weights is dependent on other features being fixed).
But the linearity makes them a lot more interpretable than any NN.

"Interpretability might create an illusion of understanding the model"

This is a problem, an old one. For example in classic statistics, when applying a - let's say linear - model to a problem. It creates the illusion that the world really is linear, even when it is not.

We have to be diligent when analysing black box machine learning models and be sure to draw the right conclusions as with any analysis we do. But it should not stop us from asking for insights!

That concludes my dissection of "anti ML-interpretability".

Did I miss anything?