Thread by @mark_riedl

What’s wrong with “explainable A.I.”

(Making the rounds again.)

1. Most XAI algorithms were not designed with humans in mind or consideration of what doctors really need help with

2. I still believe in the power of XAI, but research is still nascent ...
fortune.com/2022/03/22/ai-…

3. Verification, which is advocated, serves a different purpose than explanation. Verification tells us whether a system is compliant with regulations. Explanation can tell us on an instance-by-instance timescale whether to trust the system.

4. It’s not one-or-the-other, do both

5. But first, we need to be farther along with human-centered XAI so we don’t develop the wrong thing. Or worse, deploy a system that accidentally engenders inappropriate trust.

6. We would never want to prematurely deploy AI, right? … Right?
*Padme/Anakin meme image here*

7. It is unlikely that any AI/ML system will be right all the time. Even if verified. One thing that XAI can do is help non-AI-experts construct mental models of the system so that they can make more informed decisions about whether the system is providing output they can act on.

8. In fact we have recent work (hopefully I will be able to share soon) that shows that explanations can /calibrate/ user’s trust on an instance-per-instance basis. A system can be trustworthy all the time based on the inputs given; start thinking about instance-level trust.

9. I’m not ready to throw the baby out with the bath water. We just need to remember that XAI is in service of humans and therefore we need to study humans.

10. This is the topic of the #CHI2022 Workshop on Human-Centered XAI hcxai.jimdosite.com See you there