Thread by @math_rachel

The false hope of current approaches to explainable AI in health care: current explainability approaches can produce broad descriptions of how an AI system works in general, but for individual decisions, the explanations are unreliable or superficial 1/
thelancet.com/journals/landi…

Explainability methods of complex AI systems can provide some insight into the decision making process on a global level. However, on an individual level, the explanations we can produce are often confusing or even misleading. @MarzyehGhassemi @DrLaurenOR @AndrewLBeam 2/

Increased transparency can hamper users’ ability to detect sizable model errors and correct for them, "seemingly due to information overload." 3/

All of these examples reveal another major challenge: explanations have no performance guarantees. Indeed, the performance of explanations is rarely tested at all. 4/

The medical system is already extremely adept at validating various kinds of black-box systems, as many drugs & devices function, in effect, as black boxes. Eg Acetaminophen has been used for more than a century but its mechanism of action remains only partially understood. 5/

Explainability methods are incredibly useful for troubleshooting & system audits. However, we must treat these systems as black boxes, justified in their use not by just-so rationalisations, but by reliable & experimentally confirmed performance. 6/

thelancet.com/journals/landi…