Regulations, arguably, should not be based on detailed understanding of how AI systems work (which the regulators can't have in any depth). However, AI systems need to be able to explain decisions in terms that humans can understand, if we are to consider them trustworthy. 1/
Not explanations involving specifics of the algorithms, weights in a neural network, etc., but explanations that engage people's theories of mind, explanations at the level of Dennett's intentional stance - in terms of values, goals, plans, and intentions. 2/
Previous computer systems, to be comprehensible, and, yes, trustworthy, needed to consistently present behavior that fit people's natural inferences to physical models (e.g., the "desktop"). Anyone old enough to remember programming VCRs? Nerdview is a failure of explanation. 3/
AI systems will need to engage not the mind's physics inference, but its *social* inference (theory of mind). AI systems should behave as minds, and explain their behavior as minds. They must have failure modes predictable and comprehensible like human ones (or physical ones). 4/
The counterargument of "How does a system explain how it decided that a stop sign was there? By listing network weights in a perceptual model?!" is a crimson herring. If the perceptual system is accurate enough in human terms (i.e., without crazy error modes), it can just say 5/
"I saw a stop sign," just like a human would. Full stop. More complex decisions, like swerving to avoid a bicyclist and then hitting a pedestrian would require more complex explanations involving values, goals, and intentions, as well as perception ("I didn't see the guy"). 6/
Saying that we attain trustworthiness of AI systems just based on experimental performance metrics, however rigorous and comprehensive, is utterly misguided. Measurable performance is necessary, but not at all sufficient, for (at least) two reasons. 7/
First, it is virtually impossible, for a non-specialist to evaluate the sufficiency of an experimental methodology or significance of the results. It's very easy to create misleading experiments, even without intending to. So this can only create trust among the credulous. 8/
Second, experimental performance guarantees do nothing to develop trust between an AI system and the humans it interacts with - that trust must develop through the interaction, which must therefore be comprehensible and explainable. 9/
In a word, to attain trustworthiness, AI systems must be able to form *relationships* with people. Not necessarily deep relationships, but still they must engage with the human mind's systems of social connection. /FIN
Also, consider this:
@threadreaderapp unroll please
