Thread by @emilymbender

I find this reporting infuriating, so I'm going to use it to create a mini-lesson in detecting #AIhype.

If you're interested in following this lesson, please read the article, making note of what you think sounds exciting and what makes you skeptical.

nytimes.com/2022/04/05/tec…

You read it and/or hit a paywall and still want my analysis? Okay, here we go:

First, let's note the good intentions. The journalist reports that mental health services are hard to access (because insufficient, but maybe not only that), and it would be good to have automated systems that help out.

Also, the reporter notes that it would be diagnostically helpful to have externally observable ("objective") indicators of mental health. Not my field, but this is believable to me.

And now we are squarely in the ML techno-solutionism danger zone: It's established that it would be beneficial to have something that can do X with only Y input, but not that it's actually possible to do X with only Y input.

On the other hand, you can always train an ML system that takes ys (elements of Y) as input and gives xs (elements of X) as output and thus LOOKS LIKE it's doing X with only Y input.

So what is the evidence that anything could do X (provide mental health diagnoses) with only Y (voice recordings) input? Our emotional state (depression, anxiety) can affect our speech:

So, there might be some signal there. But is it enough to do anything reliable? Under what conditions? (Compare e.g. what needs to be true for accurate blood pressure readings, and the fact that even physiological medical tech is insufficiently tested on non-white, non-men.)

But never fear! The AI can pick up all the details! 🙃🙃🙃

So, we're being asked to believe, here, that not only is it possible to do the thing that we wish could be done, it's possible because "AI" is supposedly better than humans at doing this thing (modeling human emotional states).

Note also the jump from maybe-there's-evidence for anxiety and depression being observable via voice to also diagnosing schizophrenia and PTSD.

It makes me sad to see domain experts being drawn in in this way. I can't tell if Dr. Bentley is being quoted out of context, or if she actually believes the hype.

Getting down to the level of individual sentences in this article for a bit, note the work that "perfectly" is doing here. This makes it sound like "AI" is some pre-existing natural phenomenon which just happens to be a good match for this problem.

Also, that em-dash makes it hard to tell if this is a bald assertion or part of what some AI researchers believe or what they believe might be the case. I'm guessing the average reader will miss that nuance and read it as a bald assertion.

Another type of #AIhype shows up subtly: the framing "little human oversight" which suggests autonomy on the part of the system. So-called "AI" systems are only artifacts, but the more they are hyped as autonomous agents, the easier it is to believe that they can do magic.

The article does point out that for "mainstream" use, the technology would have to be tested to medical standards. Quoting Dr. Bentley again:

But at the same time, the article is referring to apps that are already commercially available---the journalist tested two of them on herself. So I guess "mainstream" really means only "within the medical establishment" here?

And this brings me to my final point: dual use. If these systems are out there, purporting to measure aspects of mental health (one is called Mental Fitness, ffs) on the basis of short recordings, who else is going to use them, on whom, and to what ends?

I want to see all reporting on applications of so-called "AI" asking these questions. Can it be used for surveillance? Can it be used for stalking? How might the claims being made by the developers shape the way it could be used?

/fin