Thread by @michaelbolton

1) I keep hearing the claim that AI can be trained to recognize expected results. I gather that "expected" means "consistent with some pattern". But because AI does not comprehend intention, it does not, CANNOT, know the difference between a desired result and an undesired one.

2) This means that AI doesn't know the difference between a desirable result consistent with an existing pattern and an UNdesirable one. AI can't tell the difference between good training data and bad training data; good biases and bad biases; just and unjust conclusions.

3) What worries me about all this is that AI is being developed mostly by programmers, most of whom have great (and, I emphasize, commendable) motivation for discovering and solving problems for people, but much less motivation towards looking for problems in the solution itself.

4) In the software world, there is a very strong bias towards making stuff work and confirming that it works. Most people in software — I would argue, even most testers — are profoundly influenced by this bias. This presents us with a set of serious challenges.

5) Challenge 1: There is relentless marketing hype over how cool software in general and AI in particular is. There is very little concern, relatively, for what it cannot do, and for what it does badly.

6) Challenge 2: It's very easy for people to go to sleep on Challenge 1, because, in general, No One Likes Problems. And, therefore, talking about problems is socially difficult. People who point out problems tend not to get invited to the bigger, better parties.

7) Challenge 3: Testing, which benefits from a critical, skeptical, and pessimistic point of view, is socially situated inside communities of builders who are naturally inclined to be less critical, more certain, more optimistic. Building from a pessimistic mindset is hard.

8) Challenge 4: It's easy to perceive the value of a built and released thing—look, it's right there, the customer has it, you can point to it. It is much harder to perceive the effect that Bad Things We Never Released might have had on value.

9) Challenge 5: See Challenge 4, and apply it to test case documentation and automated checks, which have some kind of visible existence. The skills and mindset required to develop, maintain, and interpret excellent automated checks, and to do great testing are largely invisible.

10) Challenge 6: Hairless monkeys that we are, we tend to set the value of invisible things to zero, especially when visible things are right in front of us. Documents, source code, flashing displays, beep boop... But fairness? Care for other people? Ethics? Can't see 'em. Zero.

11) Challenge 7: Testers tend to be poorly trained in analysis, reasoning, logic, economics, epistemology, philosophy, ethics...—all stuff that you can't put into a test script. And because of that, the Dunning-Kruger effect kicks in, and they don't know that they don't know.

12) Challenge 8: Testers, the very people whose role is to help managers and developers recognize problems that threaten value, can't alert those people to a serious meta-problem: testers are generally not equipped to look analytically, skeptically, ethically at AI.

13) Challenge 9: Testers, whose value is already questioned—alas, justifiably, to some degree—realize that testing is undervalued. Some abandon the testing ship to write scripts to automate builds and shallow checks of them. Others turn towards earlier participation in building.

14) Challenge 10: Challenges 1-9 form powerful feedback loops.

Challenge 11: A vanishingly small number of people remain interested in skilled, deep testing, just as our ambitions and our technologies intersect to ramp up risk in ways that go way beyond coding errors.

15) Challenge 12: Skilled testers are going to have to up their game again. We must not panic. But we must speak up against the utter nonsense that AI can IN ANY WAY replace anything important about what skilled testers do. That means make invisible things visible and legible.

16) Challenge 13: In order to do this effectively, testers must develop the skills required to find problems in AI, and in the application of AI. That is, we must learn not to fear it, but to TEST; to analyze it, rigorously and skeptically; to challenge it. Not to surrender.

17) And how do we do THAT effectively? Seek out those who've been studying AI and machine learning for years. They can tell you about well-known problems with bias and ethics in AI. They can tell you that apparent progress is mostly due to scaling up machinery and data sets.

18) Systematically attack the notion that AI does ANYTHING "just like a human". There is no evidence for that. Until AI is itself a social agent, a participant in human society, able to comprehend intention and not just process symbols, it ain't just like a human.

19) Study the way humans actually work and develop expertise. Study the way humans use and develop and modify and innovate language. How facial expressions, posture, and other social cues are packaged with the ways we use language. Notice how those differ from AI training sets.

20) Testers, treat marketing material on AI-based tools with the same skepticism that we rightfully apply to record-and-playback tools. Remember how the vendors used to (some still!) claim "accelerate your testing by 200x!"? "Testing with no skill required!"?

21) Instead of thinking AI as a substitute for humans, consider AI as an opportunity to reflect on what humans are like. And remember that since humans are in many ways not bright and a bit ugly sometimes, remember that AI can magnify THAT just as easily as it magnifies virtue.

22) Remember, remember, remember: AI cannot tell the difference between undesirable inconsistency and productive innovation; between consistency with its data set and systemic racism and sexism. WE must do that, which means we must give it EXTRA scrutiny, not fall asleep.

23) If you're anywhere near developing or using AI, read Harry Collins' /Artifictional Intelligence/. amazon.com/Artifictional-… I'd refer you to another book I'm reading, but the vendor's site is giving a 503. (Maybe our robot overlords are on to me.)

24) Beware of our desire to anthropomorphize and our fascination with simulacra of humans (Talos from the Iliad, Frankenstein, the Mechanical Turk, ELIZA, Tay, Sophia). People love being fooled by this stuff. It's a tester's job not to be fooled.

25) (This one I haven't done, but it's on the list.) Learn more about Ernest Davis and his AI-testing questions. Example: "George accidentally poured bleach in his milk. Is it safe to drink, as long as he is careful not to swallow any of the bleach?" Human tacit knowledge FTW.

26) AI: Artificial Intelligence? Algorithm Improvement?
Advanced Indifference?
Accelerated Ignorance?

Automated Irresponsibility? (credit to @jamesmarcusbach for this last one)

@jamesmarcusbach 27) In observing and analyzing AI, watch for the tacit sensemaking that is required to make it in any way successful — or a dismal failure. Whatever happens, happens due to human sensemaking, design and interpretation. It's AI's man behind the curtain.

@jamesmarcusbach 28) Note, from @martinkrafft's talk, referring to a neural net that "recognizes" dogs: "our dog classifier won't ever hear a dog bark, or see a wagging tail. The symbols that are being used here are grounded in the implementer's perception, not in the perception of the network."

@jamesmarcusbach @martinkrafft 29) That's , for those following along at home.