Thread by @michaelbolton

No one ever sits in front of a computer and accidentally compiles a working program, so people know (intuitively, correctly) that programming must be hard. Almost anyone can sit in front of a computer and stumble over bugs, so they believe (incorrectly) that testing must be easy!

There is a myth that if everyone is of good will and tries really, really hard, then everything will turn out all right, and we don't need to look for deep, hidden, rare, subtle, intermittent, emergent problems. That is, to put it mildly, a very optimistic approach to risk.

The trouble is that to produce a novel, complex, product, you need an enormous amount of optimism; a can-do attitude. But (@FionaCCharles quoting Tom DeMarco here, IIRC), in a can-do environment, risk management is criminalized. I'd go further: risk acknowledgement is too.

@FionaCCharles There's a terrific documentary, "General Magic", about the eponymous development shop that in the early 1990s (!!) was working on a device that—in terms of capability, design, and ambition— was virtually indistinguishable from the iPhone 15 years later. It's well worth watching.

@FionaCCharles "There was a fearlessness and a sense of correctness; no questioning of 'Could I be wrong?'. None. ... that's what you need to break out of Earth's gravity. You need an enormous amount of momentum ... that comes from suppressing introspection about the possibility of failure."

That's from Marc Porat, the project's leader, much more recently, talking about why it flamed out without ever getting anywhere near the launchpad. And that persists all over software development, to this day: systematic resistance to thinking critically about problems and risk.

That resistance plays out in many false ideas: that the only kind of bugs are coding errors; that the only thing that matters is meeting the builders' intentions for the product; that we can find all the important problems by writing mechanistic checks of the build. And more.

Another is the unhelpful division of testing into "manual" and "automated", where no other aspect of software development (or indeed of any human social, cognitive, intellectual, critical, analytical, or investigative work) is divided that way. There are no "manual programmers".

Testing cannot be automated. Period. Certain tasks within and around testing can benefit A LOT from tools, but having machinery punch virtual keys and comparing product output to specificed output is not more "automated testing" than spell-checking is "automated editing". Enough.

It's unhelpful to lump all non-mechanistic tasks in testing under "manual", as though all of the elements of cooking (craft, social, cultural, aesthetic, chemical, nutritional, economic) were "manual" and unimportant; all that matters is the food processors and blenders. Geez.

If you want to notice important things in testing, consider some things that get swept under the carpet of "manual testing": *experiential* testing, in which the tester's actions are indistinguishable from those of the contemplated user. Contrast that with *instrumented* testing.

Instrumented testing is testing wherein some medium (tool or technology) gets in between the tester and the naturalistic encounter with and experience of the product. Instrumentation alters, or accelerates, or reframes, or distorts; in some ways helpfully, in other ways less so.

Are you saying "manual"? You might be referring to *attended* or *engaged* aspects of testing, wherein the tester is directly and immediately observing and analyzing aspects of the product and its behaviour in the moment— and contrast that with mechanistic, unattended activity.

Are you saying "manual"? You might be referring to testing activity that's *transformative*, wherein something about performing the test changes the tester in some sense, inducing epiphanies or learning or design ideas. Contrast that with *transactional*, a box-checking activity.

Did you say "manual"? You might be referring to "exploratory" work, which is interestingly distinct from "experiential". "Exploratory"—in Rapid Software Testing at least)—refers to agency; who or what is in charge of making choices about the testing. How is that not experiential?

You could be exploring—making unscripted choices—in a way entirely unlike the user's normal encounter with the product—generating mounds of data and interacting with the product to stress it out, to starve it of resources, to investigate like a *tester*, rather than like a user.

And you could be doing experiential testing in a highly scripted, much-less-exploratory kind of way; for instance, following a user-targeted tutorial and walking through each of its steps to observe inconsistencies between the tutorial and the product's behaviour.

For a while, we considered "speculative testing" as something that people might mean when they spoke of "manual testing"; "what if?" We contrasted that with "demonstrative" testing—but then we reckoned that demonstration is not really a test at all. Not intended to be, at least.

The thing is: part of the bullshit that testers are being fed is that "automated" testing is somehow "better" than "manual" testing because the latter is "slow and error prone" — as though people don't make mistakes when they use automated checks. They do. At colossal scale.

Sure, automated checks *run* quickly; they have low execution cost. But they can have enormous development cost; enormous maintenance cost; very high interpretation cost (figuring out what went wrong can take a lot of work); high transfer cost (explaining them to non-authors).

There's another cost, related to these others. It's very well hidden and not reckoned. A sufficiently large suite of automated checks is impenetrable; it can't be comprehended without very costly review. Do those checks that are always running green even do anything? Who knows?

Checks that run RED get frequent attention, but a lot of them are, you know, "flaky"; they should be running green when they're actually running red. Of the thousands that are running green, how many should be actually running red? Cognitively costly to know that—so we ignore it.

And ALL of these costs represent another hidden cost: *opportunity cost*—the cost of doing something such that it prevents us from doing other equally or more valuable things. That cost is immense when we contrast trying to automate GUIs with interacting with the damned product.

Something even weirder is going on: instead of teaching non-technical testers to code and get naturalistic experience with APIs, we put such testers in front of GUIish front-ends to APIs. So we have skilled coders trying to automate GUIs, and Cypress de-experientializing API use!

And none of these testers are encouraged to analyse the cost and value of the approaches they're taking. Technochauvinism (great word; read @merbroussard's book) enforces the illusion that testing software is a routine, factory-like, mechanistic task. It isn't that at all.

@merbroussard Testing must be seen as a social (and socially challenging), cognitive, risk-focused, critical (in several senses), analytical, investigative, skilled, technical, exploratory, experiential, experimental, scientific, revelatory, honourable craft. Not "manual" or "automated". FFS.

@merbroussard Testing has to be focused on FINDING PROBLEMS THAT HURT PEOPLE OR MAKE THEM UNHAPPY. Why? To help the optimists who are building the products to be aware of those problems, and to address them. Whereby they make themselves look good, make money, and help people have better lives.