,
16 tweets,
5 min read
Read on Twitter
1) Ten years ago today, I wrote a blog post distinguishing checking (a prescribed, mechanistic, confirmatory activity) from testing (oriented towards exploration, discovery, investigation, and learning). developsense.com/blog/2009/08/t… It takes time for ideas to develop. /
2) One problem was expressing a distinction as “vs.”, which many took to mean “in opposition to” (Real Madrid vs. Manchester United) rather than “as distinct from” (trees vs. forest; biting vs. eating). Checking is not *opposite to* testing; it’s *inside* testing. /
3) A few years later, @jamesmarcusbach and I got much closer to what we were trying to say. satisfice.com/blog/archives/… Checking is a part of testing that can be done entirely algorithmically, by a machine. But that part needs human risk analysis, design, programming, and evaluation.
@jamesmarcusbach 4) Checking can be important if you want to detect problems that have calculable, deterministic answers: did the product produce a specific, anticipated, desired, or required output? Does this result fall within this range? Are all the bits and pieces of the product in place? /
5) No amount of checking, however, can reliably identify unanticipated problems. Machines can't conjecture; can't reason; can't empathize; can't invent; can't be self-critical; can't recognize new risks. We need people for that stuff. Machines are not social agents. /
6) In these ten years, our purpose in talking about testing and checking has been to emphasize the importance of people in finding problems that matter to people. Checking can suggest that checked outputs are correct; checking alone can't alert us to problems that wreck value. /
7) The trouble really starts when fascination with programming checks (or, heaven help us, applying AI somehow) displaces the goal of gaining interactive experience with the product, which is a critical element in finding problems that matter. The machinery isn't doing that.
8) The machinery is pressing virtual buttons, calculating, and comparing product output to some desired result, and doing so very quickly. That can be a very useful thing to do. But let's not fool ourselves into believing that repetitive machinery is acting like a real user. /
9) Real users don't do the same things over and over again, and they don't do so blindly, divorced from a task to perform, a mission to accomplish, or a problem to solve. Real people experience confusion, frustration, impatience, annoyance, fear... They act unreliably, too.
10) All the stuff that people call "manual" testing, to me, is really *experiential* testing: gaining real experience with the product by interacting with it on some level. That interaction may be mediated and aided to some degree by tools. Big deal; almost everyone uses tools. /
11) That interactive experience may be obtained at the user level or elsewhere. But calling it "manual" testing focuses on the means of providing input, rather than on the important stuff: the cognitive experience; the learning; the way the testing develops the tester's mind.
12) @jamesmarcusbach and I are exploring an idea he had recently. Checking and regression testing are always done *after* someone has developed a model of the product. In the kind of testing that we practice and teach focus is on the development of new models AND of the tester.
@jamesmarcusbach 13) In order to find unanticipated problems in a product, it's essential to be able to develop and apply models that are novel and distinct from any models used to build or test the product so far. This applies whether one is a tester, or a developer testing one's own code.
@jamesmarcusbach 14) We can develop and use powerful tools in powerful ways. But tools extend us; they do not and cannot replace us. Consistent with Weick's comment Ashby's Law of Requisite Variety, in order to understand something complicated, we must complicate ourselves. /
@jamesmarcusbach 15) There is much more to think about, discuss, and develop on this thread. But thinking and talking of experiential testing—and of @jamesmarcusbach's ideas on how excellent testing changes people and requires people to change—has me as excited as I was ten years ago. -fin
@jamesmarcusbach Postscript: Just as I'm typing the last character of that last tweet, Tweetdeck mangles the columns (yet again), making it impossible to tell whether I had got what I wanted. So much testing still to be done... 
