Thread by @michaelbolton

1) "Test automation" as commonly discussed is not the automation of testing; it's the automation of *examples*.

An example is not a test.

Checking an example has some value, but to the degree it's a test at all, it's a pretty shallow test.

2) Checked examples can help to detect some kinds of problems, and thereby reduce some kinds of product risk. This is a fine thing. It is not the same as testing for deep, rare, hidden, subtle, emergent, parafunctional problems. It is not the same as learning about new risk.

3) Checking examples contributes to a disciplined approach to developing, building, and maintaining software. This too is a fine thing. Yet it is significantly limited in terms of the kinds of problems that it can reveal, problems that we discover via experience with the product.

4) When thinking about gaining experience with the product, it helps to broaden the idea beyond user experience. What will experience be for people who are providing technical support for the product? Maintaining infrastructure for it? Testing it? Downstream of people using it?

5) What will the experience be like for people trying to use the product's services? Its APIs? For people trying to document it? To do maintenance work on it? That goes far beyond what automated, checked examples can tell us (though we may learn from trying develop such checks).

6) Here's the issue, to me: that role which we call *testing* conflates at least two important roles these days: build engineering, developer support, infrastructure support — DevOpsy stuff on the one hand; and *testing* — exploration and learning about product risk on the other.

7) Referring to "(test) automation strategy" muddies the water, causing significant confusion and distress among testers, developers, and managers. Any such strategy is not about automation of testing, but of *examples*. And, once again, an example is not a test.

8) Nothing of what I say here is intended to diminish any valuable work done by those who have expertise in programming that aids in automating those examples, providing fast feedback to developers, making it possible to build efficiently and cleanly. That *enables* deep testing!

9) Products and systems are getting more complex. Problems in complex socio-technical systems emerge even from cleanly-built components that, taken individually, work perfectly. (cf. Leveson, /Engineering a Safer World/, @james_christie's recent series clarotesting.wordpress.com/2018/10/12/fac…)

10) My concern is that (worthy!) attention to *efficient building* is being confused with *testing* and why we do it. Evidence: read a treatise on "test automation", and look for "risk", "bug", "error", "defect", "problem". If you find those words, they refer to the automation!

11) We *test* to help our clients and our teams *to find problems that threaten the value of the product*. We test *to learn about the product*, the better to find problems and risks about it. We test *to find out if the product our clients have is the product they want.*

12) Checked examples help to find certain problems that could threaten the value of a product; that's great. However, such problems are only a subset of the problems that can beset that product, or the technological and human systems within which the product exists.

13) Checking examples can really help to find functional problems, and involves using programming and tools—so it's easy to understand why developers and aspiring developers tend to connect with that easily. Deeper testing tends to require some distance from developers' mindsets.

14) Checking examples can look like it's speeding up "testing", and it does help to find certain kinds of bugs—so it's easy to understand why managers connect with that too. Thinking and talking of testing as button-pushing to confirm or demonstrate "it works" reinforces this.

15) The testing world, for a long time, has largely neglected skills of critical thinking, modeling, risk analysis, experimentation, exploration, expert and responsible tool use, gaining experience with the product, finding problems that matter. I recommend we get on that.

16) One approach, it seems to me, is to speak and write more clearly. That way, we can think more clearly and make useful and important distinctions between lowering the cost of development on the one hand, and helping to defend the value of the product on the other.

-fin-