Thread by @michaelbolton

1) Alas far too many managers—and perhaps worse, testers—believe that creating automated checks is the main goal for testers. That’s mistaking means for ends. Testing’s main goal is to reveal risk and problems about the product. Automated checking is but one means for doing that.

2) eXtreme Programmers (whom I admire) and .*DD advocates tout creating automated checks as a means of instilling discipline in development. This is a Good Thing, as are many other means of instilling and maintaining discipline in development. Let’s not forget those.

3) Let’s accept (but only for a moment) the idea that a tester's work is exclusively focused on producing automated checks. How would we evaluate the quality of that work? a) The checks would have to detect actual problems, if they’re there.

4) b) The checks would not miss problems, if they’re there. c) The checks would not be focused non-problems. d) The checks would not distractingly alert us to non-problems. The sum of a,b, c, and d is that the checks are valid and reliable.

5) Now, e) To the degree that the checks are valid and reliable, they’re also valuable; that is, they point to *important* problems and risks. f) To the degree that they are of sufficiently high value, the checks are also of reasonably low cost. Cost has lots of dimensions.

6) Execution time for a check tends to be of low cost. Other costs include development cost; maintenance cost; training and transfer cost (to impart knowledge about the checks to those who need it); equipment cost; and result-interpretation cost.

7) Those costs are sometimes well-understood, sometimes taken somewhat for granted, sometimes profoundly ignored, depending on how people are attuned to looking for them. And those costs may be reasonable, worth incurring for the value the checks deliver. Maybe well worth it.

8) However, there’s another kind of cost that frequently goes unrecognised: opportunity cost. That is: the degree to which stuff you’re doing displaces your opportunity to do other stuff that you might value more. Evaluating all this is a critical task for testers and managers.

9) I choose the word "critical" deliberately to take advantage of the double meaning. "Critical", in the sense of "important, crucial"; and "critical" in the sense of "thinking critically"—that is, thinking about our thinking, with the goal of avoiding being fooled.

10) Evaluating cost and value of automated checks is highly vulnerable to what George W. Bush might have called misunderestimation, especially when we misoversimplify (if I may) our observation and evaluation.

11) For instance: what do the checks actually check, and how much work does it take to obtain the check mark? If you look closely, dear manager, you might find that hours, days of work are being spent in simple programming problems, when the tester isn’t getting training or help.

12) Why does the tester need help? Maybe because he’s not a programmer; because the interns left a mess; because elements aren’t tagged and the tester has entered XPath Hell; because the tester doesn’t have a decent IDE... and there’s another problem...

13) Maybe the tester doesn’t even know she needs help, because she’s immersed in a culture where problems like the ones l've mentioned—and there are MANY more like them—have become normalised. In many places, the Dunning-Kruger Effect thrives around testing.

14) And maybe the tester knows he needs help, but is afraid to ask for it, because This Stuff Is Supposed To Be Easy. The developers say it’s easy, a lot of the tool vendors say it’s easy. With a few extra secret hours of overtime, that login check will actually log in.

15) The goal of finding problems about the product gets displaced by the goal of just making the damned automated checks check *something*, quickly. Which can lead to code that is simultaneously not very reliable, nor maintainable, nor adaptable—and focused on shallow problems.

16) Mind you, when developers are programming with a great deal of discipline, testers will encounter a lot fewer problems developing automated checks. Ironically, that discipline greatly reduces the need for testers to focus on automated checking.

17) Right now we have a world in which more and more shops have testers shifting focus to building the product, and using tools to check the build. It’s really good to be able to build quickly and reliably, so deeper testing can start at the drop of a hat.

18) Trouble is, far too many shops remain fixated on the idea that testing is all about testers pressing buttons to complete test cases. So what happens next? Where does the effort go? GeMPuB: Getting Machines to Push Buttons. Once the buttons are pushed, though, what happens?

19) Functional checks, mostly. Upon pushing this button, look for this element. Given these inputs, check for this output. There's some value to that. The latter, especially, is way cheaper and easier, providing faster feedback, when it's part of the developers' discipline.

20) GeMPuB isn’t easy. It’s trained on the interface most amenable to humans and least amenable to machines and checks. It displaces the goal of investigating the product, exploring it, experimenting with it, gaining human experience with it, finding important problems.

21) Imagine a world in which developers AND testers who aspired to build programming skills could develop tools, create data, perform analytics, produce visualisations, to help themselves AND the less code-focused testers develop deep product understanding and find deep bugs.

22) Here’s a heuristic for determining that you’re NOT in that world: if people are referring to an "automation strategy", beware. Automation is NOT the goal. Tools and automation are *means* of advancing *some aspects* of your TEST strategy.

Aren’t they?

/end

Note: For the ideas on costs, I'm indebted to @jamesmarcusbach and Cem Kaner for ideas in their Black Box Software Testing class. There’s more to come on the subject of costs—but some other time.