1) An epiphany a year ago informs part of our definition of testing: Testing is the process of evaluating a product by learning about through *experiencing*, exploring and experimenting, which includes to some degree questioning, studying, modeling, observation, inference, etc.
2) That stuff that people call "exploratory" testing (all testing is exploratory) might also—and arguably, better—be considered as *experiential* testing. Direct, interactive, and to a large degree unmediated experience with aspects of the product *as people would experience it*.
3) We might use tools to help with aspects of exploration, or with extending or accelerating aspects of gaining experience, but experience with the product is essential. Too often, tool use in testing these days is directed entirely *away* from gaining experience of the product.
4) The crazy thing is, we could absolutely use tools and tool-friendly means to help us with stuff that matters. Yet today's Big Deal seems to be GeMPuB: Getting Machines to Push Buttons. GeMPuB is dazzling; because Buttons Are Being Pushed, people thing "Lo! There be testing!"
5) We need tools to support our observation; to help us generate and wrangle and (especially!) analyze and visualize test data. Tools to help us see the invisible. I won't name names, but the equivalent, for testers, of what rich, powerful intellijent tools can do for developers.
6) Too often, I'd say, testers are saying "we have to repeat ALL the checks we've EVER done, because we have NO IDEA of what has changed in the product. And even less of an idea of how the changes we don't know about affect stuff we can't see." So: more programs to check output.
7) Here's how I would approach that problem if I ran the zoo. Programmers write code. That code gets written and managed in a disciplined way, with LOTS of collaboration, review, pairing, automated unit checks, contract test^H^H^H^Hchecks, comments where needed; clear docs, too.
8) Testers who code would not be obliged to do this kind of following-the-elephant work. Testers who code, toolsmiths, would be coding tools to aid with investigation, probes, data generation and visualization, to support *experiential* testing.
9) So, two important flavours of testing: one focused on the *discipline* of product development, addressing the question "Are we building something reasonably close to *what we think* we're building?" That's largely the domain of automated unit checks, compiler checking, etc.
10) Then a different flavour of testing, focused on *realization*, in two senses: realizing—achieving— our goals for the product, and realizing—recognizing—that there might be problems in it. The question here: "are we aware of every *elusive* bug *that matters*?" *Deep testing*.
11) It seems to me that deep testing should include lots of tool use, if we use tools to aid in deeper exploration of the product and the things around it. Tools that help us see patterns in output; that help us to generate and perturb and diversify data in interesting ways.
12) Tools that help us to model the product and its interactions in various ways. Tools that help us see connections between parts of the product, or between the product and outside systems that we wouldn't notice otherwise; and tools to make that stuff easier to grasp, too.
13) Such tools could help us to observe, probe, interact with, analyze and understand the product and its output; that would be all to the good. But all that experience WITH the product doesn't help much if we don't get experience OF the product. We need that, for our customers.
14) @jamesmarcusbach has said that much testing these days seems informed by an elaborate conspiracy to make sure that no one actually interacts with the product. One antidote might be: are your automated checks focused on risk? On finding problems? Or are they just GeMPuB?
@jamesmarcusbach 15) Automated checking does take time and skill to set up. The touted justification is that checks "save time for exploratory testing". That could be true IF they're inexpensive to prepare, maintain run, and interpret; AND if they're more than just GeMPuB. But if they're not...
@jamesmarcusbach 16) So: good checking is fine, but we must be cautious and skeptical, focused on cost, value, and risk associated with bad or weak checking. Why? Because it our opportunities to perform experiential testing on the product and to find problems that matter to real people. -30-
