1) Pretty much as long as I've been in this business, I've been griping about *reification* That's the problem of turning ideas and concepts and images — and people, for that matter — into *objects*. Reification is pandemic in software development, and it's pandemic in testing.
2) Reification causes people to confuse ideas with more material things that represent them. Those material things provide *representations* of sets of ideas. By nature, those representations are to some degree inaccurate; they're definitely incomplete; and they're time-bound.
3) No document, diagram, string of words, sketch, painting, graph, chart, table of numbers, body of software code — nor any combination of them — can represent faithfully, accurately, and completely what you thought when you thought it; certainly not what you think right now.
4) It is a serious mistake to confuse the product requirements *document* with the *product requirements*. It is a serious mistake to confuse the test strategy *document* with the test *strategy*. It is a serious mistake to confuse the test script (or test code) with the *test*.
5) These serious mistakes are easy to make because requirements documents, test strategy documents, and test scripts or test code are *useful* to some degree, for some purposes. They help to foster communication of *some* ideas. As maps, they give us *some idea* of the territory.
6) There's a famous story of an army unit that's lost but finds its way home despite following the wrong map. A suggested moral of the story is "when map and territory disagree, believe the territory." But Karl Weick, in /Sensemaking in Organizations/ offers a different moral.
7) Weick points out that maps (and plans, and for our purposes here, documents and other artifacts) help to *animate* and *orient* people. A map helps to trigger the process of sensemaking when people are confronted with a difference between map and territory. They THINK and DO.
8) That is: in the story, the big deal is not *believing* the territory; the big deal is the continuous, forward-and-backward, social, *heuristic* process of *making sense* of the territory *because* the map and the territory disagree. (Yes; a lot of emphases, there.)
9) The story, according to Weick, raises the intriguing possibility that when you're lost, *any old map will do*; that when you need a strategic plan, any old strategic plan will do. Maps and plans—the explicit, the artifacts—help to get people started on making sense of things.
10) The trouble is, Weick continues, managers (but I would say people generally) don't realize that it's what people think and what people do that matters. But managers (and other people) keep giving credit to the wrong thing: the plan (the document, the artifact, the script...).
10) Then, Weick says finally, the managers are baffled when more planning improves nothing. I infer he means mostly the development of more specific, explicit, time-bound, planning documents.
In software development and in testing especially, we can see this problem playing out.
In software development and in testing especially, we can see this problem playing out.
11) People spend endless amounts of effort on making sure that artifacts agree with each other. They mistake the product-as-code for the idea of the product, and the requirements document for the idea of the requirements, check that code and document agree, and leave it at that.
12) When we're developing the product— but especially when we're *testing*—it seems to me that we should be looking for *problems* in the relationships between *requirements* and requirement *documents*. Those problems feed problems in our concepts AND implementation of products.
13) When we're testing, we should be looking for problems in the relationship between what people require, need, or desire from the product on the one hand; and on the other, the job that that product actually does—and doesn't do—to fulfill those requirements, needs, and desires.
14) The material representations of what people want from the product are limited and frozen in time. We must remain alert to omissions, inconsistencies, or changes between artifacts and desires. One way to help notice: quick, inexpensive, alternative, shared representations.
15) Another way to help notice: *train people's minds* to developer rich mental models of desires (quality criteria), potential problems (risks), and ways of modeling the product (factors). Help people to notice inconsistencies and problems (oracles) *without needing artifacts*.
16) Instead of treating artifacts as *sources of truth*, treat them as *sources of rumours*. Treat them with the same sense of suspicion that you'd apply to a rumour. Don't assume that they're factual; check them out; investigate them. Gather evidence for or against. TEST them.
17) Don't just focus on the product as a set of artifacts (Is there are button here with this label? When I push it, does a box appear with some text in it?). GET EXPERIENCE with the product. *Challenge* the idea that it solves actual problems that actual users actually have.
18) Notice when you're reifying people. Here's evidence that you might be doing that: referring to "the user". These isn't just one! There are lots of users. Some are distracted. Some have disabilities. Some will be confused. Some will be expert, and therefore impatient.
19) Whoever users might be, their needs and desires cannot be represented completely, in a timely way, by any document—even if that document is error-free, which it inevitably isn't. So use the requirements documentation as *hints*, not as infallible stone tablets from the gods.
20) Our job as testers is NOT (I would argue) to make sure that the product code is consistent with the requirements documents. Our job is to find problems in the relationship between the services provided—or not provided—by our products and people who want to have stuff done.
21) David Platt gave a wonderful heuristic in his book and talks on Why Software Sucks: unlike geeks, most of the time, people don't want to drive somewhere; they want to BE somewhere. People don't want to use your software; they want to HAVE USED your software.
22) So I say to you, friends: we can reify, comparing artifacts like requirements documents and automated checks (and, lord help us, formally scripted procedural test cases) to other artifacts (like output from a product). Or we can look for problems that matter to people.
23) Although sometimes helpful, comparing artifacts to artifacts isn't the point. Avoid reification. Comparing ideas of our products and services with *experiences* of our products and services is the point. Embrace experiencing, exploring, experimenting. Embrace TESTING.
24) Rapid Software Testing Explored is moving online, June 29-July 2, with timeslots oriented towards India, Europe, the UK, and early risers in North America. Save the dates; registration information coming soon.
25) Rapid Software Testing Explored will be available online for North American people who don't get up crazy early July 14-17. Again, watch this space for registration information.
26) And Rapid Software Testing for Managers will be available online August 12 - 14. All these dates have just been set, so registration information is on its way.
