Thread by @michaelbolton

1) Another day, another "white paper" with some very peculiar ideas about testing and development. I won't identify the paper; there will be another one like it tomorrow, and again the next day. It seems to me that as a craft, we should try to address this stuff. Help me out.

2) The paper refers to "a single source of truth". That might be appropriate for a religion or a cult, but software that is designed to help people must recognize multiple truths that come from multiple sources in multiple forms. Single sources of truth lead to multiple problems.

3) No document, tool, artifact, or person can faithfully and accurately represent all of the needs and desires of everyone who is affected by the software. Overfocusing development and testing on a "single source of truth" means oblivion to risks and problems outside its scope.

4) It's not easy to deal with a complex landscape of users, needs, desires, stakeholders, risks, and problems. We must accept that, though. Many things that are worth doing well aren't easy. Ross Ashby's Law of Requisite Variety reminds us that only variety can overcome variety.

5) And as Tom Waits (!) said once in an interview, "'truth' is not a word that should be used in the singular." The "single source of truth" leaves out other truths, and alternative sources of them — whereby we are blindsided by problems that hurting and not helping people.

6) Many truths come in the form of *tacit knowledge*—things that we know but nonetheless have not been told; things that we have learned and have in our head, that are incorporated in us (example: touch typing, using door knobs), or that we know through immersion in a culture.

7) Testing is far more powerful when we acknowledge the role of tacit knowledge in our capacity to spot problems. In Rapid Software Testing, we highlight lots of consistency principles that form the basis of oracles—means by which we recognize problems. Many of these are tacit.

8) Sure: some problems are identifiable by observing an inconsistency between the product and some document or artifact. Yet it's easy to recognize lots of problems without specifications. We all KNOW that, because we all encounter and recognize problems every day.

9) I think by "single source of truth", white paper authors are suggesting some kind of reasonably reliable, authoritative specification, or a repository of user stories, or something like that. Not a bad idea on the face of it. Why not call them what they are?

10) The "single source of truth" notion suggests that all of the things that the product should do are explicit, and that anything NOT explicit is irrelevant. That's a problem, because bugs aren't specified; they emerge as we're developing and building the product.

11) (To be clear: a bug is not necessarily a coding error. A bug is *anything about the product that threatens its value*. Or, if you like, a bug is something that bugs somebody who matters. So that could be a problem for a customer, a developer, a tester, the business...)

12) (A bug can manifest in running code, but there are also bugs in our descriptions and representations of the product and its context, and in our ideas about the product. Ideally, we'd like to swat bugs in those things before they get turned into products that bug people.)

13) Of course, bugs can appear in "single sources of truth". However, calling them that suggests that they cannot be wrong, incomplete, inconsistent, out of date, or controversial. This creates psychological friction and misdirection away from something important: skepticism.

14) Skepticism isn't cynicism, which is the rejection of belief. Skepticism is the rejection of *certainty* about belief. If we're building a complex product to serve lots of people in lots of situations, we must remain professionally uncertain that we've got everything right.

15) Now, when we're in the builder's mindset, it's hard—REALLY hard—to switch between the optimistic mindset required to build something and the more pessimistic, uncertain mindset required to question things that we've necessarily believed along the way.

16) One heuristic for getting around that problem is to be tolerant, welcoming, of other sources of information and truths; not to harden the illusion that there's a single source of it. Another: having someone *full time* in the role of looking for trouble, problems, bugs.

17) Both this (generic) white paper AND much of the history of testing focus on *confirmation* and *demonstration*, showing that the product is consistent with this single source of truth. That shallow testing is OK, but misses the point of why we might want to test more deeply.

18) When there's risk, we *probably* want to test more deeply to find problems that are hidden, rare, subtle, intermittent, emergent... and not obvious enough to be noted either in the single source of truth OR in the designs, conversations, and xDD artifacts that led to it.

19) How do we do that well? It seems to me that we need to develop rich models of the product, and to continue to expand and refine them, consciously and deliberately. developsense.com/blog/2014/07/h…

20) We need to develop a rich set of ideas about who might get value from the product. For a product intended for people to use, talking about "the user" leads to the same problem as "single source of truth" does: there's never only one. Personas, per @mralancooper, might help.

@MrAlanCooper 21) Even personas won't cover anything like all of the ways in which people might use the product and encounter problems. How can we do that better? Try considering customer support and documentation people as part of that "whole team" we like to talk about. They ARE members—no?

@MrAlanCooper 22) To anticipate problems, or to find problems that matter, we need to develop rich sets of ideas on ways that people might find value in the product—or, in the absence of those characteristics, how value might be threatened. Example: satisfice.com/tools/htsm.pdf…

@MrAlanCooper 23) To find problems that matter, we need to develop means by which we recognize problems. Over-focusing on inconsistency with a "single source of truth" is a very limited way of doing that. There are many more ways. Example: developsense.com/blog/2012/07/f…

@MrAlanCooper 24) Which brings us to the issue of traceability. This same white paper suggests that tests (by which they mean automated checks, I think) (satisfice.com/blog/archives/…) should be traceable to user stories. Notice that user stories are typically utterly incomplete.

@MrAlanCooper 25) In most user stories, nobody is ever interrupted, distracted, naive, confused, under pressure, impatient, disabled, outside of wireless access. Nobody makes human mistakes. Nobody closes the damned laptop lid. The characters in user stories might as well be drones, robots.

@MrAlanCooper 26) Testing that's entirely traceable to the user stories is great if you believe that users are as deterministic, unimaginative, and trouble-free as the stories typically are. Real life isn't like that, though. Real life, and real users, are complex, volatile, messy.

@MrAlanCooper 27) Therefore: testing (REAL testing, not just confirmation and demonstration) needs to identify problems even though the user stories (requirements, specs, marketing material, single sources of truth, etc.) don't say anything about them — or are just flat-out wrong.

@MrAlanCooper 28) Traceability to a single source of truth is therefore awfully limited. Consider instead *test framing*, being able to connect the outcome(s) of a test at the bottom to business value and business risk at the top, via an unbroken chain of logic. developsense.com/blog/2010/09/t…

29) The kind of testing touted in this white paper focuses on a DevOps context and on shallow testing *that has some value*; testing that uses tools and automated checks to help us spot some problems. That's cool. But rather like compiler checking, that's only a start.

30) Here's how I'd like to see people think about DevOps: as a way to build and check products efficiently not to allow us to release quickly, but also to allow us to do deep testing with a fresh build at any time. AND TO INCLUDE DEEP TESTING, which is the bit that gets left out.

31) Deep testing (testing that maximizes the probability that we will find ALL the important bugs in the product; testing that may take significant skill, effort, preparation, time, skill, data, etc.) isn't discussed, mentioned, or acknowledged in white papers like this. Alas.

32) And yet it's another white paper purportedly about quality. When we're developing products, and we want to do a good job of it, I think it's awfully important to talk about deeper threats to quality, and how to discover them even though we tried our very best to prevent them.

33) If you've hung with me for this long, thanks. That's it for today. I hope it's been helpful.

Now, a word from our sponsor: me. If you'd like to develop testing skill for yourself or your team, consider Rapid Software Testing. rapid-software-testing.com