1) There's a persistent belief that test coverage refers to code; the lines of code or functions or branches or branches and conditions touched during testing. Consider this instead: coverage is *how thoroughly we have examined the product with respect to some model*.
2) It's easy to believe that the product is simply the code, but it might be a good idea to think more expansively. Try this: the *real* product is the relationship between the stuff that we've built, the platforms upon which it depends, and the the person(s) interacting with it.
3) Test coverage for *that* product could not be obtained simply running the code. Test coverage would require modeling the components or the product, the functions it performs, the data it processes, its interfaces, its interdependencies, and the things people do with it.
4) Test coverage for *that* product would also require examining relationships between the product and time, in various aspects. Test coverage for that product would require modeling customers' notions of quality, and how the product fits with them—and how it might fail to fit.
5) We wouldn't be able to claim knowledge of a city just because we had ridden in a car on every street. To really know a city, we'd have to get out of the car and experience it in many ways; walking, observing, dining, doing business, interacting with others, et cetera.
6) A mere count of the number of streets we had traveled in the car, or the number of turns that we had made on our tour, would not represent our knowledge of a city's features (and of its problems) very well. To claim knowledge, we'd have to be able to describe things about it.
7) In order to provide a reasonable assessment of what we know about a city—how thoroughly we've examined it—a count won't do. A good description would have to be based not only on where we've gone, but about our conceptions and perceptions—our models, and how we've applied them.
8) Just as with quality, you can't measure coverage. You can apply very specific formal models to things that bear on coverage. You can discuss coverage, and you can assess it. You can evaluate it based on what you and your social group consider to be sufficient and lacking.
9) You can say that you've examined functional areas of the product in a deep way, or in a shallow way. You and your clients might decide that shallow testing is sufficient for the task at hand. Or you might decide to keep going and examine the product more thoroughly.
10) You can say you've thrown a requisite variety of data at the product for finding potential bugs in its data handling. Or you can suggest that more data-focused testing might be warranted to find deeper, rare, subtle, intermittent, or emergent bugs.
11) You might have developed a model of risk, and examined the product with respect to the idea of potential problems becoming manifest. That's risk-based testing. Risk coverage: how thoroughly we've examined the product with respect to some model of risk.
12) More generally: X coverage is *how thoroughly we've examined the product with respect to some model of X*. This suggests that we must talk not only about the "how thoroughly" part, but also about the quality of our models of the product and its relationships.
13) When we claim that we've done very thorough testing based on a lousy model of the product, we're more likely to have done lousy testing than thorough testing. Excellent test coverage requires a variety of excellent models, *including* excellent models of testing.
14) These articles are old, but still relevant, I believe. developsense.com/articles/2008-…, developsense.com/articles/2008-…, developsense.com/articles/2008-… They were written before @jamesmarcusbach began to describe coverage as "how thoroughly we've examine the product with respect to some model".
@jamesmarcusbach 15) Our description of coverage took years to refine until we were both happy with it. Note: learning how to think and speak articulately about testing often requires work, study, discussion, controversy, exploring, experimenting, and experiencing—just like testing itself.
@jamesmarcusbach 16) Based on our heuristic for describing code coverage is *how thoroughly we have examined the product with respect to some model of the code*. Huh? The code is just the code, isn't it? Well... "code coverage" usually refers to the code *we* wrote. What could might be missing?
@jamesmarcusbach 17) The code our organization wrote is not all of the code in the product. There's code in the operating system, in the hardware, in device drivers, in the processor, in third-party libraries. Code coverage tools don't cover all that stuff; they can't. But that might be okay.
@jamesmarcusbach 18) All coverage and all notions of it are relative to some model. So code coverage can be useful, especially for identifying what code has and hasn't been covered and (with really good coverage tools) where there might be calls to interdependent code that we didn't write.
@jamesmarcusbach 19) Let's not reject the idea of code coverage when we want to understand something about the code. But let's also avoid the mistake of believing that good code coverage means good test coverage. We need a requisite variety of models to cover the product thoroughly with testing.
