Thread by @michaelbolton

1) In its earliest days, API stood for "Application Program Interface"; now, mostly "Application Programming Interface". We might build and test APIs far better if we think Application *Programmers'* Interfaces. Programs alone never use APIs; people writing and using programs do.

2) It might be easy to think that programs use APIs, or that programs call APIs. But that's like thinking that drill bits use chucks, or that lamps use switches, sockets, and extension cords. *People* use drills and lamps—and their elements—as parts of integrated systems.

3) APIs—like everything else that gets built—are built from the perspective of an insider. That’s inevitable; the act of building something automatically puts you inside the builder’s perspective. Escaping that perspective is essentially impossible, until you forget building it.

4) Of course we can form ideas about what outsiders might need or desire, and build something with the intention of making it useful, helpful, powerful, friendly, or reliable to outsiders. But while building, we're unable to drop what we know already; to see it from the outside.

5) It’s a really good idea to review and examine and test our products from the inside as we're building them, and to confirm that we've just built is reasonably close to what we intended to build. If it isn’t, we could reasonably anticipate plenty of surprises and trouble.

6) But if we’re building something for other people to use, we must do more than confirm that it can do what we intended. We must also anticipate and address problems that outsiders will have with it—and how we might be fooled by our insiders' perspective on what we’re building.

7) This applies in spades to APIs, because the user of the API is never a program; that’s only an abstraction. The user is always a person—indirectly, an end user or stakeholder of a product; or more directly, someone using the API as means to help indirect users get stuff done.

8) I've been doing a fair amount of coding recently. Programming something non-trivial always includes some amount of *necessary confusion* as we encounter new problems to be solved, identify and apply technologies to solving them, and develop the solutions—and rinse and repeat.

9) But whenever I'm coding, I'm also experiencing a substantial amount of confusion that I perceive as being unnecessary: in particular, weird (to the outsider) mental models of the product; unhelpful or just plain wrong error messages; non-existent or lousy API documentation.

10) The source of each of these problems is easy to explain. Insiders already have the insiders' models of the product; they know how the product is intended to work, so they under-emphasize and under-imagine mistakes outsiders might make; minimal documentation—or none—suffices.

11) So the insider's perspective for a product tends to be "it works"—which actually only means "it *can* work"; "it appears to meet some requirements to some degree”; "it works for me, on my machine, and with my (insiders') knowledge". That perspective is intrinsic to insiders.

12) The outsiders' encounters with the product are always different from the insiders'. (The outsiders' perspectives are different from each other, too; that’s for later.) An API is an interface to the product's (presumably, typically) hidden internals—as it should be.

13) Checking that API calls return a correct result or a correct error code is reassuring for the builders, that’s okay. Builders should do that. But for testers, checking the output doesn’t really address a much bigger deal, which is that APIs will be used by outsiders.

14) To TEST a product does not simply mean to check its output. To test something is to evaluate it by learning about it through experiencing, exploring, and experimenting. And whose experience of an API is crucial? The experience of an outside programmer who uses it. Test THAT.

15) As with anything else, the real test of a product comes with people’s experience of using the damned thing. So, testers: are you simply checking the output the API your organisation is developing? Or are you trying to evaluate the experience of some contemplated user?

16) To TEST a product and its API means (among other things) to evaluate output from error conditions; not simply to check if you got some anticipated return code. Does the API return a description of what went wrong, such that it helps the outsider trying to troubleshoot?

17) An API I was using recently returned "insufficient rights to perform operation", which led me down a rabbit hole of investigating why admin status wouldn’t permit it. Turns out it was looking for a string instead of a JSON. Tell your user THAT, not something misleading.

18) When I ask another API for an element Web on a page, it returns a data structure that prominently features the string "located:false", which misleads outsiders into believing that the element was not found. Turns out that "located" refers to some aspect internal to the API.

19) When testing that API, "located==false" would be a correct result, based on insider knowledge (or something like it, obtained from exasperating amounts of trying and retrying and visiting Stack Overflow). But "located" means something else entirely to those new to the API.

20) Therefore, when building and testing an API, you *could* ask "Could this call or return value afford a different interpretation to outsiders than to the builders of the product?" But your insider's perspective will bias you towards your existing, established interpretation.

21) Therefore: have someone try to USE the API. Give them ideas for something to build, if they don’t have some already. Step back as they build it, and have them log the problems and confusions they encounter. When they do, avoid explaining confusion away. Prefer fixing the API.

22) When building an API, assume from the outset that other people, outsiders, will be using it. Document it from that perspective, anticipating that they are trying to create something that other people will use. Your examples should be based on that, not on isolated calls.

23) When testing an API, try building something with it. If a function returns an incorrect value, that's a obviously bug. But if a function call is named in a potentially confusing way, that's also a bug. If the documentation doesn’t address that confusion, that's also a bug.

24) If the documentation includes only atomic function calls, and doesn’t include examples of useful multi-step transactions: Severity 1 bug. If the documentation includes only installation instructions: Severity 0 bug. Yes, I'm talking about 85% of the projects on GitHub.

25) Testers: we can provide a super-valuable service to the project by keeping a careful record of our experience with trying to test the API—when our testing includes trying to use it. Every problem you run into is plausibly a problem someone else will run into. That is: a bug.

26) Every successful interaction you have with the API as you’re trying to test it is plausibly another element in examples and stories that could go into the API's tutorial user guide. Try using the API, record your experience, and the project gets documentation work for free!

27) Every bit of confusion you experience as you develop code that exercises the API is plausibly a bug in the interface. Every time that confusion is resolved poorly, slowly, or not at all by the documentation, that’s plausibly a bug in the interface or its documentation.

28) That, by the way, is an instance of an important testing heuristic: *Every testability problem is plausibly a usability problem; and every usability problem is plausibly a testability problem.* Something that is hard to test is often a pain to use, and vice versa. Report it.

29) I've heard people say, "Avoid testing from the user interface; wherever possible, test from the API instead." This is generally bad advice on several levels, not least because it obscures the fact that the API IS a user interface for one user—a programmer—to help other users.

30) The API is also a user interface whereby testers can get access to aspects of the product that might be much harder to test otherwise. So, for an API, consider not only end users, and not only programmers, but also testers as its users. Treat the API itself as a product.

31) Model the API and test through it in terms of structural elements, including its documentation; functions and features it offers and affords; data that it accepts, rejects, stores, retrieves, processes, puts out, deletes. Consider what it does and doesn’t provide access to.

32) Consider languages and libraries that the product AND the API might depend upon—and the programs that depend on the product and the API. Consider how people will encounter and use it, and problems they will have using it—and consider reasonably foreseeable misuse and abuse.

33) Consider how time affects the product. How’s the response time? What happens if things are requested out of order? All at once? Do strings of transactions or interrupted or disrupted transactions time out? Do session tokens expire?

34) Examine the API in terms of quality criteria you apply to the rest of the product—remembering that code itself won't experience or notice problems in the product. But people using code and people writing code will. If there's a product and an interface, there's always a user.

35) For a guide to testing through an API, I offer developsense.com/blog/2018/07/e… (four parts).

Rapid Software Testing Explored for Europe runs May 25-28. Register here: rapid-software-testing.com/attending-rst/, RSTE for the America will happen June 21; registration links coming soon.