1) Still having trouble logging in to Facebook, but for mundane reasons. See, apps with 2FA send an email or a text message when you ask for a password reset. But unlike machines, people are impatient, and mash that "request reset code" button multiple times.
2) As a consequence, several reset codes get sent. Because of email latency, who knows when the most recent request has been fulfilled? So the most recent code in the email box might not be the most recent one sent, so things get out of sync.
3) This gets a richer when Messenger notices trouble. I get email from Facebook: "We noticed you're having trouble logging into your account. If you need help, click the button below and we'll log you in.” Then there’s a one-click button that will allow me to log in to Facebook.
4) EXCEPT that button uses the protocol from *Messenger* (which uses a six-digit authorisation code), so the login to Facebook fails, because Facebook wants an eight-digit auth code! I can pretty much guarantee that the automated checks for login don’t consider such scenarios.
5) Well, somehow this all gets sorted, and I get a usable code FOR Facebook, FROM Facebook, and I change my password. But then, logging in via my browser (which is the user journey here), I get asked "Please enter the login code from your Code Generator below". Code Generator?!
6) (While trying to mark the quoted text from the previous tweet, I find that someone has rendered the text on that page unCopyable. But I digress.) I can pretty much guarantee you that this scenario is not covered by automated checks. I agree that it would be silly to do that.
7) Such checks would be super brittle, annoying to develop, and worse to maintain. But I can also pretty much guarantee that this scenario (and many variations on it) are not covered by testing either, even though Facebook and Messenger are owned by the same company.
8) I would offer good odds that there's a log-in-to-Facebook group, and a log-in-to-Messenger group, and they're on different floors in different buildings, maybe even in different countries, and they almost never talk. Plus there are equivalent monitoring groups, also separated.
9) No matter what, monitoring groups are overwhelmed with data. If this affects one person in a thousand, Messenger’s user base means 1.3 million instances of the issue, with dozens of log file entries for each instance. Without knowing the scenario, try making sense of it all!
10) Testers will miss problems affecting real users when testers are steered towards confirming that everything's okay. Yes, Dear Software Folk, that’s true for your organisation too. And it’s especially true when that confirmation is obtained in instrumented and unattended ways.
11) This is NOT to say that automated checking is a Bad Thing. Automated checking, especially at low, machine-friendly levels, can be a Very Good Thing. Especially when it’s a routine part of the developers' diligent discipline—just as the original eXtreme Programmers advocated.
12) And it’s not to say that experiential and attended testing will find all the problems that elude even a highly disciplined development process. No one knows how to find all the bugs in non-trivial systems. That’s a given; it’s epistemically impossible, to say it fancy-like.
13) But here’s the thing (and it’s a software industry pandemic at the moment, so it seems): there is this myth that automated checking "saves time for exploratory (experiential, attended) testing". It's a very seductive myth, because of an observational and time-accounting bias.
14) When a buhzillion automated checks run, it’s tempting to say "Look at all that testing! And look how FAST it’s going!" And those checks are indeed blindingly fast, but that’s not all that’s happened. We're seeing a ship in a bottle, but NOT what it took to get it in.
15) To get those automated GUI checks to run *at all* can take enormous amounts of time. To adjust them when they get out of sync (because the product changes) can take just as much time. To deal with inconsistent behaviour (because of inconsistent environments)—even more time.
16) Some inconsistent behaviours are often written off as "flaky checks" after a while, but at significant sensemaking and interpretation costs. Then everyone gets frustrated because they realize that the tools or frameworks are problematic, so now there’s migration cost.
17) Now, it’s true that in the course of automating checks at the GUI level, some bugs will get found. Any interaction with the product, by any means, affords the chance to stumble over *some* bugs.
18) Of course, if the task is to automate the GUI, a large proportion of those bugs will pertain to problems with automating the GUI. Those are testability problems without being usability problems; no end user on the planet cares about element ids in the DOM. Well, very few.
19) A handful of the bugs found while developing or maintaining automated GUI checks will be genuine bugs. These will get fixed, whereupon the automated GUI checks are relatively unlikely to detect that bug (especially if the developers implement lower-level checks for it.)
20) Other bugs will be ignored and swept back under the rug, and then something a little more insidious will happen: the checks will be changed so that they run green. This becomes a dirty secret, but everyone's in on it, because dealing with bugs is a pain in the ass.
21) Even the most diligent developers heave a sigh when a bug is noticed and reported. Managers don’t want to have to deal with the complicated decisions about how to deal with bugs. Often there’s subtle yet significant social pressure for testers not to report "too many" bugs.
22) Considering the political and social pressure, it’s unsurprising that many testers will keep their heads down and keep plugging away at the programming, debugging, maintenance, interpretation and occasional migration work around automated GUI checks. Plus it’s mandated.
23) Here's the kicker, though: all the work that I described above largely disappears into the Secret Life of Testers and the Secret Life of Automated GUI Checks. It’s largely invisible; taken for granted; unobserved and unquestioned except in a handful of development shops.
24) There are very few incentives to raising these questions:

A. How much of our time are we spending on the whole effort—not just the execution time when we run the automated GUI checks, but all of the effort around the care and feeding of them?
25) B. What *important* bugs are we finding with our automated GUI checks? C. What important bugs might we miss? D. What important bugs have we missed already, and what have we learned from that? E. What kinds of testing work are we NOT doing; what is the *opportunity cost*?
26) Opportunity cost is crucial. Every moment a tester spends on programming or fixing another GUI check is a moment that she can’t spend on learning about the customers' needs and desires; developing tools to generate rich test data…
27) …collaborating with customer support people for test ideas; reviewing relevant and important regulations or standards documents; visualising and manipulating output data for consistencies and inconsistencies; following up on mysteries; investigating new tools…
28) (Tool vendors in particular are eager to sell stuff to management that "will save 80% of the testing time"—which is always a bogus claim, since they don’t know a damned thing about how *your* organisation is spending *its* testing time.)
29) (They apply the 20/80 rule, touting a tool that addresses 20% of your actual testing work, and you spend 80% of your time dealing with the tool, leaving you with 20% of your time to do the other 80% of your work. Credit to @jamesmarcusbach; we call it the Oterap Principle.)
30) But in all this, the opportunity cost really lands on *attended* *experiential* testing—that is, testing wherein the tester is present and engaged, and in which the tester’s encounter with the product is practically indistinguishable from that of the contemplated user.
31) The prevailing ideas behind automated GUI checking are that it be *unattended* and *instrumented*—some medium gets in between the product and a human being's naturalistic experience of it. That can have some value, but getting there reliably is difficult and expensive.
32) And all that, Dear Testers, is why customers encounter problems that automated GUI checks won’t reveal. It’s not that automated GUI checks *replace* testers; they *misplace* testers' focus, and they *displace* experiential testing.
33) Now, of course, I could be wrong about all of this. Maybe in your shop, there is regular scrutiny of where testing time is going. Perhaps your organisation has done a diligent job of examining the value of your automated GUI checking, and contrasted it with other approaches.
34) Maybe you do diligent scenario testing, where testers have the freedom and responsibility to explore the product with maximum agency and without a prescribed set of steps to follow. Maybe you have time to surf the customer forums and let user experiences flavour your testing.
35) Maybe you're a toolsmith with freedom and responsibility to craft all kinds of useful tools for yourself and other testers, instead of automating the GUI to reproduce steps that never found a bug in the past and won’t in the future, either.
36) Maybe you're collaborating with your developers, and they are taking on the responsibility of output checking at the unit and integration level. That’s not only reasonable but also important if they’re mandated to produce reliable code that they themselves can warrant.
37) Maybe, because of all that, you’re doing really minimal GUI checking because you don’t need much of it. But maybe you’re up to your eyeballs in it, because (again credit to @jamesmarcusbach) when it comes to technical debt, automated GUI checking is a vicious loan shark.
38) And finally, a word from our sponsor: Rapid Software Testing Explored (timing friendly for Europe, the UK, and points east) runs November 22-25. Learn about RST here rapid-software-testing.com; about RSTE here rapid-software-testing.com/rst-explored/; and register here: eventbrite.com/e/rapid-softwa…
39) Epilogue: I eventually got into Messenger first, then Facebook having changed my password, but never to my knowledge having successfully supplied the second authentication factor. That’s an investigation for another day.

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Michael Bolton

Michael Bolton Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @michaelbolton

24 May
18. Learning about problems that will threaten value to customers certainly requires scrutiny from the builder/insiders' perspective. The code shouldn't be inconsistent with builders' intentions. And among themselves, the builders can be pretty good at spotting such problems. /19
19. But to be really good at spotting problems that threaten customer value requires builders' savvy PLUS a significant degree of estrangement from the builders' set and setting, and requires immersion in the consumer/outsiders' form of life. And there's a problem here. /20
20. The problem here is that, with a few exceptions, *deep* immersion in the user/consumter/outsider form of life isn't easy to come by for most testers. Some testers have come from the product's intended domain; that can help a ton. Others study the domain deeply; also good. /21
Read 6 tweets
24 May
5. This is not to say that testers can't be helpful with or participants in checking. On the contrary; we likely want everyone on the team looking for the fastest, most easily accessible interfaces by which we can check output. Testers know where checking is slow or expensive. /6
7. But here's something odd: testers don't always point out where checking is slow, difficult, or expensive—and, just as bad, maybe worse—where checking is misaligned with product risk. I suspect there are some fairly gnarly social reasons for this goal displacement. /8
8. In some organizations, testers prestige is based on programmers' prestige. Do you write code? Then you're one of the cool people. You don't? Then who needs you, really? This is a mistake, of course, but it's true that testers don't produce revenue. /9
Read 6 tweets
24 May
The tester’s mission is not the builder’s mission. The builder's mission is to help people's troubles go away, envisioning success.

The tester's mission is to see trouble wherever s/he looks, anticipating failure. The tester’s mission helps to serve the builder’s mission. /2
2. The tester's mission helps to serve the builder's mission in at least two ways: a) in noticing where initial problems persist; where the builder's work might not be done yet; b) in recognizing new problems that have been created while attempting to solve the initial ones. /3
3. Some problems can be anticipated, and then noticed by performing checks in a rote or mechanistic way. That kind of checking is part of a disciplined development and building process; very good to do, but it doesn't hold much hope for identifying many unanticipated problems. /4
Read 4 tweets
22 May
20) If you present testing as a complex, cognitive, *social*, *self-directed*, *engineering*, *problem-solving* task, I guarantee more programmers will happily throw themselves into it. And, if you have testers, MORE TESTERS WILL TOO. So what is the problem to be solved here?
21) One big problem is: we have a new, complex, technological product that we intend to help solve a problem; and that we may not understand the problem or our solution as well as we'd like; and whatever we don't know about the all that could bite our customers and could bite us.
22) Finding problems that matter in that product is greatly aided by access to rich models of the product itself; of customers, how they use it, and what they value; of who else might be affected (support, ops, sales...); of coverage; of recognizing possible trouble; and of risk.
Read 5 tweets
22 May
15) There are ways of addressing those problems, but I don't think an appeal to quality is enough. Developers are already satisfying lots of quality criteria—it's just that they're specific quality criteria that are important to some managers: SOMETHING, ON SCHEDULE, ON BUDGET.
16) When programmers are satisifying those quality criteria, it's not helpful to suggest that they "learn about quality", or worse "learn to care about quality". They already care plenty about quality; but maybe they rate some dimensions of quality different from your priorities.
17) If testers and managers treat testing as a rote task of confirming that something works, it's inevitable that programmers will find it tedious and boring: they KNOW it works. They built it, right? Why would they build something and then give it to others if it didn't work?
Read 5 tweets
21 May
1) When managers say "testing is everyone's responsibility", ask if they're supporting or mandating developers to perform experiential testing, read testing books, take testing classes, study critical thinking, read bug reports from the field, set up test environments...
2) Ask also how many developers are hurling themselves towards these activities. Some developers (interestingly, the very best ones, in my experience) will be quite okay with all this. Other developers won't be so enthusiastic, and that might be both explicable and okay.
3) It might be okay for developers not to be too interested or engaged in deep testing, because certain kinds of testing are a) a side job for developers and b) really disruptive to the developers' forward flow. For developers, shallow testing might be good and sufficient.
Read 14 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!

:(