Thread by @noahsussman

@alanpage It's very telling that manual testing is seen as cheap.

IME this perception exists because (as I have repeatedly seen at clients) the absolute most junior people are hired as testers in order to keep costs down. Because they’re so junior they don’t contribute very much. 1/2

@alanpage 2/3 They stay junior because they have no mentorship (because mentorship is expensive).

The “5x to 15x” cost of automation proposed by the OP fits with this model: median developer pay is around $150k US. A team of five devs then costs ~$750k annually (considering TC only here)

@alanpage 3/4 Glassdoor lists “QA Analyst” roles as starting at $18k US. I’m in NYC so this is lower than I’ve ever seen. Let’s assume QA roles start around $25k, which is just enough to rent a room and live paycheck to paycheck in NYC.

The cheapest QA analyst costs 17% of a dev salary.

@alanpage 4/5 So it’s totally feasible to hire SIX entry-level QA analysts for the cost of ONE reasonably experienced developer.

Add to this that test automation is a senior dev activity. It isn’t taught in school nor in boot camps. Nor can you learn it at just any dev job.

@alanpage 5/6 So I’m comparing the lowest QA salaries to the median dev salaries for reason: a dev out of college or out of coding boot camp doesn’t have the skills needed to build a continuous integration and test automation system let alone build it at scale.

@alanpage 6/7 Aside from the senior-level skills (not just technical but also *social* or “soft” skills) that are *required* to build CI and customize an xUnit framework, it’s also the case IME that one dev alone cannot manage such tasks at scale, regardless of that dev’s level of skill.

@alanpage 7/8 So to do automated testing at all (at a scale bigger than an early-stage startup) it is necessary to have at least two senior devs who specifically have experience in automated testing. Just pulling two senior devs off of customer-facing application development doesn’t work.

@alanpage 8/9 So to even discuss deploying and supporting CI and xUnit and GUI automation at scale we are talking $300k and that’s just for personnel, it doesn’t count vendor contracts or hosting fees. Nor does it count conferences and training, which are things senior devs will want.

@alanpage 9/10 Meanwhile if I go with the cheapest QA Analysts money can buy (according to Glassdoor) then I can conservatively have a team of TEN plus a manager. More if I just make one of the analysts the manager even though they have no previous experience.

@alanpage 10/11 So the cost of manual #testing is rightly perceived to *potentially* be, as the OP says, “5x to 15x” less than testing that involves automation.

But the potential depends on hiring the most junior testers, because they are the ones who come so cheap.

@alanpage 11/12 What’s the point in hiring only entry-level or extremely junior people into the #testing role? How can inexperienced people with no mentorship deliver the same or *better* ROI as experienced senior devs?

Many would say “they can’t.” That again is an act of foot-shooting🦶🏼

@alanpage 12/13 What is the value of #testing? What is the deliverable from QA?

If you answered along the lines of "to improve customer experience" or “to increase product quality and decrease risk” congratulations you have been spending too much time around consultants and executives! 💀

@alanpage 13/14 The purpose of QA is deniability. This is not true in every org but it *is* the default and anyone who tells you otherwise is trying to sell you something (or is a toxic narcissist who wants your attention, or both).

@alanpage 14/15 Most organizations that have a QA team, in my experience, do so because their investors expect it. QA is one of the things you are supposed to have if you are a real corporation that does its dilligence.

Now let's look again at why cheap junior QA teams are so popular.

@alanpage 15/16 If the purpose of QA is to demonstrate that (by having a QA team) you are doing your full diligence as a corporation then it does not matter how well your QA team does at "improving customer experience" full stop.

@alanpage 16/17 Customers can be happy with really shitty products. Customers sign multi-year contracts that lock them in regardless of product shittiness. It's absolutely not the case that "quality" makes or breaks a product. It can be so in specific cases but is NOT TRUE generally.

@alanpage 17/18 Otoh one thing that absolutely *does* make or break a product is how much its *investors* believe that the corporate staff are doing their dilligence. An org without a QA team looks bad. An org with a QA team is at least doing the basic thing everyone else does.

@alanpage 18/19 Seen in this light the ROI of an extremely junior or all-entry-level QA team is clear: all that is needed for *investors* to be satisfied, is that the QA team exists. Hiring expensive experienced testers and automating drudgery doesn't add any value in cases like this.

@alanpage 19/20 Corollary: This describes the vast majority of corporations (the vast majority of which are not even software companies, they just "needed a web site / mobile app / AI offering"). The vast majority of corporations do not see QA as necessary to customer happiness / revenue.

@alanpage 20/21: When QA is seen as part of dilligence rather than a path to quality that is for one thing COMPLETELY NORMAL AND TO BE EXPECTED and for another thing invalidates almost all the received knowledge about QA and #testing.

@alanpage 21/22: An overwhelming majority of the discourse on #testing (including the OP's article) center on providing a completely abstract and non-quantifiable deliverable called "high-quality software."

@alanpage 22/23: The idea that #testing's ROI is to help orgs in the pursuit of "high-quality software" whatever that is (because there is no way to measure it nor even to describe it concisely) is one of the most egregious and widespread examples of the Streetlight Fallacy in all of tech

@alanpage 23/24: It would be hard to impossible to get a conference crowd excited about "due dilligence" which is all that most orgs expect from QA. It also is not generally true that ALL orgs want only dilligence. Some want a mix of dilligence AND improved quality. It's complicated.

@alanpage 24/25: Because it is both complicated and unglamorous to describe the real ROI of QA (demonstrating due dilligence to investors plus maybe improving customer experience) the #testing community seems to have retreated into focusing on "quality" which is much easier to talk about.

@alanpage 25/26: The focus on this abstract concept of "high-quality software" coupled with the very real LACK OF DEMAND for testing-as-a-way-to-improve-customer-experience seems to have led to a deep stagnation marked by illogical cult-like behaviors.

@alanpage 26/27: "Cult-like" is strong language outside of dev circles (where we accuse each other of it on a daily basis lol) so I need to give some solid examples. What about the meme that "manual testing and automated testing are mutually exclusive"?

@alanpage 26/27: One of the characteristics of a cult is that its members are asked to reject the evidence of their own senses. To reject reality. Stating that manual and automated testing are mutually exclusive is such an invitation.

@alanpage 27/28: What is meant by "automation" in the phrase "test automation"? Code. Code is automation. Computers are automation. So what then is manual testing? It is the testing of code using a computer. "Manual" software testing is not manual in the same way as carpentry or masonry.

@alanpage 28/29: The phrase "manual testing" as applied to software is an *abstraction* not a description of reality. Computers *are* automation. Using a computer is the same as using automation. Typing on a keyboard and using a mouse are ways to *use the automation* provided by computers.

@alanpage 30/31: The statement "manual software testing exists as distinct from automated software testing" is incorrect in that "manual" computer use is an *abstraction* not an empirical statement. There is no such thing as a non-automated computer.

@alanpage 31/32: The invitation to accept and evangelize an alleged empirical difference between "manual" and "automated" computer activities is an invitation to reject observed reality, and also to accept an abstract metaphor as fact. Cult-like af.

@alanpage 32/33: It makes total sense that faced with a complex, irregular, non-generalizable sociotechnical and *economic* problem like the ROI of QA, people would reflexivelly want to retreat into a comforting but unhelpful bubble. What is fucked is making that into a business model.

@alanpage 33/33 Anyway "manual" QA is 15x to 50x cheaper than "automated" QA because what is sold as "manual QA" is really just a demonstration to investors that a corporation is doing its basic dilligence which let's be honest is not all that expensive since there's so much demand for it.