1) Two large consultancies (or is it one?) have done a survey on "Continuous Testing". The press release is accompanied by an infographic (capgemini.com/us-en/wp-conte…) titled "Continuous
Testing 101:
What CIOs
need to know". The data is interesting. Please join me while I analyze it.
Testing 101:
What CIOs
need to know". The data is interesting. Please join me while I analyze it.
2) A page links to the infographic here: capgemini.com/us-en/news/con…. On one part of that page, it says the report is "based on 500 interviews with senior-level IT executives from large and medium-sized companies (over 1,000 employees)...".
3) But later on the same page (capgemini.com/us-en/news/con…), in the "Methodology" section, there's a difference worth noting. In that section, the claim is that report is not *based on interviews*, but *buttressed by a survey*. Specifically...
4) The report is "based on the *considered opinions* of several subject matter experts from Capgemini, Sogeti, and Broadcom, *buttressed by* the results of a global survey of 500 senior decision-makers in corporate IT functions." Who actually responded? The execs themselves?
5) It seems to me pretty unlikely that the execs themselves responded. It seems the incentives for them to do so would be pretty poor. So who responded? Execs? Their reports? Their admins, perhaps? Is this solid data from any of those, or a guess? A feeling?
6) The infographic says "Agile teams are spending 40-70% of their time on clarifying, communicating requirements." All of them? Later: "81% of Agile teams are wasting a third of their time or more in search of test data, its generation and management".
7) Later still: "76% of Agile teams spent a third of their time or more on fixing environments." If three data points are right, then 62.6% of all Agile teams are spending between 106% and 136% of their time on those three things together. That can't be right.
8) I could go on with other threats to validity; I'd have to look at the whole report for more. But an instant review: that first set of data points is like running a program that crashes when you start it up, or a pinata that showers candy upon being poked with a finger.
