Thread by @duarte_vasco

Thanks to @DrAgilefant and friends, just got my hands on a thesis that shares some enlightening insights into how common and impactful estimation errors are #NoEstimates
I will be publishing more of what I read in this thread.

"Outliers are so frequent that the noise drowns out the signal in the data"
In other words: even if you have data from "actuals", you don't really know if you will be late because the outliers are only visible too late and have a huge impact on delays
#noestimates

Kmart may have gone bankrupt (at least in part) due to a failed 1.4bn USD IT project.
One more case where estimation, did not - at all - save a company. Indeed, the errors were so large (both business estimates and SW estimates) that the company went bust
#NoEstimates

Researches, when faced with a large amount of extreme cost/schedule overruns in IT projects, choose to ignore those, and use only the observations that fit the "normal" distribution (thin-tailed). Therefore HIDING the fat-tailedness of estimation errors! #NoEstimates 🤦‍♂️

Turns out that the studies into estimation accuracy and track record excludes "outliers" from the study, which in practice means they are *FAKING* regularity and low variance, when in fact variance in estimates is much larger than reported in those studies! #NoEstimates 🤦‍♂️

The thesis I'm reading finds that the data shows a very large bias towards projects being late, or very late (right-tail skew). Which is exactly what I've been saying all along, and a great reason to find alternatives to estimates! Like #NoEstimates for example

No surprise, but it bears repeating: Estimation *DID NOT* help in securing the investments in IT projects. Here's a sample from the thesis: Projects under study were overwhelmingly skewed towards cost/effort OVERRUNS *AND* benefit SHORTFALLS! The unholy triad! #NoEstimates

In other words: there's NO EVIDENCE that estimates help with cost control, schedule control or value delivery!!!!!
#NoEstimates (thread, check it)

A detailed analysis of schedule performance: Half of the projects were delivered under 25% Effort OVERRUN! Which means that 50% of the projects had 26%+ Effort OVERRUNS (median) #NoEstimates

The mean on Effort OVERRUN is 2.18 times the *planned* effort. In other words, your "average" project (if such a thing exists") took more than 2.1x the effort when compared to the initial plan

Cost-wise the picture is *grimmer*. There are 3600+ projs in this sample (largest sample in the study).
The "average" project cost 2x more than estimated (mean), 50% of projects had similar costs to what was expected (median), which means 50% had higher costs #NoEstimates

Schedule-wise the situation is not as bad, but still pretty awful: even if "average" project was "on time" (mean), 50% of projects took 1.38x longer than expected (median). Sample N=965
#NoEstimates

Let's translate this into a bet: If you bet on estimation to "secure" your IT project investment, you may deliver on time or cost, but...

...are likely to put in 25% more effort. However, *if you fail* at estimation, you might *DIE* (or bankrupt the company). This is the grim reality of estimates! #noestimates
The data has spoken: Estimation = #RussianRoulette !

This finding is interesting (if predictable): The risk of cost OVERRUNS grows considerably with duration (left graph), but not with estimated cost (right graph). IOW: the longer the project, the more likely it is to cost a lot more than expected! #noestimates

An interesting insight that will (no doubt) piss off the estimators: The study did not find that the "root causes" for schedule overruns explain the overruns. In other words: the root causes claimed by experts are not really the root causes for estimation failure!
#NoEstimates

The link to the full thesis: ora.ox.ac.uk/objects/uuid:9…