I've written several times about my personal dislike of the Standish Reports.
- Standish Report and Naive Statistics
- Finally, A Challenge to the Standish Report
- Project Failure Rate
The current assessment of the Standish Report is:
"The Rise and Fall of the Chaos Report Figures," J. Laurenz Eveleens and Chris Verhoef, Vrije Universiteit Amsterdam, IEEE Software, January/February 2010, pp. 30-36.
The core findings are:
- There are misleading definitions - the definitions used by Standish are based solely based on estimation accuracy of cost, time, and functionality. Standish labels projects as successful or challenged, suggesting much more than deviations form their original estimates.
- Unrealistic Rates - estimation accuracies are questionable. These estimates are "one sided" and neglect under runs for cost and time and overruns for the amount of functionality delivered for either the under or over runs on cost and schedule. Institutional biases exist and are unaccounted for in the Standish numbers. (see Software Engineering Economics, Barry Boehm, Prentice Hall, 1982)
- Cost Forecasting - using independent cost forecasting databases, it can be shown that the Standish number forecast a 59% success rate for cost adherence, when the baseline database from actual projects - 140 projects from 2005 to 2006 at a large financial services firm - follow Boehm's "conical shape" for accuracy and produce an estimating quality factor of 8.5. This means that half the projects have a time-weighted average deviation of 12% or less from the actual forecast.
- Accuracy Again - using their definitions of cost and schedule overestimation and large functionality underestimations perverts the estimation accuracy.
All of this information and the references point to a statistical assessment gap. The core problem is:
- There is two-sided measurement from a statistical point of view. They only ask about failure rates, not the success rates. This would produce a biased outcome.
- There is not a statistical sample selection process. Send out a survey, get back the responses, and total the number. How many surveys are needed to establish a credible sample size in order to establish the confidence intervals.
