On a recent forum discussion a poster mentioned the Standish Chaos report and the reduction of project failure rates. I've spoken to the Standish Report in the past. I mentioned in the forum the problems with the underlying statistics in the report. This got me digging in my library for some refresher reading (I now remember all the reading needed for that quantitative research methods class).
- How To Think About Statistics, John L. Phillips
- Chances Are: Adventures in Probability, Michael and Ellen Kaplan
- How to Lie with Statistics, Darell Huff
- Random Data: Analysis and Measurement Procedures, Julius Bendat and Allan Piersol, Wiley-Interscience
- Probability, Random Variables and Stochastic Processes, Athanasios Papoulis, McGraw Hill
- Probabilty and Random Processes: Introduction for Applied Scientist and Engineers, Wilbur Davenport, McGraw
Then I came across several recent articles
- "The Standish Report: Does it Really Describe a Software Crisis?" Robert Glass, Communications of the ACM, August 2006
- "How Large are Software Overruns? Critical Comments on the Standish Group's Chaos Report," SIMULA Research Laboratory, 2006-03-21
- "Software Cost Overruns: How Large are They and How Should They be Measured?" SIMULA Research Laboratory, 2005-08-30
There are some interesting quotes from these reports.
"Let me say that again. Objective research study findings do not, in general, support the Standish conclusions. "
Robert L. Glass, "The Standish Report: Does It Really Describe a Software Crisis?," Communications of the ACM 49:8 (August 2006).
The results of that report (1994 Standish) have been used in several recent governmental reports, project reviews, and research studies. Examples are the PITAC 1999 report [2] and the cost estimation study described in [3]. An important result from the 1994 CHAOS research is the reported 189% average cost overrun of so-called challenged projects, i.e., projects not on time, on cost, and with all specified functionality. In this paper we argue that the 189% average cost overrun number, as it is commonly interpreted, is not consistent with results of other cost accuracy surveys and probably far too high to reflect the average cost overrun in that period. The measures and the research method of the CHAOS survey are insufficiently described to evaluate the quality of the results, e.g., there are many possible interpretations of what is meant by ‘cost overrun’ in the CHAOS reports. We should therefore cease to trust the 189% average cost overrun as a reference point for performance of software projects until such time as the Standish Group disclose how they measure cost overrun and how they conduct their research.
This paper tries to summarize estimation knowledge through a review of surveys on software effort estimation. Main findings were that: (1) Most projects (60-80%) encounter effort and/or schedule overruns. The overruns, however, seem to be lower than the overruns reported by some consultancy companies. For example, Standish Group’s ‘Chaos Report’ describes an average cost overrun of 89%, which is much higher than the average overruns found in other surveys, i.e., 30-40%. (2) The estimation methods in most frequent use are expert judgment-based. A possible reason for the frequent use of expert judgment is that there is no evidence that formal estimation models lead to more accurate estimates. (3) There is a lack of surveys including extensive analyses of the reasons for effort and schedule overruns.
These and the Robert Glass article bring out several issues:
- Standish does not reveal its underlying data
- The primary question asks of the "random" sample was "tell me about your failed projects"
- The bin-skewing approach of the boundaries creates on unreliable measure of the actual behavior of projects
- There is no description of the underlying sample population statistics - how many project should be in each bin if they were to represent the entire population of projects?
- There is no inference measure - how many samples needed to be taken to properly represent the underlying population of projects?
So in the end what are we to do? Ignore the report? Probably that's the best thing to do. But that doesn't solve the issues with troubled IT projects. They're still there. But anyone referring to the Standish report as a source of information about the state of the IT industry needs to do a little home work on see that Huff's "How Lie with Statistics" actually works.
