There remains serious misunderstandings of how, why, when, and for what purpose estimates of cost, schedule and delivered capabilities are made in the development of software systems using other peoples money.
There are three distinct approaches to the problem:
- Schedule Estimation and Uncertainty Surrounding the Cone of Uncertainty, Todd Little, IEEE Software, May/June 2006.
- The Rise and Fall of the Chaos Report Figures, J. Laurenz and Chris Verhoef, IEEE Software, January/February 2010.
- Quantifying IT Forecast Quality, J. L. Eveleens and C, Verhoef, Science of Computer Programming, Volume 74, Issues 11–12, November 2009, Pages 934–988.
The first paper shows the self-selected projects and how they have completed - for the most part - lonfer the ideal initial estimates. These estimates are not calibrated, meaning they are not assessed for credibility, error bands, or confidence. The paper mentions the the solid line, initial versus actual is the ideal line where actuals meet estimated value. In any stochastic estimating process, it will be unlikely ant estimate will result in a match with the actual for the very simple reason that the work processes are random and when the estimates don't contain the probabilistic confidence intervals, the actual MUST be different that the estimate.
As well, no root cause for the unfavorable performance of the actuals compared to the initial estimates is provided. This is a core failure to understand the process of estimating, rot cause analysis, and the discovery of the corrective actions needed to improve both the estimating processes as well as project performance management.
This fundamental failure is not limited to the self-selected set of projects in the paper. This failure mode can be found a wide variety of project domains in and out of the software business.
The second paper speaks the the major flaws Standish Report - meaningless figures. self-selected samples, perverted accuracy, unrealistic rates and misleading definitions. The paper states the root causes and suggested corrective actions.
The third paper shows how to quantify IT forecasts (estimates of future outcomes) in a mathematically sound manner.
- Software Cost Estimation and Sizing Methods is a more in depth report on the issues, root causes, and corrective actions is a good starting point for further understanding. There are numerous other reports, guides, assessment, and corrective actions.
- Analytical Method for Probabilistic Cost and Schdeule, NASA Office of Program Analysis and Evaluation Cost Analysis Division, 5 April 2013.
All five papers are useful in the right context. Little re-introduces Boehm's cone of uncertainty, assessment of Standish shows the traps that can be easily fallen into when good statistical practices are not followed, the third provides the mathematical foundation for restoring those sound practices, and the RAND report shows the mechanics of the corrective actions to restore credibility in software estimating.
A risk based view of the estimating problem developed for the recent successful launch and recovery of Orion, then called Crew Exploration Vehicle.
