There is a current rash of suggestions on how to improve the performance of software projects. Each of these, while well meaning, are missing the means to confirm their credibility.
They are many times end up being personal anecdotes from observations and local practices that may or may not be applicable outside those anecdotes and more importantly may not be statistically sound in principle, let alone practice.
I work in the Software Intensive System of Systems domains in Aerospace, Defense, Enterprise IT (both commercial and government) applying Agile, Earned Value Management, Productive Statistical Estimating (both parametric and Monte Carlo), Risk Management, and Root Cause Analysis with a variety of capabilities. In this domain, we are guided by credible results using principles, processes, and procedures to increase the probability of program success. References below.
The growth of cost and schedule is not unique to commercial development. One of my work colleagues is the former NASA Cost Director. This is from one of our presentations from a International Cost Estimating and Analysis Association meeting on the same topic. And there are many other examples (see references).
One case of 12 projects from a large contractor of Software Intensive System (SIS) shows similar variances
More research at www.ida.org and www.acq.osd.mil/parca/ has shown there are four core root causes of this unfavorable growth in cost and schedule and shortfalls in technical capabilities.
Some might say these are domains unrelated to our domain. I'd suggest the Principles for project success on non-trivial software efforts are universal. Your project may be different in practice, but the principles are the same. These principles are:
So let's look at an example
Here's a typical graph showing a core problem in the software development domain.
There are a collection of projects that started with an estimate at completion and as these projects are executed they didn't turn out as planned - and most importantly they didn't turn out as needed.
Figure 1 - Planned Estimates versus Actual Performance from [1]
Some unanswered critical questions at charts like Figure 1 are:
- What's the credibility of the initial estimates?
- Are they risk adjusted?
- Is there Management Reserve for in scope but unplanned activities?
- What's the confidence in this estimate?
- What are the Cost Element Relationships that drive risk
- What processes are in place to execute according to plan?
- Are there measures of physical percent complete to provide the feedback needed to take corrective actions?
- Is there a risk retirement plan to ensure Risks don't turn into Issues and delay the project?
- What irreducible uncertainties and the resulting risks were Not handled?
- Is there schedule, cost, and technical margin in the plan?
- Do you know the margin burndown rate and is that measured to assure you're on plan for the burn down of margin rate?
- Was the scope controlled, properly funded, staffed properly, defects too high - causing rework, and a myriad of other operational and developmental things that could have caused the actual result to not match the initial estimate. What are these and what corrective actions from the root causes were not in place?
- Is there a change control process in place of some kind?
- Does that process prevent impacting the plans without authorization?
- What irreducible uncertainties were not considered for cost, schedule, and technical margins?
- Is there Management Reserve for this?
- Is that MR managed in accordance with the governance processes for the project or the firm?
Each of these questions and the others needed to determine if the samples like those Figure 1 have any root causes not identified by the author of the chart.
Without determining the cause of why the sampled value is what it is, the chart is missing one half of the needed information to make a decision for the corrective actions, and the unfavorable answers to the questions above.
A Few References and Resources
- Schedule Estimation and Uncertainty Surrounding the Cone of Uncertainty, Todd Little, IEEE Software, May/June 2006
- "Sources of Weapons Systems Growth: Analysis of 35 Major Defense Acquisition Programs"
- GAO cost estimating and assessment guide
- Parametric Cost Estimating Handbook
- NASA Cost Estimating Handbook
- USAF Cost Risk Uncertainty Handbook
- Department of Energy
- OMB A-11 Part 7
- Defense Acquisition Universty Cost Estimating
