The 2009 Standish Report is out. Many Blogs have made comments. All obvious comments. But there is a fundamental flaw in any report of this type:
The concept of a assessment baseline is missing from the conversation.
- Is there a Performance Measurement Baseline?
- Does this baseline represent a credible plan for cost, schedule, and technical performance?
- How would we know how many projects "should" fail in the population of all projects?
- Is the project sample used by Standish representative of the total population of projects?
- If so, what's the correlation between the sample population and the total population in terms of failure rates?
These are basic statistical analysis questions. Of course the report does not state these for the right business reasons, but the result is now limited in terms of its interpretive power.
The first big question is - Is there a control group?
- How many projects should fail as a matter of course of being members of a underlying population?
- What is the proper statistical population for projects?
- What is the underlying distribution of project success versus project failure for this population?
In the absence of this information - being made public - the Standish numbers are just a marketing document for their training and consulting services. In the same way the weight loose, hair growth, lawn growth, and other sales approaches are frankly - Nonsense.
The easy starting for discussion of this sort is How To Lie With Statistics. This little book wias written in the 1950's and is still an important contribution to the discussion. After reading this short book, you too will be suspicious of Standish style reports.
Just remember how you feel, when you see and ad for a weight lose products reporting how much weight will be gone in the coming 6 weeks with their product.
In the end though Standish provides no solution, just an 800-number to call for the "magic elixir"
More Standish
From the 1994Standish report - not so much the percentages - but the classifications:
- Resolution Type 1, or project success: The project is completed on-time and on-budget, with all features and functions as initially specified.
- Resolution Type 2, or project challenged: The project is completed and operational but over-budget, over the time estimate, and offers fewer features and functions than originally specified.
- Resolution Type 3, or project impaired: The project is canceled at some point during the development cycle.
Analysis of Resolution Types from a Statistical assessment point of view.
- Type 1 - it is unlikely that any non-trivial project comes in on-time, on-budget, with all the features and functions. In the DoD Program Controls business this would be called "Blue." Meaning its too good to be Green, so someone is cooking the books. All projects have issues. Period. That's just the nature of projects. Especially software projects.
- Type 2 - the term "challenged" is loaded. All non-trivial projects are continuous challenged. Ranging from simple lawn mowing to flying to the moon.
- Type 3 - canceled for what reason. Was it a good reason? A bad reason?
