Myths, misinformation, and magic bullets abound in the software development business. No more so than in the management and estimating of complex development projects.
The Standish report is a classic example of applying How to Lie with Statistics. This book is a must read for anyone responsible for spending other peoples money.
The current misrepresentation approach is to quote people like Bent Flyvbjerg - who by the way does superior work in the domain of mass transportation and public work. Bent, along with many others, one of which is a client, have studied the problems with Mega Projects. The classic misuse of these studies starts with the reading of the introduction of a report and going no further. Here's a typical summary.
9/10 Costs (are) underestimated. 9/10 Benefits (are) overestimated 9/10 Schedules (are) underestimated.
OK, we all know that's what the report says, now what?
- Do you have a root cause for each of the project's overages?
- Did those root causes have sufficient assessment to show:
-
Primary Effect – is any effect we want to prevent.
-
Action – momentary causes that bring condition together to cause an effect.
-
Conditions – the fundamental causal element of all that happens. It is made up of an effect and its immediate causes that represent a single causal relationship.
- As a minimum, the causes in this set consist of an action and one or more conditions.
- Causal sets, like causes, cannot exist alone.
- They are part of a continuum of causes with no beginning or end, which leads to the next principle that Causes and Effects are Part of an Infinite Continuum of Causes.
NO?, then the use of reports and broad unqualified clips from reports is just Lying With Statistics.
The classic example from the same source states Steve McConnell PROVES estimates can't be done in his book. Which of course the antithesis of the title of the book and the content of the book.
This approach is pervasive in places where doing your homework appears to be a step that was skipped.
From our own research in DOD ACAT1 programs (>$5B qualifies for Megaprojects) here's the Root Cause of program problems in our domain.
When we hear some extracted statement from a source in another domain - Bent for example is large construction infrastructure projects - roads, rail, ports - moved to our domain without the underlying details of both the data, the root causes and all the possible corrective actions to avoid the problem in the first place - that idea is basically bogus. Don't listen. Do your own investigation, learn how to not succumb to those who Lie With Statistics.
So let's look at some simple questions when we hear there are problems with our projects or projects in other domains trying to convince us it's applicable to our domain.
- Was there a credible set of requirements that were understood by those initiating the project?
- No? You're going to be over budget, late, and the products not likely to meet the needs of those paying.
- Was there a credible basis of estimate for the defined work?
- No? Then what ever estimate was produced is not going to be what the project actually costs, its duration, or the planned value.
- Was the project defined in a Risk Adjusted manner for both reducible and irreducible uncertainties?
- No? Then those uncertainties are still there and will cause the project to be over budget and late.
- If the reducible risks are unaddressed, will come true with their probabilistic outcomes will drive the project over budget, late, and not provide the needed capabilities
- If the irreducible risks don't have margin, you're late and over budget before you even start.
- Was there a means of measuring physical percent complete in meeting the needed Effectiveness and Performance measures?
- No? Then money will be spent and time will pass and there is no way to know if the project is actually producing value and meeting its goals.
These questions and their lack of answers are at the heart of most project performance problems. So pointing out all the problems is very easy. Providing corrective actions once the root cause is discovered is harder, mandatorily harder by the way. Because Risk Management is How Adults Manage Projects.
First let's look at what Bent says
He states the political economy of megaprojects, that is massive investments of a billion dollars or more in infrastructure or technology, consistently ends up costing more with smaller benefits than projected and almost always end up with costs that exceed the benefits.
So the first question is are we working in the mega project domain? No? Then can we assert Bent's assessments are applicable. If we haven't then we're Lying with Statistics. (Read Huff's book to find out why).
Flyvbjerg then explores the reasons for the poor predictions and poor performance of giant investment projects and what might be done to improve their effectiveness. Have we explored the reasons why our projects overrun? No? Then we haven't done our homework and are speculating on false data. Another How to Lie With Statistics.
Stating that projects over run 9 out of 10 times without also finding the reasons for this is the perfect How to Lie with Statistics. Make a statement, no supporting data, be the provocateur.
The End
When we read a statement without a domain or context, without a corrective action, that is intended to convey a different message, taken out of context, without the evidence it is applicable in our domain, than the person writing the original statement, is Lying with Statistics - don't listen, go find out for yourself.
Related articles
