We hear all the time about how project managers can't make credible schedules or generate credible cost estimates for IT project or even other types of projects - mega project construction for example.
I'd have to say Flyvbjerg is a bit over the top calling estimaters fools and lairs, but he does have a point about the in-credibility of cost and schedule estimates.
We about how building such estimates is a waste of time and we should switch to doing everything in an agile manner and let the requirements emerge over time while spending the clients money to discover them.
This is an exaggeration I know and it literally doesn't work that way even for agile processes. But it brings out the point that developing credible estimates of cost and schedule is actually hard and requires careful consideration to the processes.
The first thing to recognize is there is no such thing as a credible estimate in the absence of the underlying statistical and resulting probabilistic processes being applied to that estimate. The notion that we can forecast the duration of a work activity or the cost of that activity without a probabilistic confidence to that forecast is näive at best and delusional at worse.
Maybe if Flyvbjerg had used the word "delusional" it would set better with me. Liar implies malicious intent. Fool implies no one else was looking after the work and a "fool" was left in charge of the estimiating process.
What To Do About This?
Here's an example of the probability distribution of a cost estimate for a project.
This is from a model project that has a cost loaded IMS. The model will be only as good as the IMS is good and the resource loading is good. So this approach ONLY works when the model is right. That's a problem of course, but for now let's assume the model is right, since it came from a VERY credible source. David Hulett and I work the Risk and Opportiunity policy segment of the US DOD, along with others from government and industry.
The 50th percentile cost is $2,296,898 (a small project in the A&D world BTW). That means there is a 50/50 chance the project will cost more than that or less than that. That number is the median - the middle of the range. If we want to know what the number is with an 80% confidence, than it is $2,333,153. That says there is an 80% confidence that the cost of the project will be $2.3M or less.
When we speak about schedule, we can use the same terminology. In this case there is an 80% confidence the project will complete on or before 11/06/2015.
These graphs are generated from a Monte Carlo simulation tool applied to a resource loaded Integrated Master Schedule (IMS). If there is any doubt as to why you MUST create a resource loaded schedule, please please put those doubts away. These two graphs are the source for the Joint Confidence Level showing the over all probability distributions for both cost and schedule for the project.
This picture should convince anyone that the "joint" probability of completing on or before the target date (I didn't say what that was) and at or below that planned budget (didn't say what that was either) needs to be modeled in a way that would be considered credible.
Outside of a small domain of aerospace and defense this is rarely if ever done. Most IT projects have some made up budget and schedule. Their strategy is based almost entirely on HOPE with no underlying assessment of the statistical and probabilistic drivers of the cost and schedule, let alone the technical aspects coupled with risk.
And we wonder why IT projects come in late and over budget.
Turns out of course large defense and space programs have the same problem - poor estimating, politically motivated estimating, naive estimating, etc. But they are required to produce data like this, so we know it is bogus sometimes early enough to fix it - sometimes not.
But the real root cause of most serious overages for A&D comes from other sources. Those can be found in the Nunn McCready breach reports at RAND in Volume 1 and Volume 2. IDA has there assessment as well. There are some professional provocateur voices that conjecture all this can be fixed with a good WBS, a scaled down Earned Value Management System, and "firing the program managers when they go over cost and schedule." That of course is probably not the real solution. Read the RAND reports for the root causes of major overages. Read the Feb 18th issue of Aviation Week, "Slaughterhouse Five: If sequestration can rewrite U.S. strategy It's time to kill the sacred cows in industrial policy." F-35's current cracked first compress blade grounding and the world wide subcontracting effort that provides the program with nearly invincible support in congress.
We only wish it were as simple as many would have us believe. But sadly it is one of those wicked problems that we simply have to manage in the presence of. Many have suggested solution - credible ones, not the cocka mamy ones - but those solutions usually don't address the political forces in play when you are working with the public's money.
Why I've Followen in Love with R
In the programming language R, drawing the joint probability plot is done with a simple command
> pairs(X)
produces a scatter plot of the variables defined by the columns in X. Every column in X is plotted against everyother column of X and the resulting n(n-1) plots arranged in a matrix with constant plot scales over the rows and columns of the matrix.
