We hear all the time about how project managers can't make credible schedules or generate credible cost estimates for IT project or even other types of projects - mega project construction for example.
I'd have to say Flyvbjerg is a bit over the top calling estimaters fools and lairs, but he does have a point about the in-credibility of cost and schedule estimates.
We about how building such estimates is a waste of time and we should switch to doing everything in an agile manner and let the requirements emerge over time while spending the clients money to discover them.
This is an exaggeration I know and it literally doesn't work that way even for agile processes. But it brings out the point that developing credible estimates of cost and schedule is actually hard and requires careful consideration to the processes.
The first thing to recognize is there is no such thing as a credible estimate in the absence of the underlying statistical and resulting probabilistic processes being applied to that estimate. The notion that we can forecast the duration of a work activity or the cost of that activity without a probabilistic confidence to that forecast is näive at best and delusional at worse.
Maybe if Flyvbjerg had used the work "delusional" it would set better with me. Liar implies malicious intent. Fool implies no one else was looking after the work and a "fool" was left in charge of the estimiating process.
What To Do About This?
Here's an example of the probability distribution of a cost estimate for a project.
This is from a model project that has a cost loaded IMS. The model will be only as good as the IMS is good and the resource loading is good. So this approach ONLY works when the model is right. That's a problem of course, but for now let's assume the model is right, since it came from a VERY credible source. David Hulett and I work the Risk and Opportiunity policy segment of the US DOD, along with others from government and industry.
The 50th percentile cost is $2,296,898 (a small project in the A&D world BTW). That means there is a 50/50 chance the project will cost more than that or less than that. That number is the median - the middle of the range. If we want to know what the number is with an 80% confidence, than it is $2,333,153. That says there is an 80% confidence that the cost of the project will be $2.3M or less.
When we speak about schedule, we can use the same terminology. In this case there is an 80% confidence the project will complete on or before 11/06/2015.
These graphs are generated from a Monte Carlo simulation tool applied to a resource loaded Integrated Master Schedule (IMS). If there is any doubt as to why you MUST create a resource loaded schedule, please please put those doubts away. These two graphs are the source for the Joint Confidence Level showing the over all probability distributions for both cost and schedule for the project.
This picture should convince anyone that the "joint" probability of completing on or before the target date (I didn't say what that was) and at or below that planned budget (didn't say what that was either) needs to be modeled in a way that would be considered credible.
Outside of a small domain of aerospace and defense this is rarely if ever done. Most IT projects have some made up budget and schedule. Their strategy is based almost entirely on HOPE with no underlying assessment of the statistical and probabilistic drivers of the cost and schedule, let alone the technical aspects coupled with risk.
And we wonder why IT projects come in late and over budget.
Turns out of course large defense and space programs have the same problem - poor estimating, politically motivated estimating, naive estimating, etc. But they are required to produce data like this, so we know it is bogus sometimes early enough to fix it - sometimes not.
But the real root cause of most serious overages for A&D comes from other sources. Those can be found in the Nunn McCready breach reports at RAND in Volume 1 and Volume 2. IDA has there assessment as well. There are some professional provocateur voices that conjecture all this can be fixed with a good WBS, a scaled down Earned Value Management System, and "firing the program managers when they go over cost and schedule." That of course is probably not the real solution. Read the RAND reports for the root causes of major overages. Read the Feb 18th issue of Aviation Week, "Slaughterhouse Five: If sequestration can rewrite U.S. strategy It's time to kill the sacred cows in industrial policy." F-35's current cracked first compress blade grounding and the world wide subcontracting effort that provides the program with nearly invincible support in congress.
We only wish it were as simple as many would have us believe. But sadly it is one of those wicked problems that we simply have to manage in the presence of. Many have suggested solution - credible ones, not the cocka mamy ones - but those solutions usually don't address the political forces in play when you are working with the public's money.
Why I've Followen in Love with R
In the programming language R, drawing the joint probability plot is done with a simple command
> pairs(X)
produces a scatter plot of the variables defined by the columns in X. Every column in X is plotted against everyother column of X and the resulting n(n-1) plots arranged in a matrix with constant plot scales over the rows and columns of the matrix.
Trending is a critical tool in forecasting the future performance of a project. "We've be doing this well for awhile now, what's going to happen in the future?" is a very common question for project management to ask of the "project controls" staff.
Two indices in Earned Value are the Cost Performance Index (CPI) and Schedule Performance Index (SPI). They are calculated from the Earned Value Management numbers of BCWP (Budgeted Cost of Work Performed), BCWS (Budgeted Cost of Work Scheduled) and ACWP (Actual Cost of Work Performed). This post is not about Earned Value, but about statistical analysis of Earned Value numbers. You can find good tutorials on EVM in several places, including A Gentle Introduction to Earned Value.
What is critical is the forecast for where the project is going with the information from the past. In the standard approach static information is used to compute a forecast of the future. This information is a cumulative measure of CPI/SPI for all past periods and the current period of performance. Together these are used in a linear formula to forecast future performance - again in a linear, non-probabilistic manner.
All of this has lead me to think about the probabilistic processes needed to improve the credibility of project performance measures. I discovered through our BioChem son, the programming system R. R is a functional programming language and system for statistics and probability.
Along with R the US DOD has rolled out an XML format for capturing the monthly Integrated Performance Management Report. The time series of CPI and SPI are available monthly in electronic readable form. With this data forecasting using R and the myriad of tools can be done.
Using a simple functional command line,
will produce a nice picture of the time series data with forecast value at defined confidence levels.
The next step is to do this on actual project performance data (using SPI/CPI) and see what can be reveiled about future performance.
The problem to be solved is simple:
Mathematically this means that what drove the variances in the past is assumed to no longer be driving the future. Those sources of variance are washed out. This may be the case from corrective actions for the Epistemic uncertainties (uncertainty due to lack of knowledge). But the Aleatory uncertainties (uncertainty due to randomness or luck) that drive the variances are not subject to corrective actions. They are baked into the system and no amount of new knowledge or corrective effort is going to make them go away.
This is simply bad data analysis in any discipline I've ever worked, except program controls.
Background Materials
This idea started with Eric Ducker's paper and presentation at ICEA titled, "Performing Statistcal Analysis on Earned Value Data."There are some other sources as well:
There is a class of responders on the forums that continue to seek simple and simple minded solutions to complex problems. Claiming the problems can be solved with simple solutions. Of course those solutions cannot be demonstrated with any credibility outside of a limited domain. Let's start with Mencken
For every complex problem there is an answer that is clear, simple, and wrong. - H. L. Mencken
In the project domain most of the problems start with poor estimates of cost and schedule. Hollmann starts it off http://lnkd.in/vjf4X4. Bent Flyvbjerg takes a more radical view, calling estimators liars , but I chalk that up to a language barrier.
The core problem starts with the politics of estimating. Rarely do the buyers of the project know what it should cost. So when they hear a number that is larger than their expectations, they reject the number. They signal to the sellers what they expect to pay for something. This expectation can come from poor knowledge or simply a naive understanding of the problem, or worse a public promise that simply can't be kept.
But a second problem exists. That across domains, there is the wrong headed belief that one domain has the solution to another domains problems. We need to break the loop we're stuck thinking that problems are only found in other domains and making the ill-informed suggestions that "if you just did it my way," there would be no problem. There is a fundamental issues with managing complexity and managing in the presence of uncertainty. We want it to be simple, it is not, but since it is hard, people don't want to actually work hard on the solution. Instead seeking simple and may times "simple minded" solutions - buy this tool, use my excel spread sheet, dumb down the processes.
So What Is The Solution
First is to recognize that ALL domains have similar problems. Cost and Schedule estimates have been discussed for many decades. Lots of literature leads us back to the singular source - the political will to search for the credible estimate, even if that means not starting the project. Or better yet, to start the project knowing the estimates are flawed.
One provocateur voice in our community suggests that PMs that take on project with poor estimates should be held criminally liable for the results. This of course is laughable, but it demonstrates the complete understanding of the problem and the solution.
This approach reminds me of Pauli's quote when presented with a nonsensical submission from a student.
They were worse than wrong because they could not be proven wrong. Famously, he once said o: Das ist nicht nur nicht richtig, es ist nicht einmal falsch! "Not only is it not right, it's not even wrong!"
Thus is where we are now. Not much in the form of research and questioning, lots of pontificating about how the other guy is all messed up, and if you'd only do it my way, you'd have your solution. Reminds me of the current US political problems. Do it my way. Of course that doesn't work, because we have complex - wicked - problems, and there is no single way. Mencken was right, and he was right long before any of the current problems.
We may be doomed to repeat history with first recognizing the problem is us.
There a number of people in the Project Management world that seem to take delight in standing on the side of the road pointing out all the failures of projects, processes, and the people managing them. I'm working a program for a federal agency to produce a set of Essential Views of program performance. These views will be leading indicators of performance measures, which will provide management with insight to take corrective actions.
During the research side of this work I came across this ...
My first thought was how can we improve our forecasting processes. Then I thought of those voices essentially bitching that projects aren't meeting the expectations of the public, while at the same time never providing any actionable suggestions beyond their personal gripes.
I concluded to start ignoring those type of statements and work instead on providing small and incremental steps toward Increasing the Probability of Program Success. The book Little Bets is one of our guides.
This brought me to the quote of Theodore Roosevelt, from his 1910 "Man in the Arena Speach," at the Sorborne, Paris, France.
It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.
Recent Comments