Pat Weaver has a nice post about risk and uncertainty. Like all of Pat's posts, he brings out an important discussion point. This one is about how to think about risk in the context of managing projects.
First let me speak to the article Pat references. Michael Hatfield's post on PMI's Forum. Michael does his typical slinging words around without the benefit of context or domain. Typical is
Finally, Monte Carlo analysis is essentially a decision tree on steroids, with lots of statistical chicanery thrown in.
If Michael understood how the mandated processes of DID 81650, he would know how Monte Carlo Simulation actually works and how it serves the Program Mangers of DoD and NASA programs and not be making unfounded statements like this. And most importantly how programmatic and technical risk management are used repeatedly on the program - weekly assessments once the CAM interviews are complete on our programs.
As well
My second objection has to do with the use of risk management after the cost and schedule baselines have been set. I agree that prior to the finalization of the baselines, risk analysis is crucial to identifying and quantifying cost and schedule contingency amounts. The risk analysis can lead to informed decisions on how much and what type of insurance to buy, and what sort of alternative plans should be in place if a contingency event occurs.
But once the baselines are final, persisting in risk management strikes me as institutional worrying expressed in mind-numbing statistical jargon. To what end? Unless the response to a contingency event (in-scope, uncosted) was to significantly change from how the project team would have reacted normally, what difference does it make if it was anticipated?
Michael fails yet again to understand how programs work in practice and how risk management and especially Monte Carlo simulation are used in practice. With the baseline set, and the program executing - past performance is a forecast of future performance. Monte Carlo tools and just plain "olde" risk management for cost, schedule, and technical performance make use of this past performance.
- Performance to plan is used to forecast future performance to plan.
- This is the very basis of Earned Value Management and especially the probabilistic Earned Value Management used in Defense and Space.
Does Michael think that CPI and SPI are steady during the program or that TCPI is a fixed number?
Back To The Point
Pat's description of positive and negative uncertainty confuses statistics with probability. Here's the starting picture
In a project context:
- Statistics - I know the completion durations of the last 20 times I did this task. Or I have a model I can use to inform me about the completion durations, cost, and technical performance.
- Probability - with these statistics and an activity network (schedule), cost estimating relationship (CER) model, or CAD dynamics model, what's the probability I'm going to be disappointed with the outcome of my work efforts?
For some good background see the NASA Cost Estimating site or the GAO Cost Estimating Handbook for some in depth processes and general guidance.
For program risk management we need to know the underlying statistics of the processes. These can be cost, schedule, and / or technical performance of the products or services. This "variability" is modeled in a variety of ways. Picking an underlying statistical distribution for the process is one. Using past performance data to construction the statistical distribution is another.
If a model is used the upper and lower limits of the range of values defines how this distribution impacts the behavior of the cost, schedule, or technical performance.
With the statistical distribution in place we can then ask and answer some questions:
- What is the probability of completing on or before a desired date?
- What is the probability of the project cost a desired amount or less?
- What is the probability that the spacecraft mass will be a certain value or less at some point in the future. (27 metric tons is the upper limit to reach low earth orbit by the way).
These probabilistic questions are "modeled" using the underlying statistical behaviors of the process the drive the answer.
Now for the Risk Question
Knowing something, through the modeling about the probability of achieving a goal - cost, schedule or technical performance - we can assess the risk associated with that probability.
Knowing the probability of achieving some goal informs the risk discussion like.
- What level of risk do we carry in not meeting the launch date?
- What level of risk to we carry for not making the performance goal of the second stage separation event?
- What level of risk to we carry for not being able to process each insurance claim at $0.07 / transaction?
Then we can calculate the financial or operational outcome knowing something about the risk
Risk and Opportunity are Not Interchangeable
There is a notion out there that opportunity is just positive risk. This is not the case from a decision making point of view. And it is not the case from a mathematical point view.
Opportunities and Risk do not have the same units of measures. They both exist for sure. But they are not "tradeable" in the sense of Reciprocity between the units. See Ed Cornrow's article in AT&L about this common mistake. Edmund H. Conrow and Robert N. Charette, “Opportunity Management: Be Careful What You Ask For,” Defense AT&L, Defense Acquisition University, March-April 2008. Buy Ed's book, read about how to do risk management. Look for the NASA, DOE (Department of Energy) and DoD (Department of Defense) Risk Management handbooks to see the discussions of Michael follow Wolfgang Pauli's guidance
This is not right, it's not even wrong
or my new favorite
That theory is worthless it's not even wrong
Early in a program - usually pre-PDR, the trade space can address the risk versus opportunity paradigm. This is the domain of systems engineering. Once we're close to CDR, the trades have been made and the remaining outcomes have risk associated with them.
In the End
Don't look to PMI for risk management advice. Look to places where unaddressed risk kills people. NASA, DoD, DOE, oil and gas operations, airlines, surgery. The very notion that probability of occurrence multiplied by the consequential outcomes is a measure of risk shows that they were asleep in the calculus class. Both those entities are integral equations (probability density generating functions) that do not have the operator "multiplication" applied to them. I see this all the time. It is DEAD WRONG. Look at the Conrow book as the starting point. Then build a 5x5 calibrated (ordinal) risk matrix where the two dimensions - probability of occurrence and consequential outcomes - are color coded by subject matter experts and the nonsense of multiplying to integral equations together.
