Herding Cats

The Very Notion of Probabilistic Modeling is Lost on most Projects

 The obvious picture at the left is how nature works. Nature driven by stochastic processes. The same type of stochastic processes that drive project work.

Cost, Schedule, and Technical Performance Measures are based on random variables. These are all connected in networks of dependent relationships.

Any thought of linear relationships is naive at best and foolish at worse.

Managing in the presence of the uncertainty created by these relationships is mandatory. Trying to avoid this uncertainty, control the uncertainty, and ignore the uncertainty will result in disappointment. But this uncertainty is not chaos. It can be modeled and those models can be used to forecast future performance, the impacts of corrective actions on this forecast. This is not a Chaos situation. If it is Chaos than it's not a project, it's just Chaos.

There are four types of Uncertainty in Projects

1. Variation - comes from many small influences and yields a range of values on a particular activity
2. Foreseen Uncertainty - are identifiable and understood influences that the team cannot be sure will occur.
3. Unforeseen Uncertainty - can’t be identified during project planning
4. Chaos - projects subject to unforeseen uncertainty start out with reasonably stable assumptions and goals, projects subject to chaos do not. Even the basic structure of the project plan is uncertain, as is the case when technology is in upheaval or when research, not development, is the main goal.

These ideas are from the work of Arnoud De Meyer, Christoph H. Loch and Michael T. Pich in the MIT Sloan Management Review, WINTER 2002. Their work can also be found at www.insead.edu.

It Can't Be Any Clearer Than This

If you're going to have any hope what so ever of being successful with your project, here is some advice. Only with this integration can you know the answers to the 5 Immutable principles of project management.

1. What does DONE look like? - define the needed capabilities in Measures of Effectiveness and Measures of Performance in units meaningful to the decision makers.
2. How are you going to get to DONE? - construct a plan showing how you are going to assess the increasing maturity of the deliverables and a schedule showing the order of the work needed to produce these deliverables, and the measures of Technical Performance needed to assure these deliverables meet the needs of the customer.
3. What Resources are you going to need along the way? - resource load the Integrated Master Schedule - or what ever artifact you are using to show the path to completion.Show the profile (demand and capacity) for each resource.
4. What Impediments will you encounter along the way? - without a risk management plan and a risk management process to support the plan, showing how you are going to handle each risk, you're going to be late, over budget, and your gadget probably isn't going to work.
5. How are you measuring progress to plan? - this measure needs to be in units of physical percent complete. This measure needs to be adjusted for compliance with the Techncial Performance Measures, and this measure must be probabilistically risk adjusted for the variances in performance.

This text in the picture is taken from the new Integrated Program Management Report (IPMR), DI-MGMT-81861.

Four components of risk

Project Managers constantly seek ways to eliminate or control risk, variance, and uncertainty. This is a hopeless pursuit.Managing in the presence of risk, variance and uncertainty is the key to success. Some projects have few uncertainties – only the complexity of tasks and relationships is important – but most projects are characterized by several types of uncertainty.

Although each uncertainty type is distinct, a single project may encounter some combination of four types:

1. Variation – comes from many small influences and yields a range of values on a particular activity. Attempting to control these variances outside their natural boundaries is a waste (Muda).
2. Foreseen Uncertainty – are uncertainties identifiable and understood influences that the team cannot be sure will occur. There needs to be a mitigation plan for these foreseen uncertainties.
3. Unforeseen Uncertainty – is uncertainty that can’t be identified during project planning. When these occur, a new plan is needed.
4. Chaos – appears in the presence of “unknown unknowns.”

“Managing Project Uncertainty: From Variation to Chaos,” Arnoud De Meyer, Christoph H. Loch and Michael T. Pich, MIT Sloan Management Review, Winter 2000.

Books PMs Should Read and Put To Use

There are several shelves of books about many topics in our house. Project Management and related topics are in the Office. The list of books below have specific attributes. They have not only been read, they have been marked up, reread, and used in practice. I have many books that are read, but not marked up. Other books that have been read, marked up, but not put into practice. And a hand full of books that have all three attributes plus one more. The contents of the book has changes how I do my job.

There are other books that I've read, marked up, and concluded the content was either not applicable to the problems I work on, or in a few cases complete nonsense. By this I mean the content is not only not appropriate to the problems I work, but if applied will result in disappointment.

So here's my must have list. This list is not populist in nature. By populist I mean a book or paper in which you CANNOT calculate some outcome. A populist book is something to read if you are just coming to the topic and want to discover the lay of the land.The books listed here are nit populist books

This list is in no particular order, other than they were taken off the shelf in this order. One other attribute is if I know or met the author.

General Project Management

These books are about the general concepts of managing projects, including agile projects. They are from authors who are or have recently benn project and program managers. This is important for the simple reason that who wants to read about something from someone that actually doesn't practice what is in the book. By the way, project management techniques that do not address agile approaches are not that useful anymore. As well, authors with 1996 style credentials have likely been passed by the current approaches as well.

• Making the Impossible Possible: Leading Extraordinary Performance, Kim Cameron and Marc Lavine - this is a case study about the management processes at Rocky Flats. Here's a quick summary of the concept of Positive Deviance and Heliotropic Abundance.
• Strategic Performance Management: Leveraging and Measuring your Intangible Value Drivers - Bernard Marr speaks to managing the business value from projects. There is lots of discussion around this, with lots of soft terms. This is a book that turns that populist view into a balance sheet view.
• The Handbook of Program Management, James T. Brown - I have met Dr. Brown. He is a former Program Manager at NASA. This is a book about program management. Program Management is about managing collections of projects. It is a hands on instructional advice type of book.
• The Radical Elements of Radical Success, Dan Ward - this is a book from a person practicing Program Management. Dan is an Air Force procurement officer, just back from theater. It's one of those books that causes me to think. Dan and I have had long discussions about the issues in government procurement. Here's a pre-defined search for Dan's work
• Project Management the Agile Way, John C, Goodpasture - I know John professionally, but have never been face to face. John and I share many common principles. This book is published by Management Concepts, which I do work for. MC publishes books, provides courses, and services for the Federal Government. John's book shows you how to deploy agile processes in the presence of the governance based environment. One where you are spending other peoples money and are accountable for success to external parties, like the Federal Government, a Board of Directors, and share holders.
• Flawless Execution, James D. Murphy - I cam across the After Burners at a PMI conference. The speakers are all ex-fighter pilots. There approach is logical, concise, and clear. The perfect paradigm for managing projects. This approach is not for everyone. It holds all members accountable for their actions, builds teams in ways not found in any other approach and is not one of those touchy feely styles.
• Catastrophe Disentanglement: Getting Software Projects Back on Track, E. M. Bennatan - there are many books about recovering projects. I'm reading another and writing a review. But this book is a check list book, with a step-by-step process of recovering projects.
• Beyond the Hype, Robert G. Eccles and Nitin Nohria - this is one of those must read for anyone interested in processes. Many if not most of the new ideas are actually old ideas in new cloths. Or worse, poor copies of old ideas promoted by self aggrandizing authors who know how to market. This book speaks to the problem of not knowing the past, so you
• Reinventing Project Management, Aaron J. Shenhar and Dav Dvir - this is a good starting point for PM in the IT environment. With field proven approaches, the books provides the step-by-step activities needed to manage projects in the Enterprise business domains.
• The 7 Habits of Highly Effective People, Stephen R. Covey - this is one of my populist books that has direct applicability to project management. Our method - Performance Based Management - contains most of the content of Covey's book is for Project Manager, including the Performance Based Management process we use on our programs.
• The Seven Secrets if How to Think Like A Rocket Scientist -Jim Longuski describes the processes needed to manage complex projects. It's a check list of items rarely thought about in the normal project management paradigm. This is done, because the domain Jim works in has projects that blow up and kill people of not properly managed.
• A Bee in a Cathedral - Joel Levy's book is here because using analogies in the project management world is very common. And most often the analogies are bogus. read this to see how to use analogies properly, since they are powerful tools to get points across, reveal understanding.

Leadership

There are so many books on leadership in the project management space. I'm simply burned out listening to all the voices on leadership from folks who speak and write about the topic, but who have not actually lead people under circumstances that require real leadership. Here's the books that you need from those people who have.

• How NASA Builds Teams - Pellerin led the Hubble Space Telescope program and speaks to how that worked and didn't work.
• Augustine's Laws - Norman was the leader of Martin Marietta here in Denver. These Laws are general business processes but for high tech firms.
• 8 Basic Habits if Exceptionally Powerful Lieutenants - this book and the next come from the Air University Press,  many in downloadable PDF.
• Commanding an Air Force Squadron in the Twenty-First Century - this is another survey books about how leaders learn to lead.

Systems Engineering

In the aerospace and defense business Program Management is a Systems Engineering paradigm. This is because the technical architecture of the program has to be matched with the programmatic architecture. The product that is being designed must also be built and shipped. This means the value stream flow of work needs an architecture as well. This is the purpose of the Integrated Master Plan (IMP) and the Systems Engineering Management Plan (SEMP)

These books are the basis for being a good project manager. Here's a list of specific topics, meaning more technical, needed for project success.

• System Engineering: Coping with Complexity, Richard Stevens - this is another handbook for designing, developing, and deploying systems.
• The Art of Systems Architecting, Second Editon, Mark W. Maier and Eberhardt Rechtin - everything is a system. No those silly descriptions of those populist systems where you can't design or calculate anything. But system we encounter in the real world. The air traffic control system, the banking system, the internet, the enterprise IT systems, manned space flight, building the 787. Systems like that. Rechtin is the father of the approaches used to build real systems. Yes, people are in these systems. And people are an important part, but in these types of systems, the components of the system are the basis of success.

Risk Management

There are lots of risk management books, articles, websites, and Bloggers out there. Here are the two you should own and read.

•  Risk Happens!: Managing Risk and Avoiding Failure in Business Projects, Mike Clayton - I've corresponded with Mike for several years. This is a great book for starting the process of risk management. It follows the steps defined in many locations - the best is the SEI Continuous Risk Management (CRM) model. There are simialr guides from DOD and DOE. PMI has a Risk Management Practice Guide, but like most things PMI they have their own way of doing things and rearrange the work processes.

As an aside determining risk drivers is a critical success factor for any Risk Management process. Typically risk management means making lists of risks, ranking them, and doing some kind of handling. This is not only naive, but it is wrong. You must determine which risks drive which undesirable outcomes of your project. Start with A Framework for Categorizing Key Drivers of Risk and Taxonomy-Based Risk Identification. I worked with one of the authors.

• Effective Risk Management: Some Keys to Success, Edmund Conrow - I've worked with Ed on a proposal and he turned me on to several concepts that I use daily. The first is the serious error in the PMBOK about multiplying the probability of occurrence with the ranked outcome to produce a risk score. This is not mathematically possible, since both are probability distirbution functions. The Method of Moments can help if you don't have a Monte Carlo Simulator. But if you don't buy Ed's book and read how to do Risk Management the right way.
• Effective Opportunity Management for Projects: Exploiting Positive Risk, David Hillson - I have met David, and spoken with him over the web. This is good book for addressing the opportunity concept. You should read Conrow's advice for mixing opportunity and risk before doing so yourself.   

Earned Value Management

There aren't really a lot of books about Earned Value Management. But there is a ton of misinformation about how EV is used, no used, applied, and misapplied. So these books and sources should be the starting point

• Performance Based Earned Value - Paul's book came out a few years ago, and the ideas have been with me ever since. The notion that Technical Performance Measures are part of Earned Value was present in the C/SCSC days, but was lost in 748-B and needs to come back. See Paul web site of the same name for good background.
• Earned Value Management Intent Guide - is the National Defense Industry Association's offical document for US Government Earned Value Management. This guide tells you how to comply with the 32 Guidelines of Earned Value Management. For most projects outside the government this is not applicable. But you should read this anyway, just to have an understanding of what the largest implementations look like.
• Earned Value Management Maturity Maturity Model - Ray's book provides a framework for assessing the processes needed to improve the maturity of your Earned Value Management implementation
• Earned Value Management site at the Office of Secretary of Defense, Performance Assessments and Root Cause Analysis.
• Earned Value Management at the NASA site has lots of background as well.

There are other sources of Earned Value, even in the agile world. But care is needed. To properly use Earned Value you must be able to do several things:

1. Maintain a baseline of the capabilties at least - the end outcomes of the project should not be changing
2. Know the total budget - the end budget and some notion of how this is spread over time.
3. Have some measure of physical percent complete and know how much physical percent complete you should be at when you measure it.

If you can't do these, you shoulded be considering Earned Value for your project

Statistics and Math

Project managers need to know somethings about probability and statistics. All the numbers in project management, cost, durations, head count, technical performance are random numbers. In the absence of knowing how to use statistics and probability, you're going to be late and over budget before you start.

• How to Lie With Statistics - every PM must have this book, must have read it, and must put it to use. When you have a conversation with anyone on the project, have this conversation in the context of this book.
• Then get Advanced Statistics Demystified or Statistics for Dummies and learn about sampling, variances, inferences, and other models of how your project is going to be working even if you don't know that is working that way.

 

Rings of Success

Olympic Rings featured in nearing everywhere these days. Those rings represent the 5  parts of the world which now are won over to Olympism and willing to accept healthy competition. While there is much talk about the projects of the Olympics, some difficulties and many successes, there are other rings that must be connected for project success:

1. Risk - all project elements have risk. Cost and Schedule risk are a starting point. These can be addressed through straight lists of risks, their probability of occurrence, impacts, costs to mitigation and residual impacts after mitigation. Most approaches to risk management assume an independent association between the risks. That is they are decoupled risks, existing on their own. Many times this is not the case. What is Program Controls? speaks to some of the complexity of integrating technical risk with programmatic risk and addressing the coupling issues that are ignored in the Monte Carlo Simulators of project duration and their resulting cost models.
2. Cost - the olde phrase time is money is tossed around without any real consideration for what that means. What is the relationship between time and money. Where is that relationship defined? What are the statistical attributes of that relationship and its impact on the probability of success for the project. This must be addressed through a cost modeling system. SCEA is one place to start.
3. Statement of Work - What Does DONE Look Like? The SOW is one place to look. Assuming it is written properly. But Statement of Objectives, Concept of Operations, Scenarios, Use Cases, and Performance Work Statement are also places to look. Without a clear and concise collection of deliverables
4. Technical Performance Measures - define what DONE looks like in units of measure meaningful to the decision makers. TPMs involve a technique of predicting the future value of a key technical performance parameter of the higher-level end product under development based on current assessments of products lower in the system structure.
5. Work Breakdown Structure - describes the products or services to be delivered by the project, how they are decomposed and related. At the end of the tree of the WBS, is usually the Packages of Work that produce these outcomes. Without a properly formed WBS, you cannot determine what done looks like in any meaningful way. A typical mistake is to develop a WBS based on work types - design, code, test. This simply states the effort not the result.
6. Integrated Master Plan / Integrated Master Schedule - The IMP is the Plan for the delivery of a set of Capabilities from the project and describes the increasing maturity of these capabilities through Significant Accomplishments and their Accomplishment Criteria. The Integrated Master Schedule describes the sequence of work, usually in Work Packages needed to produce the deliverables for the Accomplishment Criteria.
7. Performance Measurement Baseline - brings the cost, schedule, and outcomes together in one place, puts this under change control, and measures progress to plan through assessments of Physical Percent Complete, and forecasts future performance (Estimate At Completion and Estimate to Complete) using Earned Value Management.

Putting This All Together Through the Tangible Evidence

Each of the Rings are represented by tangible evidence in the form of a document. I can hear the agilest now, no documents. Well sorry, in the domain we work, we need evidence that you do in fact have a SOW, a Risk Plan, an IMS, a cost basis of estimate, a Technical Performance Management plan and other items that show the project is planned properly, being managed properly, and measured properly.

Here's the top level connections...

What Is Program Controls?

There is a discussion on the LinkedIn #ProjectControls forum. It started out with a simple question of what does PERTMaster do? PERTMaster is one of many tools that provides results from a schedule, showing the probability of completing on or before a date and a confidence of that forecast, using Monte Carlo Simulation. The Monte Carlo method is used in assessing project schedules, but more importantly it is used to modeling a wide variety of physical processes. The term Monte Carlo was coined in 1940, but the real algorithmwas developed at the casino's in Monte Carlo to determine how much money to have on hand for a fair roulette wheel. This was the origin of 00 on the wheel.

Since then the notion of analyzing the performance of a schedule and its cost using random numbers for the duration and assembling a Histogram of the completion times to determine the probability of completing on or before a date has been baked into the government Integrated Master Schedule processes, through DI-MGMT-81650.

Let's start with a core problem statement

"Despite Using All the Metrics Commonly Employed to Measure Cost, Schedule, Performance and Program Risk, There are Still Too Many Surprises (Poorly Performing 
 or Failing Programs) Being Briefed “Real Time” to Army Senior Leadership"

Is a quote from John Higbee around the problem of using Monte Carlo and leaving it at that. There are several important understandings needed before the MCS tools can be used:

• Your schedule has to be credible. By credible I mean a schedule that is actually executable. By executable I mean a schedule that passes the smell test of how to Increase the Probability of Success.
• By credible it also means that the network of activities that make up the schedule have themselves credibility. One place to looks (actually several) is
  • GAO Schedule Assessment Guide: Best Practices for Project Schedules
  • Planning and Scheduling Excellence Guide
  • Integrated Master Plan / Integrated Master Schedule Preparation and Use Guide

Now with a credible schedule, comes the resource loading of that schedule and of course the Earned Value Management of the work to forecast future performance. This work because the probability distributions for the activity durations in the schedule - when connected in a well-formed network - can be modeled to produce a picture of the confidence of completing on or before a planned date. Along with comes cost models if the schedule is resource loaded and Other Direct Costs are added.

This approach is mandated for Earned Value Management programs through DI-MGMT-81650 and the latest integration of the Schedule Risk Assessment (SRA) in DI-MGMT-81466B. Both require Schedule Risk Assessment, which is usually done with Monte Carlo Simulation. Method of Moments can be used, but I don't know how to make that work. I have colleagues that do.

But Here's the Real Problem

Having a Monte Carlo Simulation of your schedule and let's say a possibly credible model of the costs - By The Way most MCS tools build models of cost from the resource loaded hours, not the material or ODCs.

First, the MCS is for duration (and maybe cost) "risk." The term "risk" means the variability of the durations of the work (and maybe their cost). But there are lots of other risks besides duration risks. Technical risks, operational risks, political risks, environmental risks. All of which can have unfavorable impacts on cost and schedule. So a credible model of the actual Probability of Success for a project needs to include not only the variances in the activity durations and their resulting costs - from labor hours and maybe ODCs, but also impacts on those hours from other risks.

This problem has several layers:

• What are the risks? Can you name them, analyze them, categorize them, come up with a handling plan for them, monitor them and essentially retire them.
• How are they related? Are they all independent? Meaning each risk lives alone in its own little world of probability of occurrence and probability of impact, with cost and schedule impacts?
• How do they drive the actual variances in cost and schedule?

BTW, there are many words flying around here, and no real independent dictionary. Many authors use the same words for different things, so care is needed, and I'll try to provide the definitions as I go.

An accessible book on the topic of risk management is Mike Clayton's Risk Happens! It's easy to read has all the facts right and can be put to work.

The next step is to determine where all this stuff lives on the program. From my experience it lives in the Systems Engineering domain. These types of discussions (other than the actually scheduling and cost) belong to the System Engineers. This is because the desgin and construction of the thing is directly connected to the ability to actually make the thing on-time, on-schedule. And to do that we need to have a credible model of not only the thing but how the thing is made, the cost, schedule, processes, tooling, facilities, testing, validation, etc. etc.

This is called PLANNING and the Plan is the Integrated Master Plan.

So What Does All This Mean and Who Actually Cares?

It means that simple Monte Carlo Simulation of the Integrated Master Schedule is absolutely necessary but it is far from sufficient to assess the Probability of Program Success (PoPS). What this means is we must discover how all these moving parts interact with each other, how they influence each other, how a small distruption in one place can cause big trouble some where else.

Risk Management is Systems Engineering and that means we a Managing a System. One of the best populist books (I know I'm not a recommender of populist books, but this one is a must read) is John Gall's Systems Bible. The best Gall quote is on the cover of my Master's Thesis

In a complex system, malfunction and even total nonfunction may not be detected for long periods, if ever.

This is the case for programmatic systems as well. Programmatic systems are everything about cost, schedule, technical performance and risk.

So the first thing to do in a Programmatic System is to determine the topology of the system. What are its pieces, how are they connected, what are the dependencies (or couplings), what is the strength of these couplings, and what are the dynamics of the resulting system.

These couplings can be  “interdependent” or “coupled” work activities and their probabilistic attributes, including risks associated with those “coupled” tasks. These form a model of the program – the probabilistic aspects of each task, cost, schedule, probability of technical performance; physical performance failure, as well as process failures and their impacts in the system or subsystem failure in any dimension. These couplings have several classes of dependency

1. A  binding dependency represents the dependency which is regarded as a constraint between two dependent tasks.
2. A  non-binding dependency represents the dependency which is  not regarded as a constraint between two dependent tasks.
3. A  critical binding dependency is a binding dependency between the tasks with  zero level slack.
4. A  non-critical binding dependency is a binding dependency which is not critical binding. Following critical binding dependencies along the tasks, a critical dependency path goes through the entire process hierarchies of a project from the first level to the last level. This path is different from a critical path because it is determined without considering time aspects. However, this path provides guidance for process improvements in the early stage of planning processes when detailed data for durations are not available or the probabilistic models fail to provide sufficient confidence intervals for decision making.

These are all assembled into a model of the program. A wonderful tool for modeling all these things that make up the Programmatic Architecture is a Design Struture Matrix (DSM). Here's a notion DSM of a project. This could be a real project, so let's use it. This is taken from the handbook we use for this approach.

I discovered DSM through Tyson Browning, while he was at Lockheed Martin Aero in DFW. There is a whole discipline for DSM. Search for Dr. Browning's work and example of how DSM is integrated into the Program Controls activities of Boeing, BMW, GE Aircraft Engines, and dozens of other firms who understand that Programmatic Architecture and the Program Controls discipline is Systems Engineering.

One irony is the 2012 DMS conference is titled "Managing Complexity by Modeling Dependencies

DSM is one tool that can be applied here. But DSM fits into a broader range of tools around System Dynamics. Since all the programmatic aspects are embedded in the broader system aspects of the program, we can start to see that simple cost and schedule modeling in the absence of dependencies, in the absence of risk events which can be called causal risks or casual events all need to be integrated with the Master Plan, the Master Schedule, the physical system taxonomy and topology, and of course all the categories of risk.

This let us start to answer the question - What is the Probability of Program Success?

Planning means planning for success and for not-success

John Goodpasture has a nice post, that I'll repeat here for effect.

The Korean War was planned to last only a few days so we did not plan anything in case things might go wrong. If you plan a war without planning for failures then you are asking for trouble

Yoo Sung Chul, North Korean General, 1950

We hear a lot about risk management, the application of risk management, the lack of risk management, and all sorts of approaches - some good, most bad.

Here's some simple ways to know if you're actually doing risk management

1. Do you have a register of all the risks that have even been thought of on your project?
2. Do you analyze these risks and determine which ones are really going to cause you grief?
3. Do you keep all the other just in case they might turn into a bad boy some day?
4. Do you have an actual for real risk management process? This means a process that was copied from another no kidding real risk management process? Like one you would find at NASA, DoD, DOE, or other places that when risk happen people die, crap blows up, and billions of dollars go up in smoke.
5. No? Then stop rigth here and go get one.
6. Do you have risk review meetings month?
7. Does you risk management process address the following?
   • The probability of occurance of the risk
   • The impact of the risk if nothing is done about it. That is the risk comes true and turns into an issue - how much will that cost in tiime and money?
   • The cost needed to "retire" the risk, that is make it go away. Care is needed here, beacuse many peopl use the term mitigate. Instead use retire. That says what done looks like for the risk - it is gone. How much does that cost.
   • The probability that once you spend that money, there will still be some residual risk? What is the cost and schedule impact of this residual risk?

If you don't have a starting point here is one...

Now the hard point:

1. Take each retirement plan, and monetize it. You can have plans that don't retire the risk. No problem. You can ignore the risk, transfer it to someone else, handle it when it comes true. You have to decide this.
2. But for each risk that you are going to retire you need a schedule for doing that. You need resources, time, a plan.
3. Then what happens is the risk doesn't come true, expect with the probability remaining after you have retired it.

How To Manage In The Presence Of Risk

When We Say Risk What Do We Really Mean?

The term risk is tossed around way too easlily these days. Just like we toss around complexity, complex, or those other terms that have self induced confusions - leadership versus management, complex versus complexity, coaches versus mentors, responsibility versus accountability or agile versus lean. When we assume - wrongly that definitions don't matter, we're just being lazy.

So back to risk. First of all risk is based on uncertainty. There are two types of certainty

• Aleatory Uncertainty - that arises through the natural, unpredictable variation in the performance of the system under study.
• Epistemic Uncertainty - due to a lack of knowledge about the behaviour of the system that is conceptually resolvable.

We must distinguish between these two types of uncertainty when starting our probabilistic risk assessment for project work. The next understanding is about probabilities.

"probability is fundamentally the same concept regardless of whether it appears in the model of the worldor in the subjective distributions for the parameters. There is only one kind of uncertainty stemming from our lack of knowledge concerning the truth of a proposition, regardless of whether this proposition involves the possible values of the hydraulic conductivity or the number of earthquakes in a period of time. Distinctions between probabilities are merely for our convenience in investigating complex phenomena. Probability is always a measure of degree of belief."

In,  Mosleh, A., Bier, V. M., and Apostolakis, G. (1988). A critique of current practice for
the use of expert opinions in probabilistic risk assessment. Reliability Engineering and
system safety, 20, 63-85.

So How Do We Manage In The Presence of Uncertainty

This first thing to do is NOT assume that the system is emerging in a random manner. That notion provides no support for those funding the project. We must have a PLAN to determine is the project is headed in any direction that will result in it be DONE in any form acceptable to those providing the money.

So we need to start by answering the question what is the probability of some future event. The notion that we can't ask and answer this question is one of those other nonsense concepts. Of course we can ask and answer. You do it all the time. What's the probability of rain today is answered by loking outside or maybe looking in this mornings paper, or even on your Android at weather.com.

Our first step is to not try to manage the Aleatory Uncertainty but to understand the probability distributions resulting from this uncertainty and manage in the presence of this uncertainty.

What's the probability of completing on of before May 3rd, 2013? Good question. What uncertainties are in the current schedule? What's the probability of this projects costing $136M or less on or before May 3rd, 2013? Good question. What uncertainties are in the Basis of Estimate?

Next for Epistemic Uncertainty, we must discover what we don't know. This is called mitigating the risk or buying down the risk. This is one way Agile is connected to risk management. Let's build this gadget and see what happens. From that we'll know more and can remove some of the risk when we build the real thing. This by the way is called prototyping - Dah, been doing that for centuries, don't know software development didn't figure that out.

As an aside there is another approach using Evidence Theory, for managing in the presence of risk. This approach requires more advanced understanding of the project attributes, so for now the approach described here will be the starting point.

Just as an aside

There are differences between terms, this is why we have dictionaries.

One of my favorites is Accountability versus Responsibility. We use a Responsibility Assignment Matrix (RAM) on our programs. It is required by the ANSI-748-B Guidelines. It states who is doing what. But Accountability is not the same as Responsibility. The Accountable person is just that singularly accountable for the outcome. The person - the single point contact for making things happen. No group accountability. The group can share responsibility. But only one can be accountable. This is a notion not well understood in many domains.

Agile versus Lean - ask the Agile and Lean folks for the difference. They are distinctly different at their core. Lean manufacturing may contain some agile processes. Lean Aerospace defined at the Lean Aerospace Initiative is now called the Lean Advancement Initiative.

Management versus leadership - read the US Army Field Manual FM6-22 titled Army Leadership, Competent, Confident, Agile. Notice FM 6-22 is about agility on the battlefield, interesting? Wounder if all those agile management "leaders" have read how real agile leadership works when lives are at stake?

Leadership is the process of influencing people by providing purpose, direction, and motivation
while operating to accomplish the mission and improving the organization. Management is about the processes used to lead. There can be managers, but they most likely follow a process.

Complex versus complicated has nice clean, well established dictionary definitions. I know there is a tendency to make up new words and claim credit, by read Beyond the Hype to see that it's generally a bad idea.

So in the end the Program Manager is Accountable for managing in the presence of risk. The PM can have others Responsible for managing the risks - the Risk Manager. But in the end when the launch vehicle blows up on the launch pad, it's the Program Manager who is accountbale. Our when JP Morgan looses $2B is a risky trade it is Jamie Diamond who is accountable.

 

Risk Management Measures

Risk Management is a topic that is either ignored, done hap hazardly, and many times done correctly in domains that depend on managing risk. Of course ALL domains that work projects depend on properly managing risk.

<RANT> Doing Agile development is NOT - I repeat NOT - managing risk. They may be doing fine grained exposure to potential risks, but they are not MANAGING risks. Anyone claiming so is simply confusing risk response - by revealing short cycle functionality testing - with the management of the risk handling process. </RANT>

OK, now let's look at a Risk Matrix from a Real Program. This one is the Space Based Infrared System (SBIRS). SBIRS is the current anti-ballistic missle defense system, some of which is built in my neighborhood here in Boulder Colorado.

This is the risk categorization of the risks in terms of liklihood and severity of the outcomes. This answers the question what is likley to happen and if it does how bad is it going to be for our program?

 

 This chart is embedded in the process of Managing Risk. One process - the one used by US DoD - for these activties is shown below.

 

So when we hear we need to manage risk or we're doing risk management look to see that you have four things in place:

1. A risk management process flow as shown above
2. A Cardinal risk and impact ranking matrix define before the start seeking out risks. If you have an Ordinal risk process - like many do - of ranking relative comparisons Low, Medium, High, then you risk process is seriously flawed.
3. Finally to you have a Risk Register, where the risks are kept, along with the Cardinal information about the risk, and most importantly the Risk Handling processes or Risk Responses.
4. Do you have the risk handling planned and budgeted? If not when your risk comes true - that is turns into an issue, you'll be late and have no money for handling it. A general bad response.

 

Fail Forward

I was reading a Bloomberg News article while stuck at the airport heading home. Waiting for the inbound flight Denver where they had 2 feet of snow today. The artilce used the term fail forward. The concept is around innovation and the culture of innovation.

There were example of famous lack of innovation stories. The Western Union internal memo stating This telephone has too many shortcomings to be considered as a means of communication. The wireless music box has not ... commercial use, was the response to David Sarnoff's request to investing in the radio.

These may be urban legends, but Edison did hire dozens of "inventors" to helo him fail forward and share what they learned from these failures.

What Does This Mean For Projects?

It an obvious tennet of agile to fail early and fail often. But that is just a buzz word phrase in practice. It's of no real value in the absence of a PLAN for failure.

Remember a Plan is a strategy for the successful execution of the project. And Strategy is a hypothesis that needs a test to move forward. This is where the fail forward takes place in an actionable manner.

One program I've worked is a BioPharma contract with the US Army for an anti-virus. The Integrated Master Plan has multiple steps to fail forward in the development and testing of a molecule that is the basis of the drug. The researchers are seeking failure in a sense. They having paths they are taking - with intent - to fail forward.

So what this means is:

• We need a risk buy down plan - this plan performs specific work to buy down or retire the risk that the outcome of the project will not be reached on time or on budget. The outcome must of course be compliant with the Technical Performance Measures, otherwise no one will call the project a success.
• We need specific Measures of Effectiveness (MoE) and Measures of Performance (MoP) for the increasing maturity of the product or service the project is producing. These measures have been described before

So the fail forward paradigm must define how you are going to perform work the actually moves the project forward. Edison did not randomly go try out things in his search. He had a Plan and Strategy. It was not a random walk in the woods looking for the solution.

There were 1,000's of trials. 10's of 1,000's of attempts to invent Radar as described in the book Tuxedo Park. All the processes were fail forward, not self organized, self actualized, self guided. But structured, organized, intent of the command efforts to discover something that was not discoverable without failing forward.

This is were the agile approach is at its most powerful.

We know what Done looks like - a blub of light, a radar systems where non existed before. Richard Rhodes The Making of the Atom Bomb describes similar efforts to Fail Forward.

The critical concept, many times missed or mis-used in the agile world, comes from the MoE paper referenced above.

The design of a system is an ill-posed problem that has no solution without a set of criteria to guide choices.

Without this set of criteria to guide the choices, the project wanders aimlessly in the forest of all options. This is the role of a Capabilities statement. The role of a Strategy for fulfilling those capabilities.

The agile notion of Epic provides this. It shows what done looks like in units of measure meanigful to the decision makers. It shows how the fail forward in the pursuit of DONE.

So ask the simple question

What is the Plan that shows the sequence of processes or activities we are going to need to fail forward?

 

Your recovery Plan is Not Thorough and Not Credible

When I encounter a triage situation, like the one starting in January, I'm always surprised at how many steps it took for the program to get to the ditch. Small, simple steps that led them over the cliff without even knowing it.

The discussion usually starts with blaming the tool. In the program management and program controls business PROCESS IS KING. Many decades ago, we used to do the program perform by hand with a quadrille pad and a pen. So tools only make the work effort more efficient, not necessarly more effective.

When we're managing a program we need both thorough and credible processes for the plan to

GET  BACK TO GREEN and KEEP THE PROGRAM GREEN

Thorough Processes

For the GET TO GREEN plan to be thorough it must:

• Identify and classify sources of all data and the variances for that data. This means that measures of Physical Percent Complete must be used for all work activities at the Task level.
• Assure all related systems and processes have bi-directional traceability between the work being performed and the data the provides visibility into that work.
• Identify underlying / system-related cause(s) of gaps in this data traceability.
• Continuously focus on opportunities to improve systems, processes, and people, and if none are apparent, can explain why.
• Define a plan to address any and all opportunities to improve or explains why the organization isn't addressing those opportunities.
• Explain — when improvement plans are justified — who will carry out the plan; when that person(s) will carry out the plan; and the methods for measuring result.

Credible Processes

For the GET TO GREEN plan to be credible it must:

• Involve people closely associated with all aspects of the systems and processes under review.
• Receive support, authorization, and encouragement from senior leadership.
• Present findings that are consistent and whose conclusions the Root Cause Analysis developed during the triage process.
• Consider relevant external examples of working systems, processes, and people skills needed to keep the program GREEN. No need to invent something new.

The challenge them becomes how to KEEP THE PROGRAM GREEN. Again Process is King. Know working management processes are available for any program or project domain. If the processes are not applied correctly, consistently, thoroughly, and credibly, then the project or program will be back in the ditch soon.

I have a colleague who has a phrase he uses on troubled projects. In his best southern (below I-10) Louisiana accent, he says...

How do ya know you'rer in the ditch? The weeds are hitting the windshield. Turn the wheel to the left and get out of the ditch. From then on try and keep the car between the white lines.

Denial Just Ain't a River in Egypt

The Mark Twain quote is a good starting point for risk management. The picture below is from Formal Risk Management, David Nicholls in the Data and Analysis Center for Software, 1/1/2004. The DACS is a Department of Defense (DoD) Information Analysis Center (IAC), serving the DoD for over 30 years. As an IAC, the DACS is Center of Excellence, and technical focal point for information, data, analysis, training, and technical assistance in the software related technical fields.

We've all seen these excuses for not doing risk management, for not handling the risks, for not retiring the risks before they become issues.

I'm working on an NDIA Program Management committee that is building a Risk Management Practice Guide (not the official name yet). The committee is trying to frame a set of practices for good risk management on National Defense programs (hence the National Defense Industry Association), without defining the prescriptive steps. Instead to make the practices descriptive in the same way the Earned Value Management Intent Guide.

The problem of course is that many organizations see Risk Management as a necessary evil, rather than a project management method. The best description of how to think about risk management is Tim Lister's How Much Risk is Too Much Risk

The notion Lister states is that Project Management is Risk Management. All other activities in the project are data generating or data producing. The verb management is about managing risk.

An Example of Looking Outside the Limits For Those Pesky Swans

I found a nice example for the discussion of Fault Tolerance to Black Swans. The J-2X engine will power the current manned spaceflight vehicle - once an actual launch vehicle is available. This engine must start in space - a non-trivial problem.

From the parabolic Arc article

The Dec. 1 test firing focused on characterizing the new engine’s combustion stability, a critical area of development. During the test firing, a controlled explosion was initiated inside the engine’s combustion chamber to introduce an energetic pulse of vibrations not expected during nominal operations. Data from this and future combustion stability tests will help engineers understand more about the engine’s performance and robustness during engine operation.

Note the phrase a controlled explosion that was used to introduce a disruptive event that would not be expected during operation. But IF it ever happened, the last thing you want is the engine that puts into orbit after main vehicle separation to not work.

It's the thought process - the disciplined thought process - of finding the boundaries of the pesky swans.

More Black Swans

Some time ago there was a discussion around Black Swans and the knowability of low probability - high impact events. The Black Swan is a metaphor that encapsulates the concept that an event is a surprise (to the observer) and has a major impact. The Wild Swan is a concept that better describes what we encounter in our project work. Without a domain and a context in that domain, any discussion fails to be applicable and turns into a "populist" philosophy story.

For the Back Swan, the event is rationalized by hindsight. The critical understanding here is the surprise to the observer. Francis Bacon suggested our minds are wired to deceive. This mention of the observer rarely occurs in the popular discussion of Black Swans.

Beware the fallacies into which undisciplined thinkers most easily fall--they are the real distorting prisms of human nature.

The term undisciplined thinkers is a Root Cause of most failures. It is common to use the term Black Swan to describe some correlation that we believe (in Bacon's way) cannot be explained.But when applying a little discipline, research, and most importantly fact checking is turns out what was Black may actually be White or possible Gray.

In the current issue of Strategy & Business is an article How to Prepare for a Black Swan: Disruptor analysis can help assess the risk of future catastrophic events. (Free registration required). The clip below provides insight into the concept of a Black Swan and repeats the problem of not understanding the paradigm of black swans - a surprise to the observer.

Let's start with the Fukushima Daiichi power plant black swan. The popular press reported the earth quake was unexpected. Let's look at some articles - NOT from the popular (or populist) outlets. IEEE Spectrum is a monthly magazine of the IEEE. A special issue was dedicated to the accident.

While the earthquake itself is a complex topic, the problems at the power station are more important.

• The seawall had no safety margin.
• The diesel generators in two of the units were in the basement. The other 3 units had them on the 3rd floor. It was cheaper to put them in the basement.
• There was not back-up for the back-up power.
• The spent fuel rods were kept on site because of the politics of the disposal process, just like they are in the US.
• The killer event was that the gravity feed for reactor cooling was enabled when the earthquake struck. It was cooling the core too fast so it was turned off. Several minutes later the wave breached the sea wall, flooded the basement. The valves to turn in the gravity fed cooling loop of course required electricity. The core melted.

The earthquake was possibly unpredictable - we'll have to wait and see what the science says. But the power station and the policies were unprepared for many things, starting with the loss of site power. This is the case in many locations in the US as well. Site power is needed to control the reactor to shut down. So it went down hill from there.

The first discussion of Black Swans was prompted by Rumsfeld's comment about unknown unknowns in regards to Iraq. Like Vietnam (my war) the decision makers had little understanding and little interest in understanding the social and political ramifications of their actions. The country of Iraq is synthetic, created through the British Mandate of Mesopotamia in 1920. Tribal groups were assembled into a country. When the US came and liberated Iraq, those tribes emerged and started killing each other as well as the liberators to the surprise of the observer.

Like Vietnam, students of Iraq informed the decision makers of the hornets nest that would be released when the dictator was removed. See Fire in the Lake: The Vietnamese and the Americans in Vietnam

Black Swans ≠ Unknown Unknowns Only if it is UNKNOWABLE

The role of fault tolerant systems is to survive in the presence of failure. Fault tolerant systems are used for Emergency Shutdown, turbine controls, boiler controls. I have some experience in the domain of Fault Tolerant Systems.

This background informs (colors) my view of the notion of unknown unknowns. If you're going to venture into a realm of high risk, high consequences, you'd better damn well have considered all those improbable events that can cause your system to fail. And then in the end design that system to be fault tolerant.

So here's how it works:

• Acknowledge that Black Swans are surprises to the observer and get more observers. In fact seek out the experts in the field and have them seek out their experts and perform the needed processes to confirm that you have exhausted all the sources of faults in your system or you idea before proceeding. Table 5 of Reliability of Instrumented Safety Systems is an example.
• When the external system fails, the system in control must be not only be fault tolerant, it must fail safe. That is the system cannot fail to danger.
• If you - the observer - were surprised, ask how could that have been? Did you make every effort possible to determine causal factors that would create a fail to danger outcome? Maybe you couldn't afford to do that. Maybe you were incapable of doing that? Maybe you didn't have time to do it? Maybe you had too much hubris (in Rumsfeld's case) to admit you should be doing it. If it didn;t do all these you can't call the outcome a Black Swan.

In the absence of this approach, those pesky black swans are going to bite and bite hard.

A Counter Argument

Matthew Squair's wonderful blog has a post about probabilistic design and aviation safety. Here he restates John Downer's argument that once the risk of an extreme event has been ‘formally’ assessed as being so low as to be acceptable it becomes very hard for society and it’s institutions to justify preparing for it (Downer 2011).

So the revisiting the Black Swan and Fault Tolerant and Fail Safe concepts...

5.(a) … (1) in any system or subsystem, the failure of any single element, component, or connection during any one flight should be assumed, regardless of its probability…

This means we MUST consider that all possible failures will happen and engineer the system to deal with this. This is the failure mode of Fukushima Daiichi. The improbable was possible.

 

Two Mandatory Readings

The Association of Project Management (APM) has produced a guide for integrating Risk and Earned Value.

I sit on a NDIA Committee in the Program Management Steering Committee (PMSC) that is addressing the same issue - how can risk and opportunity be integrated with Earned Value.

The core issue here is what does it mean to be managing a program using earned value if the risk of completing on time or on budget is not considered in the Performance Measurement Baseline?

The establishment of the Performance Measurement Baseline is described in a anchor diagram.

1. Establish the project context
2. Develop the Statement of Work and initial WBS
3. Develop the top down budget and schedule
4. Identify the strategic level risks
5. Perform  the initial risk analysis
6. Revise the top down budget and schedule
7. Integrated the WBS and OBS
8. Create control accounts and perform risk analysis
9. Update the initial Performance Measurement Baseline
10. Update and baseline the project risk register
11. Approve the PMB and the Management Reserve (MR)

The second must read document is the new - Version 2 - Practice Standard for Earned Value Management. I started working on the agile and EVM appendix, but discovered the PMI Practice Guide is not based on ANSI-748B. Fortunately the Practice Guide dropped that appendix, so it was a moot point.

What is important is that the Earned Schedule Appendix made it thanks to the work of many, but especially Kym Henderson.

What's interesting though is ANSI-748B is mentioned 3 times.

• Page 8 - contingency
• Pages 46 and 47 - contingency reservers
• Page 52 - contingency

This contingency paradigm is common in construction and the Department of Energy.

No where else in the Practice Guide is 748B mentioned. I find this a big gap, since 748B desribes 32 criteria for a successful EVM System.

The chapters of the Practice Guide address many of the criteria found in 748B

• Organize the project
• Assign responsibility
• Develop schedule
• Establish budget
• Determine measurement methods
• Establish Performance Measurement Baseline (PMB)
• Analyze project performance
• Maintain PMB

These activities can be found in the 32 criteria shown below. The colored items are the 11 mandatory criteria needed for the success of any project independent of Earned Value.

1. Define the WBS - Chapter 3.4.3

2. Identify the organizaiton - Chapter 4

5. Integrate the WBS and OBS - Chapter 4.3.2

6. Schedule the work - Chapter 5

7. Identify the products and milestones - Chapter 2.4 (basics), Chapter 3.2.1 (Charter)

8. Set the time phased budget - Chapter 6

16. Record the direct costs - Chapter 9.2.2

23. Determine variances - Chapter 9

25. Sum data and variances - Chapter 9

26. Manage action plans - Chapter 10

28. Incorporate changes Chapter 10

Just a remider about risk

If you are not managing the project with Risk Management, than you're driving in the dark, with the lights off, while looking in the rear view mirror.

From The Integrated Project Management Handbook, Dayton Aerospace, Feb 2002.

Quote of the Day

One of the biggest planning mistakes people make is to treat the unlikely as impossible.

from Deferred Consumption, Better Living Through Statistics

When we're laying out the Integrated Master Schedule and assigning risk categories and levels of risk within those categories, there is always a tendency to say "we'll this risk is such a low probability, that I'll just skip over it."

Now an IMS with every conceivable risk is not much use in practice, but without knowing why you rejected a specific risk, is itself a risk. You have to know WHY, before you can saw NO.

 

Unceratinty and Risk

Risk and Uncertainty are two different things. Uncertainty does not necessarily mean risk. In the mathematical sense the terms are used differently in practical terms the meanings are the same.

But treating them the same creates a problem. When we can't tell the difference between Risk and Uncertainty, we can't tell what we have control over and what we don't. So let's look first at the theoritical difference between risk and uncertainty.

Risk is aleatoric and uncertainty is epistemic. What the heck does this mean?

• Alleatoric is Latin for alea = dice. This is a chance event. There are a know set of possible outcomes, we just don't know which one will come up this time when we throw the dice. There are only 6 possible outcomes for a single die. Add another die with its six possible outcomes and we have 36 possible outcomes. That's all there is, but we can't tell which of the 32 outcomes will happen when we toss the dice - unless of course they are loaded.
• Epistemic is episteme in Greek for knowledge. Uncertainty is the lack of "knowing." We have uncertainty of which of the 36 outcomes will happen when we toss the dice. But the dice can only land in one of the 32 configurations.

So the relationship between risk and uncertainty goes like this

Risk is measurable uncertainty

Uncertainty is unmeasurable risk

In the dice world we know the probability distribution for each of the 36 outcomes.

So we know our chances of throwing the two dice that total a 7 or 11 (called a natural) and the pass line wins. If a 2, 3, or 12 is thrown on the first roll that is craps and the pass line losses. All other combinations 4, 5, 6, 8, 9, and 10 become the point and the roller must repeat those number before a 7 is thrown.

For project elements we must know the difference between risk and uncertainty.

• There is uncertainty that the new processor will be fast enough to handle the load because there is no way to test its capacity before we put it to work.
• There is a risk - a 78% chance - that the turkey's we order will not arrive on or before November 23 in time for preparation for the 18 people coming to our Thanksgiving dinner. This is an actual number, but I don't really know the probability that the organic turkeys will arrive when needed.

This is the basis between Statistics and Probability. It is the Probability of Success we are after. But we can't answer that question without an understanding of the underlying Statistics of the processes we are trying to manage.

When we don't know the difference between risk and uncertainty and we don't know the difference between statistics and probability, then it will be likely (a probability) that our project is going to run into trouble and we don't know it is coming. We can't separate those things that are naturally variable and those thing that have probabilities of occurring that we can control.

• There is a probability that when I go to the garage and take out the lawn mower, that it will not have enough fuel in the tank to mow the entire lawn. There is a uncertainty about the level of fuel in the tank. There is a risk that if it is too low, I can't complete the mowing job. The uncertainty can be turned into a certainty by looking in the tank to see the fuel level. The risk can be handled by filling the mower tank from the storage tank.
• There is a probability that when I start my drive to the airport on Sunday to attend the NDIA conference in Maryland there will be ice on the road - since it is winter here. If I don't provide a buffer in time then there is a risk I will miss my flight. The uncertainty can be addressed by looking at the weather forecast on Saturday night, and the risk can be handled by leaving 30 minutes before my normal departure.

Quote of the Day

“It appears incontrovertible that understanding failure plays a key role in error-free design of all kinds, and that indeed all successful design is the proper and complete anticipation of what can go wrong.”

Henry Petroski Design Paradigms Case Histories of Error and Judgment in Engineering