Herding Cats

Quote of the Day

There are no points for predicting rain, only building lifeboats. - Warren Buffet

The Problem with the "Best Estimate"

The traditional approach for estimating cost or schedule starts with the "best estimates" for the Most Likely durations and costs for tasks. This is the standard approach for PERT based projects, for approaches suggested by the PMBOK or PMI style of project management.

The steps usually go like this:

• Build a network of the project’s activities
• Determine the “best” estimate of the duration for each activity in the network
• Compare the activities’ “best” estimates to find the critical path
• Sum all the “best estimate” durations of activities on the critical path
• Define this sum of tasks to be “best” estimate of the project’s schedule duration
• This will almost always be optimistically wrong (remember the discussion about Anchoring)
• Or it is pessimistically wrong
• Either way – it is wrong

The problem with the term "best" is there is no standard definition.

For each activity, the "best" estimate is ...

• The “most likely” duration – the mode of the distribution of durations? (Mode is the number that appears most often)

• It’s 50th percentile duration – the median of the distribution? (Median is the number in the middle of all the numbers)
• It’s expected duration – the mean of the distribution? (Mean is the average of all the numbers)

These definitions lead to values that are almost always different from each other Rolling up the “best” estimate of completion is almost never one of these.

Durations are probability samples not single point values

We know this because ...

• “Best” estimate is not the only possible estimate, so other estimates must be considered “worse”
• Common use of the phrase “most likely duration” assumes that other possible durations are “less likely”
• “Mean,” “median,” and “mode” are statistical terms characteristic of probability distributions

This implies activity distributions have probability distributions. These are random variables drawn from the probability distribution function (pdf). The “Actual” project duration is an uncertain quality that can be modeled as a sum of random variables. The PDF may be known or unknown.

These durations are "educated guesses," but rarely do they have known underlying probability distributions.

Quote of the Day

The difference between failure and success is the difference between doing something almost right and doing something right. — Benjamin Franklin

Independent Program Assessment Office

I'm assembling a training briefing for an upcoming Integrated Baseline Review for a client. This will be a 200 to 300 page Power Point briefing with narrated slides in the standard style of our defense practice.

During the research phase, I came across a great site for guideance in assesing the credibility of the program's plans.

Independent Program Assessment Office. This is what is many times missing from major commerical IT enterprise projects. On the Cost and Schedule analysis section of the home page, is a set of processes that can be applied to any complex program with inherent risk.

Many authors who speak about the "nonsense things" that project managers do, should point them to this site. These would include:

• Having deterministic schedules and cost to levels of detail not physically possible,
• Capturing estimates by asking simple and many times simple  minded questions without the underlying statistical assessment of the confidence in those numbers,
• Not having a Plan-B for all critical activities,
• Not understanding that making trade offs between cost, schedule, and technical performance is a Ponzi scheme, and finally,
• Confusing effort with results, by failing to define what done looks like in units of measure meaningful to all the participants.

If you have a Talking Dog, you don't need a control group ...

Lynda Bourne has a post at the PMI Voices of Project Management around Unrealistic Detail. I'm always puzzled when the conversation starts with "doing unrealistic things, and then expecting anything other than disappointing results."

This is the classic "doctor, doctor, it hurst when I do this," paradigm.

The second statement that piqued my interest was It's impossible to accurately predict the future. Yet, many project managers continue to try. This of course is both not true, and follows the quote above.

Of course you can predict the future it's done all the time, every day. From predicting the arrival of the school bus, to predicting the completion of a project and the cost of that project. What Lynda fails to include in the sentence of the probability that the prediction is accurate to some level of confidence.

The conversation goes like this There is an 80% probability of completing on of before the 3rd week of November, 2014 with a 7% error band. Or there is an 80% probability of the program costing $320M of less, with a 6% error band.

The probabilistic forecasting of cost and schedule is the core process applied to most of the program we work. It is mandated by DID 81650 and well most agencies procurement processes.

The fact that many project managers try to predict in the absence of probabilistic models is of course nonsense. Of course Lynda's statement...

They create schedules that implicitly state a task will be completed at 3:30 on a Tuesday afternoon, in four months. Or they predict that the total cost of their project will be precisely $10,986,547.55.

This is course nonsense, if there is no confidence interval and error band on that confidence. So Lynda is correct and at the same time completely wrong.

The PM might say, "Doctor, doctor, I try to make predictions with no confidence intervals and no error bands and I keep being disappointed in the result." The Doctor would tell you "Well stop doing that, and read to guidance for probabilistic forecasting in deterministic schedule and cost models."

It's really that simple. And like one of my favorite banners at Rocky Flats

Don't do stupid things on purpose

The 4+1 Questions That Must Be Answered for Project Success

There are 5 (4+1) questions that must be answered for any project to have a credible chance of success.

In the previous post and in all the discussions around project failure in IT and other domains, I wonder if these questions had answered. Or even if they were asked.

CEO/COO/CIO's not confident in IT Project Managers, really?

CXO' not happy with project management? Really, what a surprise. PMForum reports a study - not sure in the statistical confidence of the study - that CXO's see PM as critical, but have little confidence in PM's.

Why? Here's why from the hands on experience in IT and hands on experience in defense and space. Our defense programs are troubled all the time. It's a project. That's what projects do. But here's the problem in many IT organizations. Project Management is an afterthought. In defense PM is mandated. There is no choice, we must have a Performance Measurement Baseline, we must report progress to plan every month at the least, and in some programs every two weeks. Physical Percent Complete, not some developers opinion of when he'll be done.

The Statement of Work, the Work Breakdown Structure, the Control Accounts, the Work Packages that produce outcomes, the oversight of all this, and all the other "Program Planning and Controls" roles are mandated.

All this, and we still have issues. Changing requirements, technological problems, subcontractor problems, staffing problems. With all those going on in IT AND none of the oversight and progress reporting, no wonder the CXO's have no confidence. They get what they deserve. PM costs money, don't spend the money? Then as Dr. Phil say "how's that work'in for ya?"

Time to face up to the fact the projects require Project Management. Project Management is a verb. It is NOT fundamentally touchy feely stuff many PM experts talk about. It's about the hard core definition of what done looks like, the units of measure of done, the processes that measure done in terms meaningful to all the participants, corrective actions to the plan when done is not being reached on the planned date for the planned cost.

Folks in heavy construction, defense, space, high risk domains all have a chuckle when they hear about the silliness of IT PM methods that don't contain the immutable principles of project management.If you're managing projects and don't have the project laid out along these lines, you're headed for the ditch and probably don't know it. No amount of interpersonal skills development, no amount of "agile development sticky notes," no amount of anything is going to help in the absence  of:

• A description of the needed capabiltiies of the system in the form of scenarios or use cases
• A description of the requirements needed to fulfill the Use Cases or Scenarios
• A Performance Measurement Baseline of the Work Packages and Planning Packages needed to perform the work.
• A process to measure physical percent complete in units meaningful to all the project participants.
• A continuous risk management process for both technical and programmatic activities of the project.

What Good is an Observation Without a Suggested Solution?

In the continuing conversation with vocal critic of government contracts, I've come to the conclusion there is a personality type that loves to point out the obvious problems in the world - in this case management of government programs. While this approach makes for good forum fodder, it is pretty much worhtless in the actual world of a program manager.

This approach is supported at times by conference presentations that restate the obvious. The conversation - of sorts - started with the NASA PM Challanger presentation Mega Project Estimates - A History of Denial, Glenn Butts. The PM Challenge conference has many useful items, and sometimes not so useful.

The referenced presentation is one of those surveys of the wreckage from poor management approaches. This approach of course only shows the "bad" projects and fails - intentionally it seems - to include successes for statisical comparison and calibration. In the NASA world there are many success as there are in DoD, DOE, and other government agencies.

The orginal poster (the vocal critica with lots of titles behind his name) usually starts the converation about how poorly government projects are run, with overruns and waste. Yes this is many times the case. And the reference to the Butts presentation is only one of many stating points.

But here's where the conversation goes in the ditch:

• We know there are problems. That's obvious. In fact it's so obvious that restating the obvious is now considered a school-boy approach. "Every school boy knows" was a quote in some famous physics book I had in grad school.
• But now that we know projects are troubles - cost, schedule, technical performance - what is the suggested solution?

Deafenng silence is the response. Even the Butts presentation is silent beyond restating the obvious solutions. Yes we know we need to get better estimates. We know the initial estimates are flawed to the point of being uunusable. But what are we doing about it?

This is a common problem in all project domains. It is part of the process. What the real problem is the politics of projects is the source of the problem. Not the lack of credible estimates. Just the opposite. Credible estimates abound. Having those credible estimates become the baseline estimates is a political problem. By political I mean "low price wins," "best value wins," "Firm Fixed Price contracts," annual funding cycle allocations of limited money. Things like that drive estimates that are not credible.

But here's the real problem. When a someone speaks about all the problems in the government contracting domain, those same problems are in the commercial world as well. It turns out when someone has a axe to grind - anti-government politics - it's hard to discuss solutions. Because solutions then remove to reason for the anti-government axe.

It's much too easy to sit on the side lines and point out the problems in the world. There are certainly enough to go around. It is much harder - and many times simply too hard - to proffer solutions to those problems. And even much much harder to put those proffered solutions to work in real projects in an attempt to improve the probability of success.

Using the College Football Analogy

So no more "arm chair quarter backs," get up off your duff, put on the pads, read the play book, and start running the plays on the field. Get banged around by the opposing team, support your team mates and quit bitching about how bad the team is playing the game.

Then maybe something will improve. Then again, maybe not. Maybe it's the role of the arm chair quarter backs to just sit and point out the flaws in others.

Improving the Probability of Program Success (PoPS)

Pat Weaver has a nice post about project failure and the people side of project success. People are necessary of course, but far from sufficient.

Success on non-trivial projects starts and ends with process. This has been shown time and again.

Pat mentions two failures I am familiar with - Hubble (I assume the mirror) and Challenger.

Both the Hubble mirror and Challenger were first PROCESS failures. A read of the mishap reports state this. Then people failures from the absence of process.

Had Hubble followed the process of Operational Test and Evaluation (OT&E) has defined in  (NPR) 8705.2A, the mirror would have undergone integration test and evaluation.

Had Thiokol pushed back on the Marshall Space Flight center management and insisted - according to contract - on a Safety and Mission Assurance stand done, the "mishap" would have not occurred.
The right people are necessary but far from sufficient for success.

IT is another world unto its own. No formal engineering discipline, abhorrence of process and procedure and all that "emergent" nonsense on enterprise systems, results in well deserved failure.
We work a 600 million $ software development program, a $300M bioscience program, a $100M helicopter software intensive program.

We're usually over budget and behind schedule, but it is never a surprise in the way an IT $200M right off of ERP typically is.

It's people, process and tools. But it starts and ends with process on all non-trivial projects. People without process is a "beach party." This is lost on most project where they put people first. Establish the process, hire people capable of executing the process as well as executing their technical role and you've got a fighting chance at getting closer to success.

The official US DoD term is "Increasing the Probability of Program Success (PoPS)"

The Complete Failure of Our Public Education System ?!?

Sorry, couldn't resist.

Can Projects Really Be Successful?

A well known and very vocal Project Management instructor made a statement about Government Projects always being late and over budget. Then making a not so subtle suggestion that his competency based assessment approach, versus the PMI approach to certification, is solution to better project management.

Man, do I wish that was the case. Here's my hands on experience in the government (DoD, DOE, DHS) experience on the program controls side.

• Requirements changes impact programs in ways that are not desernable when first suggested. We see this all the time. "Let's make this change from one real-time control data bus to another real-time control bus for all the right reasons" - faster, better, cheaper. But then the second and third order impacts start showing up months later.
• Techncial architecture impacts are many times unknown. "Let's scale up the number of processing units from 20 to 100. It's only a 5 times increase, so let's up the cost and other impacts by 7 to account for unknowns. Turns out the real impact was not 7 but 14, twice what was estimated. Materials impacts, process flow impacts, the overall thermal impacts were huge impacts on the program.

The  processes used to manage a program are absolutely necessary for success. Earned Value Management, prorgammatic and techncial risk management, Systems Engineering, Measures of Effectiveness and Measures of Performance driving the Techncial Perfomance Measures.

But necessary of OK, what about the sufficient conditions?

• Senior management must behave in rational ways.
• Change control is mandatory, not as a "constraint," but as a process to reveal impacts
• All parts are connected, no matter what the suppliers say. A Systems Engineering paradigm is mandator for all things other then simple projects.

These, and many others, are not directly related to project management processes, even the general purpose ones found in PMBOK^®.

At the bottom of this discussion the "necessary AND sufficient" conditions starts and ends with:

• Past performance,
• Extensive experience,
• Processes and tools, and
• The necessary leadership to put all this together.

Now how can we increase the probability of project success? This is the 64 thousand, 64 Million, and maybe 64 Billion dollar question.

Simple, and many times simpilistic soultions are just that simple and many times simplisitic.

We have to remember ...

For every complex problem there is an answer that is clear, simple, and wrong. - H. L. Mencken

There are no soltions to complex problems. Complexs are part of our current life paradigm. All the easy problems have been solved. So rarely will there be a problem that can be solved with a simple solution.

Good Advice for Project Management

Project Management means different things to different people. From the social and humanistic aspects of managing the people working on and around projects, to the gritty details of a Performance Measurement Baseline, with in integrated cost, schedule, risk, and technical performance measures.

One advantage of working the in Space and Defense business is the social and personality aspects are minimized. Not gone all together, but certainly much less than commercial domains, where egos get in the way many times of getting things done. In A&D "done" is clear and concise - the machine flys away, the product does it's job as defined in the SOW, with measurable outcomes, soldiers are using the product, things like that. Black and White, either it flys or it doesn't fly. Of course it's not that simple, it never is.

But one thing that is different in A&D is the focus on the processes and tools. I'm working a moderate ($300M) Army program through January when we get to the Integrated Baseline Review (IBR). While poking around looking for some ideas for training our customer, I came across PM Lotus's "Setting Baselines in Microsoft Project."

The Baseline is just what it says it is, the "baseline" of the cost, schedule, and technical performance. Now MSFT Project is not the best at managing baselines. In fact it's pretty poor at managing baselines. No change control, no traceability, not much in the way of mandated documentation. But it's popular and most of our programs use MSFT Project.

Drill down into this site, there are some very good articles for those interested in the programmatic aspects of project management. Like Checklists.