Herding Cats

Sources of Variance

Cost variances on projects come from a variety of sources. Here's a picture of some of the drivers that are common in many domains.

The management of projects means you have to address these and many other activities to answer the questions posed by 5 Immutable Principles:

1. Where are we going?
2. How do we get there?
3. What resources are needed?
4. What impediments will be encountered?
5. How is progress to plan measured?

These impacts on cost variance are part of principle 1, 2, 3, 4, and 5. They are part of all the principles, because without a credible understanding what the project is going to cost in the end, the management processes of the project itself are no longer credible. You're then just spending other peoples money with no connection to producing value - yes the business value.

Stop Motivating People - They Hate It

A post on Curious Company had an interesting statement

Stop Motivating People They Hate It

This resonated with me for several reasons. Motivation comes from internal forces. Attempts to motivate externally - that is I'm trying to motivate you to do something - only works at certain ages and in certain situations. We were fortunate enough to discover Parenting with Love and Logic early enough to apply some of the processes to our two children when they were accepting of our efforts. The result has been good. Both are in college, both are graduating more or less on time,  one this May, one 1 year late since he added a dual major (Biochemistry and Cellular Biology, and then announced a Math Minor). But good kids and motivated to learn, share, grow, and play. I asked the Love and Logic guys if they were going to do a course for adults. The answer was it's too late, you have to get your kids on the right track, adults can't learn motivation in a way that is lasting. This by the way is why 97% of diets fail. A local riding group Phoenix Sports usues cycling as a recovery process, motivation seems to work there, but there are many external as well as internal motivators, not the least of which is to become a competent cyclist in an environment where pros share the road with amateurs.

In the business world I don't want to work with anyone who needs motivation. It's a waste of time, annoys the person and annoys me as to why I need to motivate someone I pay good money to do their job. But in reality it's a fact that some need motivation.

In the book Making the Impossible Possible, there is discussion around getting people to follow the leader and at the same time create their own solutions. This is called Positive Deviance. This may be the case of motivating, but it's in a different sense than getting people to do the right thing. PD is about as close to motivation as you should come in the business world.

In the Target Process site there is a quote let's try and do the best in the world at <something>. This can be applicable to anything actually, it doesn't have to be a fancy dancy agile project, er, software development management, tool. It can be regular work.

We just finished constructing and submitting the Contract Funds Status Report for one of our Army clients. A couple of 100 million dollars for funds had to be reconciled against the past work, the future work, the planned work, the spend rate, and the Management Reserve needed to delievr on-time, on-budget, and have the thing work as defined. This sounds like dreadful work to some. But for us it's not dreadful, it's quite exciting.

The past CFSR's were less than acceptable to the government, were not that great at forecasting the future need for money, and were pretty much lame. After 3 weeks of grueling work, with several 20 hour days, the team landed soft with a CSFR that is a thing of beauty. Every Control Account, every Work Package, every Activity foot and tied to the Contract Budget Base (the award value), tied off the the labor rating (how is charged for each person), the subcontractor contracts, commitments, and invoices, and all the other nitty aspects of  spending the governments money.

Our team worked hard, played hard, and had a sense of accomplishment, learning (more than wanted to sometimes), comradeship, shared suffering, and shared success.

But no one and I mean no one on the team needed motivating to get their job done on time and 100% correct (the federal government has no tolerance for sloppy work, since there are instructions). We had a mission. We had a vision of what DONE looked like. We each knew our role, the role of the others, the tools, the data, the process. Motivation processes will do nothing to increase the probability of success. Period, it's a waste of time. Get the right people, in the right jobs, get the wrong people off the project (off the bus as Jim Collins says), and have all the advisors, by-standers, those with opinions but not contributions - out of the room, out of the building.

We almost did what was suggested in the post

You should not track time, you should not estimate effort, you should not set deadlines. You should trust people, set ambitious goals and help them learn. We did that.

We had to track time and we had to have a deadline, because we had a due date and we had limited hours to do the work, so we had to have a plan of the week, then a plan of the day, and in the last 3 days a plan of the hour, if we were going to get the CSFR out the door on time and 100% correct.

So take care with the advice of others if they are not working EXACTLY in your domain and context, or their advice is not based on immutable principles. Use their ideas as just that, ideas. But always get the right mission, vision, people, processes, and tools, so no one needs any motivation.

Estimates and all that...

Esther Derby has a post on estimating. The thread seems to go that estimating is helpful but estimates are not.

Let's start with estimating in a domain where we are spending other peoples money. Better yet an environment where we are spending the money of a sovereign - the United States of America.

But first let's talk about the first principle of estimating anything.

Any single point estimate, without a variance and a confidence interval on that variance is a nonsense number - a number that has no meaning.

So, when someone gives you an estimate, you MUST ask and they MUST answer the question What is the confidence on that estimate? The conversation would go something like this...

What is you're planned completion date for the XYZ feature?

We'll be completing on or before the 3rd week in November of 2014 with a 80% confidence and a ±5% error on that confidence level.

It's the variance that is missing in the discussion Esther is having.

Regarding Other Ideas in the post ...

• People turn estimates into targets - don't let them do that. Estimates are estimates. Targets must have buffers. Those buffers must be developed from the probabilistic assessment of the risk in the estimates.
• People construe estimates as bids - separate those when appropriate. Estimates can certainly be a bid. A defense contractor provides a proposal with a cost volume in the proposal. That is a submittal for the proposed cost. It is a contractual number at submittal, and a definitized number at IBR (Integrated Baseline Review) and a Baseline numbers after "snapping the baseline" at the IBR. Changes to the Baseline happen all the time for all the right reasons.
• Inappropriate precision in estimates implies people know more than they do - this is a common problem when those making the estimates don't know how to make estimates - get people who can make credible estimates or train those you have. In the estimating world there are 5 levels of estimating confidence:

1. Low Risk: Historical Data, Vendor Quotes
2. Low to Medium Risk: Comparative Analysis
3. Medium Risk: Level of Effort (sizing by labor hours)
4. Medium to High Risk: Parametric estimates, Cost modeling
5. High Risk: Engineering estimates

• People game estimates - don't let them do that. Have an estimating process, follow it, oversee it, vet it, risk adjust it, learn how to do estimates, look to sources that can show you how to do estimates - the Software Engineering Institute is one place, the Software Technology Support Center and their journal Crosstalk is another.

Space flight contractors can guess the cost of a Low Earth Orbit spacecraft within 15% by weighing the vehicle. Strange as it may seem, the developers of flight avionics can guess the cost of a Guidance, Navigation, and Control (GN&C) box to within 10% by counting the lines of code. Yes those dreaded SLOC estimates, that have been calibrated over 3 decades. For those looking to see how software is estimated in domains where software dominates, is mission critical, and developed on tight budgets, and tight schedules, with emerging requirements look to:

• Seer
• Price
• QSM
• ACEIT
• Society of Cost Estimating and Analysis
• COCOMO II
• AACEI

Credible software estimates are done all the time, every month for example on large weapons and space systems.

If you want to move toward similar capabilities, become knowledgeable of how this works, adopt what works to your environment, and DO NOT let anyone get away with sloppy estimating. If you do that, you get what you deserve.

Quote of the Day - Sketch of the Project's Value Stream

In preparing for battle I have always found That plans are useless, but planning is indispensable - Dwight D. Eisenhower, 34th President, General of European Forces, WWII.

Craig Brown has a nice post titled Sketches go a long way in generating understanding. But where's the Sketch of the Project's flow? In the Integrated Master Plan paradigm, the increasing flow of maturity of the deliverables of the project is represented by a diagram that looks like this.

What this diagram tells us is what is the "value flow" for the work effort. Each chunk of work activity "Pilot" with "Data" gives us a deliverable that can be assessed for its maturity. By maturity I mean a general term. This can be usefulness. It can be verification of the concept. It can be a test of the willingness of the funders of the project to proceed.

Agile would call this an iteration or maybe a release. But what is shown here is the end to end map of the project. In this case an ERP system for insurance claims processes. The Vision of DONE in just that DONE. The senior management of the insurance copy can see to the end of the project. They can see what it means to be DONE. As well the work flow to get to DONE is there as well.

We need to have some data and a pilot before enrollments are developed. This "dependency" structure is part of the Planning process. And of course the Planning processes results in an Integrated Master Schedule. Not for the whole project. That would be nonsense, no one can see that far into the future.

But we do have a Plan for the future and this is it. We need to do these big chunks of work in the order they are planned for the business to receive the planned value from this project. Oh yea at or near (within agreed variance) for the planned cost, since we asked the Board of Directors for the budget we needed and they approved it and put that cost in the GL as an obligation and that obligation got reported in the 10-K for the stock holders to see that 100's of millions are being committed of their money.

Is there a home agile development in this approach? Sure at the work package level that is 3 to 4 levels down. This there a home for Earned Value? Absolutely, that's how progress to plan is measured down at that work package level. This progress to plan is progress against the planned budget, the Budgeted Cost of Work Scheduled (BCWS). BCWS times Physical Percent Complete = Earned Value. We "earn" our budget, if we show up on time, don't over spend, and the thing we're building actually does what we said it was supposed to do. What a concept. No hand waving, no whining about this or that. Do what you planned to do, once that plan is agreed to and baselined.

Emergent requirements? Absolutely. Plans change. Requirements change. That stuff on the right is way out there in the future, it depends on the success of the stuff on the left. No one in their right mind woudl commit to a cost and schedule for stuff that has not been vetted through "past performance" and verification of capabilities.

Just don't commit to this beyond your ability to actually make them appear. This is called rolling wave. It's the basis of Earned Value, it's the basis of enterprise project management, DoD 5000.02, and just plain olde good Project Management.

But if you consider things like detailed planning of current rolling waves, applying Earned Value to measure physical percent complete against cost and schedule "hog wash." Then try doing this and not explaining to senior management how much it's going to cost in the end, when are we forecasting completion (within a variance), how many people are needed (300 or so to start), and what is the sunk cost along the way that can be compared to the expected business return.

Try doing this without those and many other "measurement" processes, and your career as a PM will be short lived. Try doing this without measures of performance based on budgeted cost, physical percent complete, and planned percent complete and if you can, then you'll be very close to Earned Value.

Project Success

Attending the EFCOG (Energy Facilities Contractors Group) Project Management Working Group. One discussion was around the success factors for projects. There were two obvious assessments:

1. Accountability and rigor in the management of the baseline and the process for executing the baseline.
2. Experience of project management

The accountability and rigor is the source of visibility to the performance of the project and the capturing of feedback for corrective actions.

The "baseline" is a concept missing in some domains.  The baseline is the metric benchmark against which project performance in terms of time, cost and technical performance is measured. Without the baseline, there is no description of what "done" looks like. Without some description at a level sufficient to control the outcome means that the only measure of progress is the passage of time and the consumption of money.

This is the case no matter what the project domain or context of that project. Controlling the baseline does NOT mean no changes. It means knowing what happens when a change is requested to the cost, schedule, and technical performance

Experience is obvious. But the right kind of experience is critical. The notion that a PM can manage any project is nonsense. The PM's in the DOE work in three major areas: Science, Environmental Management (EM), and National Nuclear Security Administration (NNSA). Performance varies between these groups and projects within the individual groups.

In the end, success starts with "past performance." This means I've made it work before, in some way close to what the problem is today. There is rarely a project that is complete "green field," - something that has NEVER been done before.

One recurring theme for projects is the phrase more planning up front. This doesn't mean Big Planning Up Front, just better planning up front. The credibility of the plan when the project starts has to be just that CREDIBLE. The nonsensical notion that we'll figure out how much this project will cost and when it will be done as we go along, is just that nonsensical. This only works if you're spending your own money and you don't care if you overrun. When you're spending someone eles's money, you'd better can some notion of cost and schedule near the start of the project.

You wouldn't hire a lawn mowing service if they couldn't give you some estimate of the cost per month of their work. Why would you engage in an IT project without some upper bound on the cost and duration?

Are You Doing Project Management?

Long ago I had a Blog conversation with an expert in MSFT Project Server about cost management in projects. His position was managing the details of cost with a resource loaded schedule was a waste of time. I was embedded TOO much in the IT world to remember my construction and defense roots. Now that I've come back to A&D for the past 3 years, I remember that conversation fondly as laughable.

I'm editing an appendix to the PMI Earned Value Management Practice Guide, and here's a clip from a paper that defines all we need to know about projects and project cost. From "Combination of Project Cost Forecasts in Earned Value Management," Journal of Construction Engineering and Management.

Reliable cost estimates are essential for effective project control and the management of cash flows within the project and at the company level. Conventional approaches to project cost forecasting, which rely on detailed information developed for a specific project (the bottom-up estimate or inside view), often result in cost overruns. It is argued here that the inside view project cost estimates should be adjusted by combining them with the outside (or top-down) view of the project, which is based on statistical models of historical project data. This paper presents a probabilistic cost forecasting method and a framework for an adaptive combination of the inside view and the outside view forecasts of project cost using Bayesian inference and the Bayesian model averaging technique. During the project execution phase, the Bayesian adaptive forecasting method incorporates into the predictions the actual performance data from earned value management and revises pre-project cost estimates, making full use of the available information. Qualitative examples are presented to demonstrate the validity of the proposed method as a tool for effective project cost prediction and control.

If you're on a IT project, someone, somewhere is interested in cost management. If you're working a government contract we know who that is - finance. If it's construction we also know who that is - finance and the customer.

There is no such thing as doing project work and NOT managing cost. The notion - wrongly - that Agile projects are not managed in the traditional sense of cost and schedule only occurs when there is no one who cares about cost and schedule.

The key to success of any project contains Five Immutable Principles:

1. Where are we going?
2. How do we get there?
3. Do we have enough time, resources, and money to get there?
4. What impediments will we encounter along the way?
5. How do we know we are making progress?

All these questions have a component of money.

Cost Estimating

The discussion of estimating - cost and schedule - has produced references usable in a variety of domains. As a background here's the guidance in the domain I work.

This "mandates" an Estimate At Completion with a Best Case, Worst Case, Most Likely. As well a Schedule Risk Analysis (SRA) is mandated.

And just for clarification for those favoring "hand built" 3 point estimates, the 3 points estimates stated in DID-81560 above are generated using the "ordinal range" method. The outcome from this processes are the percent ranges - upper and lower - of the probability distribution (Triangle) used for the Monte Carlo tool (Risk+ and @Risk for Project). So we have the three points, but they were not gathered by asking the engineers what they should be. The Systems Engineering, Control Account Managers, and Technical Leads, plus historical data, plus some statistical analysis produced the ordinal ranges for 5 classes of programmatic risk. That is then applied to each class of work, and then loaded into the baselined Integrated Master Schedule.

The point of all this is that producing credible estimates is part of the culture of the work we do. It's still just as hard, just as complex, and just as annoying as any other domain. But doing it, and doing it right is what is expected. It's part of being credible. So for those gathering variance ranges by hand, if the credibility - statistical confidence, auto-correlation, assessment of all the biases and influences discussed in The Psychology of Risk and Sources of Estimating Error can be addressed, then you're on the path to credibility.

Sources of Estimating Errors

Starting with Tversky & Kahnemann (1974), "Judgment under Uncertainty: heuristics and Biases," there are sources of error that should lay the foundation for abandoning the simple and many times naive 3 point estimate approach to work duration and effort.

• Representativeness - occurs where probabilities are evaluated by the degree to which A is representative of B.

The last time we did this, we say durations that looked like this. Or, We've asked the guys down Florida and they think we should be able to get this done in 1,400 hours, that what it took them the last time they did it.

• Insensitivity to prior probability of outcomes - we ignore the past statistical behavior of our work efforts. This is ignorance of Bayesian statistics.

I know they took 1,400 hours, but we've learned a lot from their mistakes, so we can do it for less. We don't really have evidence, but our gut feel is they were not very good at their job and took too long.

• Insensitivity to sample size - this is a very common error, where the estimator makes a forecast independent of the sample size.

My students - all 23 of them - answered a survey I wrote about a behavior of problems with Earned Value, and they came up with answers I'll apply to a broad set of situations. I sampled the people I know and they came up some interested statements of what works and doesn't work. We hand picked 12 managers in our firm, gave them a survey we put together from some outside suggestions and we'll be making changes based on their answers.

• Misconceptions of Chance - the belief that random processes represent the core behaviors of a process, while observing a short sequence of outcomes.

I've seen this happen 3 or 4 times in other places, it's got to be the same thing happening here. It's happening in all kinds of places, so it's got to be the same here. "For every up there is a down," let's just wait and we'll be back on track pretty soon.

• Insensitivity of predictability - the future can be predicted intuitive predictions

I've seen this happen before, it's got to be the cause of what's happening this time too.

• Illusion of Validity - the observations we're seeing are a good fit with what we're expecting.

We interviewed a list of people selected by management, and they support the what management thinks is going wrong here. Management has a target budget and schedule in mind, and when we asked the developers - after hearing the management numbers - they came up with about the same numbers.

• Misconceptions of Regression - all those random processes add up to a average we can live with.

When we get all the estimates in, we can find the average variance and use that to forecast the work we're going to be given in the future. This concept is actually mis-represented in a government cost estimating guidebook. Adding a sample set of distributions results in a "normal" distribution.

• Biases due to the effectiveness of a search set -when asked to recall some data or event, the one they come up with is what is most familiar to them.

We met some guys in the cafeteria and they had lots to say about how we should estimate our costs

• Biases due to the retrievability of instances - the data I have at hand is the data I used for my forecast.

We had all the data from the projects they did in Dallas, and used that as our sample data for foresting our performance.

• Adjustment and Anchoring - the estimate is based on the starting value and is then adjusted to get the final answer.

Management gave us a starting point for the cost and schedule, so we need to improve that estimate.

• Biases in the evaluation of conjunctive and disjunctive event - in conjunctive events probabilities are over estimated.

Our rule of thumb has served us well in the past - turns up to be biased

• Anchoring in the assessment of subjective probability distribution - confidence intervals are over confident.

Let's not broaden the variances too much on those cost estimates, it just doesn't feel like right in this situation.

These topics are from Micheal Axelsen's review of of Tversky & Kahnemann 1974 paper, "Judgment under uncertainty: heuristics and biases."

These topics will hopefully put an end to the simple minded 3-point estimates extracted from the people tasked with doing the work. They certainty have a contribution to the estimating process. But serious problems arise when you take those numbers and start making management decisions. In exactly the same way you create serious problems when you take management directive and start makes estimates "anchored" on the bounds of cost or schedule they what to have you produce to.

Credible Estimates

John Goodpasture has a post on estimates from Fred Brooks. Fred's quote, which I'll repeat for effect, from John's site is

It is very difficult to make a vigorous, plausible, and job-risking defense of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers.

In many domains, estimating is performed in ways similar to Fred's description. Here's some phrases found in RFP's from the Defense Department:

• The estimate must be easily traceable from the lowest level at which the Offeror’s estimate was substantiated to the CLIN.
• When using historical data, the Offeror should describe why the system is comparable to the proposed program. This includes a functional and technical comparison explaining the differences as well as similarities between the historical and the proposed system. Also include an explanation of the relationship between the analogous element cost and the total program cost.
• Adjustments to derive the proposal estimate relate to reasons and justification for any adjustments made to programmatic, technical and actual cost data for the historical system. The Offeror should provide the basis and document any adjustments applied to the historical data (e.g., complexity factors and normalization methods) that reflect the characteristics of the proposed system. This includes an audit trail sufficient for the Government to reconstruct the proposed estimate and judge its credibility.

Now these are heavy duty words, probably not applicable outside the procurement of similarly heavy duty weapons systems.

But as John points out through Brooks' quote, to do otherwise is complete nonsense on anything but a trivial project.

So wjy does this continue to be the practice in many areas? Some would say, it's because IT projects are software projects and you really can't estimate software projects. To this I say frankly - BS. The program we work int he defense are predominately software development projects. Software development with emerging requirements, changing requirements, state of the art technology, all the things IT project managers encounter.

My personal opinion is

Project estimating, and project management is hard work. Software development is fun work. Still hard but fun. Project Management is hard work, with almost no fun.

So why work on something hard if it's not fun? Secondly, there is no real downside for missing estimates in the commercial IT world. Oh sure, there is hell to pay, people get fired, the project gets canceled. But the company rarely goes out of business, people never die, launch dates are rarely missed. I'm not talking about silly commercial product launch dates, I'm talking about launch dates to Mars, launch dates for NRO (National Reconnaissance Office) launch dates. Those kind of launch dates.

What's missing is the discipline - at appropriate levels of course - to "do the right thing." That's the root cause of most problems in the commercial world - pure missing discipline.

It's the Micky Rooney school of management. Where Micky and Judy Garland are in the alley behind a NYC theater, when they say to each other and their friends...

Hey guy, I have an idea, let's put on a show

It's that, let's get started and see where this goes, kind of approach to project management. Where it goes many times is straight into the ditch.

The Final Word on Cost (and Schedule) Estimating

This is not really the final word in the sense that there are no more words to be spoken. It's the final word in showing that point estimates are fundamentally useless. The quest to produce a credible estimate for a project's cost and schedule is not solved by serializing with work effort. It is produced by constructing a credible Cost Estimating Relationship (CER) model.

Here's an example taken from training materials in the domain where such things are used everyday.

It speaks for itself...

You Have To Know What Things Cost Before You Can Know What Their Value Is

Brian Kennemer has extensively commented on the PM 2.0 post of his experience of including cost in number in projects he is familiar with. This trigger my thinking on what the real problem is with many IT projects.

It comes down to a fundamental principle of any business, from running IT maintenance to flying to low earth orbit with 3 astronauts on board.

If you don't know the cost of something
You can't know its value

This is fundamental to every activity of every business process, including IT. It is also a fundamental failure mode of many IT departments.

When a CIO is asked "why are you here and why should I keep you here?" The answer better be, "because I provide X$'s of value contribution to this organization."

Value is derived from cost for every product or service effort ever developed or performed.

Undercorrelated WBS Elements Means Understated Cost Risk

The current issue of the National Estimator, The Society of Cost Estimating and Analysis, monthly magazine has an eye opening article titled "Correlating Work Breakdown Structure Elements."

A Key aspect of cost risk is correlation between work breakdown structure (WBS) elements.
...

For example .. with a 30 element WBS has an assumed zero (0) correlation between all elements, but the actual correlation between WBS elements is 0.2, the total cost risk will be underestimated by 60%

This should be a frightening fact for all of us who assume no correlation, or worse have no recognition that such correlations are important.
    So when the question of "why are projects late and over budget," is asked this is one of the answers, along with a myriad of other answers:

• No understanding of the needed schedule margin, derived from Monte Carlo Simulations
• No understanding of how the increasing maturity of the products or services are being measured in units meaningful to the stakeholders
• No management of the inherent risks of the program
• Insufficient partitioning of work efforts to reduce the coupling between work durations - "How Long Are You Willing to Wait Before You Find Out You're Late?"
• The inability to determine the capacity for work, so all work estimates are underestimated.

• And the final killer result - multitasking creates hidden correlation between WBS elements.

The last element alone, should be reason enough to start to understand the correlations, and remove the multitasking. After that the real correlations, need to be ferreted out and placed in the cost model.

Time Is NOT Money

A great presentation of Time and Money on projects.