Herding Cats

Learning Is Now More Important Than Knowledge

When I encounter a "subject matter expert," I've now learned to assess if this person is a "knowledge holder," or a "learning individual." This is what informed that "learning." This came to my from a friend and neighbor who is a graduate of the University of Chicago in economics and spoke at this conference. 

In our business - project and program management - there are lots of "knowledge" providers, assessors of knowledge, frameworks for measuring our knowledge. What is needed now is how to learn to make better decisions about our projects. This is not going to happen without a decision making framework for the core principles, practices, and processes for project management. The current Bodies of Knowledge, self-proclaimed providers of better BOKs and assessment tools, authors (I'm one of them), and experts in the field, all need to reassess what we are doing to increase the Learning abilities, beyond the just the possesion of knowledge and the dispensing of this knowledge to others. 

How can we actually learn to improve our probability of succes. The Decision Scientist for project success - as suggested here - needs to have art, science, and the scale to provide actionable information for the decision makers for the project.

We have most of the data we need, what we don't do is make decisions on this data, or even know how to make decisions with this data. We report it, but don't use it to make decisions. Numbers matter, but leadership needs to use the numbers. 

Related articles

Competency Model For Project Managers

Averages Without Variances are Meaningless - Or Worse Misleading

The current Flap over the Public Debt and Growth paper originally produced by Reinhart, Carmen M, Vincent R Reinhart, and Kenneth S Rogoff. 2012. Public Debt Overhangs: Advanced-Economy Episodes since 1800. Journal of Economic Perspectives 26, no. 3: 69-86, brings out some core issues we face in the management and assessment of complex projects using performance data.

As well, there is a flap over the use of Excel for modeling econometric processes. Which was addressed more in the NPR segment. More on Excel and modeling later.

In the original paper - which has now been updated, there are several key words that trigger red flags in any credible statistical analysis of dynamic systems like the economy or complex programs. There is an assessment of the original paper. Let's start with the response to the original paper from the original authors addressed to the author of the critique.

This brings us to the core conceptual issue, which Herndon, Ash and Pollin argue greatly biases our results. They argue that we (they) use an “unconventional weighting of summary statistics.” In particular, for each bucket, we (they) take average growth rates for each country and then take an average of the result.

• Averages without variances are not very useful. We all know the story of measuring the most likely temperature in two locations Cody Wyoming and Trinidad-Tobago and the misrepresentation of Averages.
• Since economies and projects are driven by stochastic processes, we must model their data as random.
• Everyone claiming to forecast the future or make correlations between random processes must have on their shelf and have demonstrated to have read How to Lie with Statistics, Darrell Huff.

So ignoring for the moment the very naive statement by two Harvard professors of averaging the averages in the absence of the variances, and then making projections of the correlations, there are other problems.

• Using Excel for modeling can certainly be done, but there are much better ways to model statistical and probabilistic processes - R for one. MathLab is another.
• Excel also suffers from the in auditability problem. If you build a spreadsheet and send it to me to make management decisions, I have to look at every cell to determine the right formula, right range for that formula, and right data is in the right rows and columns for that formula to work. A pain in the ass for simple spreadsheets. A nightmare for larger ones. This is simply bad modeling. Especially since R is free and an academic version of MathLab or Mathematica is under $300. Come on Harvard professors, earn your reputation.

So What Does This Mean for Project Performance Analysis?

There are pictures in the links above to show that projects are statistical processes. So here's the bottom line:

• All numbers in projects and econometric models are random numbers.
• Correlations between the source of these numbers may also be random or at least changing over time - nonstationary stochastic processes.
• Models that don't consider this information are probably not that useful and may actually - and I'll be crass here - BOGUS.
• If we are ever to produce credible models of how projects work - cost, schedule, risk, and technical performance models - they must be credible statistical models. That means averages and averages of averages are simply not allowed without the variances and even higher order moments and the correlations between the generating functions of these random variables.

So however, this turns out with the Bad Math people at Harvard, we've got to do better.

• The current Earned Value Management processes don't consider the statistical nature of the performance indices. This is bad. Here's a simple and very understandable paper "Performing Statistical Analysis on Earned Value Data."
• As well, the calculations of future performance use cumulative numbers which wipe out the varainces, the current period which is a point sample to compute the Estimate At Completion. This is not only bad it is naive math.
• And then we are surprised when things don't turn out as expected.

Some final thoughts

• Correlation is not causation.
• All numbers are random numbers drawn from a known or possibly unknown population. If you don't know the population statistics your assessment of the numbers is different than if you do.
• All random numbers need variance, standard deviation, and higher order moments to be considered credible as sources of any analysis.
• Drawing a picture of the average of anything is very sporty without the proper statistical analysis.
• Read Huff, reread it, read it all the time. Also go buy a good statistics and analysis book. Start with Advanced Statistics Demystified.
• If there is an academic paper that is being used for public policy has not been peer reviewed, and then throw it away. The Reinhard and Rogoff paper is the basis of Paul Ryan's economic plan. He may have really good points, but it built on sand.
• Same advice from self-proclaimed experts in anything, especially project management. If there is a PhD thesis that has ZERO references in CITESEER, ignore it. 
• Read more in the linked references below to see how bad statistics can go even more bad.
• Download R, download the free books, learn how to think and act with credible statistics. Learn how many sample you need, learn how to assess the needed population statistics confidence levels, learn how to make decisions based on confidence levels not absolute numbers. We have a 70% confidence of completing on or before a date, or we have a 70% confidence or completing at or below of specific cost must be the answers when management asks about cost and schedule performance.

Related articles

Deterministic versus Stochastic Trends in Earned Value Management Data

Everything is a Random Variable

Memo to Reinhart and Rogoff: I think it's best to admit your errors and go on from there

Reinhart-Rogoff Response to Critique

A Point Measures Need A Variance

The Reinhart - Rogoff Study Controversy

Reinhart-Rogoff Data Problems

More Reinhart and Rogoff

Another Nonsense Statistical Survey

Alert - Just So We Don't Get Confused As To The Source Of Our Little Problem

Quote of the Day - Sketch of the Project's Value Stream

In preparing for battle I have always found That plans are useless, but planning is indispensable - Dwight D. Eisenhower, 34th President, General of European Forces, WWII.

Craig Brown has a nice post titled Sketches go a long way in generating understanding. But where's the Sketch of the Project's flow? In the Integrated Master Plan paradigm, the increasing flow of maturity of the deliverables of the project is represented by a diagram that looks like this.

What this diagram tells us is what is the "value flow" for the work effort. Each chunk of work activity "Pilot" with "Data" gives us a deliverable that can be assessed for its maturity. By maturity I mean a general term. This can be usefulness. It can be verification of the concept. It can be a test of the willingness of the funders of the project to proceed.

Agile would call this an iteration or maybe a release. But what is shown here is the end to end map of the project. In this case an ERP system for insurance claims processes. The Vision of DONE in just that DONE. The senior management of the insurance copy can see to the end of the project. They can see what it means to be DONE. As well the work flow to get to DONE is there as well.

We need to have some data and a pilot before enrollments are developed. This "dependency" structure is part of the Planning process. And of course the Planning processes results in an Integrated Master Schedule. Not for the whole project. That would be nonsense, no one can see that far into the future.

But we do have a Plan for the future and this is it. We need to do these big chunks of work in the order they are planned for the business to receive the planned value from this project. Oh yea at or near (within agreed variance) for the planned cost, since we asked the Board of Directors for the budget we needed and they approved it and put that cost in the GL as an obligation and that obligation got reported in the 10-K for the stock holders to see that 100's of millions are being committed of their money.

Is there a home agile development in this approach? Sure at the work package level that is 3 to 4 levels down. This there a home for Earned Value? Absolutely, that's how progress to plan is measured down at that work package level. This progress to plan is progress against the planned budget, the Budgeted Cost of Work Scheduled (BCWS). BCWS times Physical Percent Complete = Earned Value. We "earn" our budget, if we show up on time, don't over spend, and the thing we're building actually does what we said it was supposed to do. What a concept. No hand waving, no whining about this or that. Do what you planned to do, once that plan is agreed to and baselined.

Emergent requirements? Absolutely. Plans change. Requirements change. That stuff on the right is way out there in the future, it depends on the success of the stuff on the left. No one in their right mind woudl commit to a cost and schedule for stuff that has not been vetted through "past performance" and verification of capabilities.

Just don't commit to this beyond your ability to actually make them appear. This is called rolling wave. It's the basis of Earned Value, it's the basis of enterprise project management, DoD 5000.02, and just plain olde good Project Management.

But if you consider things like detailed planning of current rolling waves, applying Earned Value to measure physical percent complete against cost and schedule "hog wash." Then try doing this and not explaining to senior management how much it's going to cost in the end, when are we forecasting completion (within a variance), how many people are needed (300 or so to start), and what is the sunk cost along the way that can be compared to the expected business return.

Try doing this without those and many other "measurement" processes, and your career as a PM will be short lived. Try doing this without measures of performance based on budgeted cost, physical percent complete, and planned percent complete and if you can, then you'll be very close to Earned Value.

Anchoring and Adjustment in Estimating

Josh from PMStudent.com has a post about estimating. There is a discussion of using 3 point estimates.

Some time ago, I came to the conclusion that making 3 point estimates is a bad idea. Instead use confidence intervals for risk ranges and drive the variance from those bands with a Monte Carlo Simulator.

This is a critical topic that goes to the credibility of any estimate no matter the domain.

The starting point is to understand the concept of anchoring and adjustments to the estimate and their impact on false optimism (or pessimism). The paper to start with is "Anchoring and Adjustment in Software Estimation," Aranda and Easterbrook.

When you read this, you may rethink the approach that asks the estimator the Most Likely estimate and Optimistic and Pessimistic estimates.

The original work in this area is from Tversky and Kahneman and their "Prospect Theory." A good place to look for some details (besides their work) is Against The Gods, Peter Bernstein, in Chapter 16, The Failure of Invariance."

It's time to move on from the classic "simple" approach found in places like PMBOK(r) to the processes used in other places that depend on probabilistic and statistical analysis - engineering, finance, medicine, physics.

We Have Met the Enemy and He is Us

From the Project on Government Oversight, a POGO moment.

It just gets worse every day.

Our Energy Crisis

Before starting any discussion around alternative energy, nuclear energy, any kind of energy, you need to watch Dan Nocera's talk. Not so much for the solution but for a description of the problem.

Standish Report

I've written several times about my personal dislike of the Standish Reports.

• Standish Report and Naive Statistics
• Finally, A Challenge to the Standish Report
• Project Failure Rate

The current assessment of the Standish Report is:

"The Rise and Fall of the Chaos Report Figures," J. Laurenz Eveleens and Chris Verhoef, Vrije Universiteit Amsterdam, IEEE Software, January/February 2010, pp. 30-36.

The core findings are:

• There are misleading definitions - the definitions used by Standish are based solely based on estimation accuracy of cost, time, and functionality. Standish labels projects as successful or challenged, suggesting much more than deviations form their original estimates.
• Unrealistic Rates - estimation accuracies are questionable. These estimates are "one sided" and neglect under runs for cost and time and overruns for the amount of functionality delivered for either the under or over runs on cost and schedule. Institutional biases exist and are unaccounted for in the Standish numbers. (see Software Engineering Economics, Barry Boehm, Prentice Hall, 1982)
• Cost Forecasting - using independent cost forecasting databases, it can be shown that the Standish number forecast a 59% success rate for cost adherence, when the baseline database from actual projects - 140 projects from 2005 to 2006 at a large financial services firm - follow Boehm's "conical shape" for accuracy and produce an estimating quality factor of 8.5. This means that half the projects have a time-weighted average deviation of 12% or less from the actual forecast.
• Accuracy Again - using their definitions of cost and schedule overestimation and large functionality underestimations perverts the estimation accuracy.

All of this information and the references point to a statistical assessment gap. The core problem is:

• There is two-sided measurement from a statistical point of view. They only ask about failure rates, not the success rates. This would produce a biased outcome.
• There is not a statistical sample selection process. Send out a survey, get back the responses, and total the number. How many surveys are needed to establish a credible sample size in order to establish the confidence intervals.

Estimating

The estimating process for project cost and schedule is usually one of the poorest skills for the project. Not because it is difficult or even because it is not well understood. But because there is a myth that estimating is not really possible.

 

Estimating cost and schedule requires skill, experience, and a method (there are many) and some tools. Here's how it starts:

• You must have some notion of what done looks like. This can be "done" for this iteration. Or "done" for this rolling wave. Or "done" for this release.
• You must be capable of controlling the scope of "done" within the period of performance of the development process.
• You must measure productivity - the capacity for work - in some unit that allows you to forecast future performance.
• You must be able to engage everyone in a conversation about estimating the future using the past.

In the absence of these basic capabilities, estimating is simply not possible. The the suggestion that estimating in the software development world is not possible, will come true.

NYT Chart on the Economy

Frank Patrick got this first, but all of us with the New York Times subscriptions should look at this clever charting processes for the Current State of the Economy.
    This reminds me of the "bulls eye" charts in winSight we used for plot the Cost Performance Index and Schedule Performance Index from Earned Value across the life cycle of large programs. Look at Page 20 of this presentation on applying EV to IT projects.

OMB IT Investment Management

    The Office of Management and Budget has released an IT Investment Dashboard for US Government IT investments. This is driven by the OMB Circular No. A-11, Part 7 Planning, Budgeting, Acquisition, and Management of Capital Assets.
    The notion of making public the IT investments and their performance is of course long over due. At the same time non-government firms would be well served if an internal version of the IT Investment Dashboard were used for the portfolio of IT projects,

A Program Manager's View of AIG

    In the application of Earned Value of programs, there is a value below which we move on when trying to get all the control accounts to balance for the period of performance. Usually if you can get all the numbers to line up for 99% of the total value, its called DONE and let's print the Contract Performance Report (CPR) and send it to the customer. 1% error depended of course on the size of the program. There are time when 5% error is acceptable. And other times when less than 1% is needed.
    But to spend time futzing around with 1/1000 of 1% is not only a silly it is a waste of time. So why is the US Congress burning huge amounts of time and money on 1/1000 of 1% they gave AIG? Mostly likely to cover their asses for not having language in the bailout bill in the first place. Probably because those sitting behind those desks have actually never really managed anything. Nor would we want then to. No because they shoudl not, but because they are representatives of the people and should be focused on doing the people's business. Making Laws, no micro managing what is clearly a bone head decision by AIG.
    National politics is all about image and sometimes about substance. So getting your face on TV is part of image. If someone would go on TV and state in clear and concise terms that spending time managing the clearly poor decision of AIG's imbattled president is a wast of time, then maybe we poor souls here in the field woudl have some convidence things will improve.
    Indepedent of one's politics, we're loosing sight of good program management practices - focus on the big picture in times of crisis.

A Bold Faced Lie

    Watching TV today and saw an ad for the Cadillac Escalade Hybrid compared to the Mini Cooper - S I assume. The ad by GM claimed the Escalade gets better millage than the Mini - 20MPG in town.
    I don't own a mini, but I do own a Honda Civic SI, with a comparable engine and drive train. 198HP and a 6 speed manual transmission and a very snappy response curve when you can heal and toe the drive train.

    While the SI is not as snappy as the Volkswagen GTI and certainly not comparable to the STI WRX or Evo-10 which have near race car performance, it is a scary ride when pushed to the limit. with a 8,900 RPM red line in a slight drift in the off ramp, a close ratio 6 speed and moderate breaking, it does have a near "no body roll" turning behavior. It'll kick ass in the unaware BMW and Audi driver waiting for the light to change in the common Colorado intersection, where two lanes go down to one in a few hundred feet. Chirping the tires in the first 3 gears is common. Add high performance snow tires and it's a fun ride for commutes and general back country road driving.
    The ad GM wants us to believe that GM has cracked the code for mileage in a SUV platform. When you go to the EPA mileage site you can compare the two vehicles. Turns out though the some kinds of numbers are being cooked.
2009 Mini S - 26MPH city and 34 highway. I can get 33 out of the SI on a 50 mile commute on flat highway. Down to 28 highway on the Denver to Breckenridge route. No tongue in the 2 liter Honda engine, so 5th gear running at 5,600 RPM going up I-70 to the tunnel.
    Having owned a Suburban (350, automatic) it had crappy torque as well. Our Honda Pilot out performed the Suburban in all ways except pure volume.
    The Esclade hybrid is rated at 20MPG city and 21MPG highway. It also give up towing capacity.
The Mini convertible is much less 23 City and 29 Highway. But both Mini's are turbo charged and the SI is not - if it were, it would produce 230HP and be really scary.
    So what's with GM. Are they trying to convince us that the Esclade is now a "green" car?
    Lies and damn lies is my response. I'll record the ad next time its on to do a more detailed assessment. I was some dumb founded when I foirst saw it, I forget to push record on the DVR.

$35B to build advanced manufacturing? Too Late!

See the video from the Detroit News on an advanced Ford Plant - in Brazil.