Herding Cats

Standish Report and Naive Statistics

    The 2009 Standish Report is out. Many Blogs have made comments. All obvious comments. But there is a fundamental flaw in any report of this type:

The concept of a assessment baseline is missing from the conversation.

• Is there a Performance Measurement Baseline?
• Does this baseline represent a credible plan for cost, schedule, and technical performance?
• How would we know how many projects "should" fail in the population of all projects?
• Is the project sample used by Standish representative of the total population of projects?
• If so, what's the correlation between the sample population and the total population in terms of failure rates?

    These are basic statistical analysis questions. Of course the report does not state these for the right business reasons, but the result is now limited in terms of its interpretive power.

The first big question is - Is there a control group?

• How many projects should fail as a matter of course of being members of a underlying  population?
• What is the proper statistical population for projects?
• What is the underlying distribution of project success versus project failure for this population?

    In the absence of this information - being made public - the Standish numbers are just a marketing document for their training and consulting services. In the same way the weight loose, hair growth, lawn growth, and other sales approaches are frankly - Nonsense.

    The easy starting for discussion of this sort is How To Lie With Statistics. This little book wias written in the 1950's and is still an important contribution to the discussion. After reading this short book, you too will be suspicious of Standish style reports.

    Just remember how you feel, when you see and ad for a weight lose products reporting how much weight will be gone in the coming 6 weeks with their product.

    In the end though Standish provides no solution, just an 800-number to call for the "magic elixir"

The Mathematical Side of Managing Projects

One common concept in project management is the three independent elements of a project:

• People
• Processes
• Technology

The people side is being hashed out in previous posts. Processes of PM are debated all the time and technology for managing projects is a hot topic as well.
    There is a orthogonal dimension here. The mathematics of projects. This includes

• Cost models
• Schedule models
• Risk models

Models for people are essentially intractable. The Complex Adaptive Systems paradigm has no closed form solutions to any of the underlying math that has any relation what so ever to the activities and outcomes of project management. At the moment CAS is a sociology paradigm with some possibilities for application.
    But the cost, schedule, and risk models do have tractable forms that are applied in many domains. The sources of materials for these applications include:

• Journal of Cost Analysis and Parametrics from the Society of Cost Estimating and Analysis and International Society of Parametric Analysis
• Journal of the International Council on Systems Engineering
• The Association for the Advancement of Cost Estimating
• NASA Cost Estimating Handbook 2008
• JPL's Parametric Cost Estimating Handbook
• NASA Systems Engineering Handbook

The point of these references is that the process of managing projects is well developed in other fields. Fields that include software intensive products and services. Projects that have emerging requirements, unstable funding, and all the attributes found in commercial IT projects.
    The guidance developed in these domains provides the discipline for the teams to move beyond just software development and into system development. One of the primary failings in IT projects is the lack of understanding of the systems impact of decisions. This is not systems in the sense of hardware and software, but systems int he sense of "Systems Engineering."

The Mathematics of Project Management

I've an interesting exchange of sorts on Jurgen Appelo's blog. It started with me making a comment about the application of Complex Adaptive Systems to projects and software development. I've somewhat familiar with CAS through several channels - my physics background tool me into deep inelastic scattering in the late 70's. This the use of electrons to look inside protons to see what they contain. They contain quarks of course, but in those days how many, what are doing in there, and what can we learn from them was still emerging. My job as a lowly grad study was to write FORTRAN code to decipher the pictures on film from the camera. I also worker from there in defense, since I did not have any original ideas required to be a post-doc. Missiles flying at hypersonic speed through varying air densities is sorta like compressible fluid flow, which is a cousin of the math found in CAS adaptive systems.
    Of course the science around CAS is not real A science, it is many sciences. Some are physics - compressible fluid flow and turbulence, some is biology, some is social science, and some is the co-opted terms used in the agile community. As you move from left to right in that list, the science is diminished and the hand waving increased. I usually loose interest right at the boundary of the compressible fluid flow equations and the start of hand waving.
    So when I mentioned that the application of CAS to projects and software development was "sketchy" in my experience the agilest on that blog came out and started down the normal path of "how could mathematics possibly be used to manage projects?"
    Well here's how math is used to answer several questions:

• When w ill this project be done?
• What will it cost when it is done?
• What are the "hot" spots in the schedule that we need to look after while we're trying to get done on time?
• What confidence do we have that the cost and schedule numbers are credible?
• How does that credibility increase with the passage of time?
• What can we do to assure the credibility is actually increasing with time?

The answers to these and other question like these starts with a Monte Carlo Simulation (MCS) tool. Monte Carlo tools are probabilistic assessments of dynamic systems. The first Monte Carlo simulations in the modern ear came from nuclear weapons research during WWII. Ulam, Fermi, von Neumann, and Metropolis developed an algorithm to simulate the collisions of neutrons inside the atomic bomb, without having to actual explode the bomb.
    The MCS approach to schedule simulation goes like this:

1. Construct a well formed activities network. This means no widows or orphans. Every task has a predecessor and successor, except the first and last task. These are usually Authorization to Process and Program Termination.
2. Assign a probability distribution to the duration of each task. This usually starts with a Triangle Distribution is there is no information about the task. Other distributions can be used is there is knowledge.
3. Assign the Most Likely duration. This is the duration that is "most likely" to take place. This value can be derived in a variety of ways. Too many for this post
4. Assign the Lowest and Highest durations around the Most Likely.
5. Indicate which tasks you'd like to learn something about.
6. Run the Monte Carlo Simulator. We use Risk+ and @Risk for Project. PERTMaster is another.
7. The "watched" tasks create a sampling of completion times through the following means:

• A random duration value is drawn from under the curve for each task.
• This number is entered into the DURATION field of MSFT Project
• Do this for all the Tasks
• "Push" F9 to calculate the network
• Record the finish date for the "watched" tasks in a histogram
• Do this 500 to 5,000 times

After the selected number of "runs" build a histogram of how many times a specific date came up. This is the Probability Density Function showing the confidence that the watched task will complete "On or Before" a specific date.
    This approach can show the project management team the confidence in the schedule. If the plan - with the schedule margin - is to complete on or before 10 Nov 2009, and the MCS shows a 14% confidence of completing on or before 10 Nov 2009 - we need to talk.
    That's all there is "talk." The MCS doesn't fix this. The MCS doesn't solve the problem. What it does is provide an assessment of the confidence of meeting the planned completion date.
    For cost the work is similar. Build a probabilistic model of the cost for each work element. ONLY in LEVEL OF EFFORT projects does Time = Money.
    This is the hidden outcome of agile project management.

• Time boxed scheduling - fixed itertaions
• Fixed resource loads - the same number of people for each itertaion
• Fungable resources - everyone does more of less the same job

Time = Money.

This is rarely true on any non-trivial project.

Finally the MCS tools can tell us how much margin (cost and schedule) is needed to raise the confidence to say 80%.

1. Build a schedule with no margin tasks on the critical path
2. Run the MCS
3. Determine where the 80% confidence end date is
4. IF the end date is to the left of the planned for (hoped for) date, then see what that gap is.
5. IF it is something non-trivial - say 45 days for a year long project, then you need 45 days worth of margin tasks to hold the planned completion date.

Of this is a notional explanation. Real projects, with real costs and schedule, real risks, and real changing requirements will require more work. But this approach is used every week on aerospace and defense programs that are software and hardware intensive.
    So to answer the question on the Blog - where the math model. It's mandated by DID-81650 and several other DFARs (Defense Federal Acquisition Regulations) for programs we work.

Modeling of Cost and Schedule

Alistair Cockburn referred to a paper posted in PMForum. (Along with Alistair's site, PMForum is a MUST subscribe site for anyone claiming to be a project manager) The paper suggests the causal source of the "fat tails" for distribution of work effort. This paper possibly provides a theoretical source of the practice of using the Triangle distribution for the source of Monte Carlo simulations.
    But several issues need to be addressed:

• The notion of infinite tails is not statistically sound. In practice triangle distributions represent bounded values for durations.
• The notion of infinite tails is not managerially sound. Models of duration and cost have management limits. In most cases these set at 125% of the baseline. After 125% overrun management intervention takes place and a new baseline established. In federal contracts Nunn McCurdy in breach is invoked. (There are numerous examples of this not being the case in commercial projects, but that's just BAD management).
• The notion of Linear Work is not used in any credible cost and scheduling model. The approach here in aerospace and defense is to bound the Work Packages with not cross more than one accounting period - usually 45 days. The allows the modeling of the Triangle distribution to be applied to a "not to exceed" duration within the two months around a single accounting period. This way the linear aspects are "forced" while the non-linear aspects are applied at a higher level. This does not prevent non-linear behavior but the lowest level is "linearized" and the macro level become tractable.
• Figure 2 of the paper is the most useful for our work. Modeling the difficulty of the work with the effort needed for that work is a Systems Engineering problem. Typically historical data is used. NASA has a extensive data base for "calibrating" work effort versus the technology. The issue of course comes when new work is needed - "inventing new physics" type work.
• The Stochastic behavior of humans is an interesting approach. The Monte Carlo simulations are driven by Probability Distribution Functions - Triangle is best used in the absence of underlying information. But true stochastic behaviors seem a bit extreme. Stochastic process are "true" random processes without correlation functions between internal variables and especially external coupling that evolve with time. This is unlikely in practice, but would make a great research topic if the historical data could be captured. But this paper is about the theory, so not an issue at this point. The author considers the task size, productivity, and task difficulty. A good starting point for this topic is Introduction to Stochastic Processes, Lawler, Chapman Hall, 1995. I actually looked at it a few months ago when dusting the bookshelf.

Here's a graph we use to show the coupling of input variances to output variances for a single random variable. Random processes are assumed to be stationary and therefore not stochastic.

    The challenge here is the define the coupling between work, productivity and difficulty. The paper does not speak directly to that. And unfortunately that is what is needed for any credible planning process. At the moment that information comes from subject matter experts.
    On the cost side there is similar "coupling" models used on the programs we work. This picture is taken from a training presentation provided by MCR Corporation - a thought leader in this subject area - cost, schedule, and technical difficulty modeling.

    My final - and constant - difficulty with the paper is the weak reference section. There is a wealth of material at the Association for the Advancement of Cost Engineering, INCOSE's Systems Engineering Journal, NASA's Cost Estimating Handbook site, and other "internal" cost and schedule modeling sites. The paper had two references.
    The paper is a "must read," if only to start the conversation about the probabilistic nature of Cost, Schedule and Technical Performance Measures in the presence of probabilistic risk models.

Probability and Statistics in Project Estimating

Bill Duncan posted a piece on PM Student titled Estimating Effort. It's a multiple part series that addresses some of the issues around estimating cost and schedule in a project context. Multipart pieces are ineffective in the medium of a Blog. But Project@Work did this all the time, so we'll have to wait for the next installation to discover the real meat. In the mean time, I've skipped to the end in the best Atkinson tradition.

    In the first installment Bill states:

Many of the terms in estimating are used inconsistently or  imprecisely.

This is the case in most project management domains, especially IT. An estimate is defined as

A judgment or opinion as to the quantities of a person or a thing

The example of "guessing" the gender of a 6'6" person and a 5'2" standing outside your door is slightly off, since the variability involved in estimating are not in place in this example. The variability between two people standing outside the door in terms of the gender is 0. The persons are either male or female.
    This introduces the misunderstanding common in all naive estimating processes of confusing statistics with probability. Several posts in the past are background

• Confidence Intervals
• More Statistics
• Mean or Median

Understanding the differences between probability and statistics is critical to understanding how to make credible estimates of cost, schedule, technical performance, and finally project risk from these variables. What the project stakeholders want to know (or should want to know) is: What's is the confidence we'll come in on-time, on-budget, and the project will produce working results (be they products or services).

Confidence Intervals

    I had a few conversations over the past weeks about "cause and effect" processes. "If you do this (X), then this will result (Y)." "X causes Y," "X and Y are related in some way," "when X appears, we should see Y,"   
    Really, how can we tell? Is there something else that might have caused Y? Maybe Y occurs randomly a number of time, and what you saw was a random occurrence. This situation is common in many areas:

• Process improvement
• Health supplements
• Quality improvement programs
• Consulting services
• Performance tuning of hardware
• Peddling style changes from our new cycle coach

    So how can we tell that the cause, X was somehow connected with the effect, Y?

Measures of Statistical Significance, that how

    Remember those high school statistics classes. Now you can put that memory to work. OK, maybe not. But try to remember what kind of questions were asked so we can sort out those trying to convince of that X caused Y, when in fact they don't have a clue what caused Y, other than they are trying to sell you X.
    The notion of inference statistics tells us how likely a given result occurred by chance alone. This doesn't actually tell us what the cause was - not by itself. But if we eliminate random results, then at least we can now go looking for the real cause.
    One way to start is the state the inverse of the cause and effect. This is called the Null Hypothesis. It is a proposal that there is no association between the cause and the effect. If the Null Hypothesis is TRUE, then there is no connection between cause and effect. Any study of cause and effect should conduct an experiment that gathers data used to test the Null Hypothesis. That is, we want a study that "rejects the Null Hypothesis."
    The measure of this null hypothesis is called the Level of Significance. It is the probability of incorrectly rejecting the Null Hypothesis. That is saying there is a difference between two groups of things (cause and effect) when in fact there is not. The Level of Significance is represented by a letter - p. p is the probability of a Type I error - we rejected the Null Hypothesis when should have not rejected the Null Hypothesis.
    By convention (convention of statistical analysis for general population of random occurrences), p < 0.5 is considered significant. That is there is a 1 in 20 chance that the finding from our experiment was due to chance alone. The lower p, the lower the chance the result is from chance.

Why Do We Care About This in Project Management?
    Because there are lots of people out there selling snake oil. Making claims about their "magic beans" and how those beans can improve your life, reduce cost, increase productivity, extend your life and grow hair where there was no hair before.
    When you encounter these people it is good to ask them a few questions:

1. What is the null hypothesis for the assertion you are making?
2. What is the confidence level for this assertion?

In other words

Can you show me there is a statistical connection between what you want me to do, buy, change, or otherwise do differently, and an actual tangible measurable outcome from this purchase or change?

If not, you still may have something worth looking at, but please stop speaking to me in term of cause and effect, because there may be no cause and effect - just a random (but desirable) occurrence of your effect.

More Statistics

The previous post mentioned several books that should be mandatory reading for anyone in the project success prediction business. I've acquired another book that is one of those "paradigm shifting" texts. Reasoning about Uncertainty, Joseph Y. Halpren, MIT Press, 2003.
    It's a book about how to think about the underlying processes of uncertainty. Since MANAGING IN THE PRESENCE OF UNCERTAINTY is the role of the project management and her team, this book speaks to the theory behind that approach to Project Management.
    As an aside, when people speak about the failure of the Waterfall  methods, Critical Path Method (CPM), PERT, GANTT and the other core processes of management projects  - they usually fail to consider that managing in the presence of uncertainty is the "norm" in mature project management organizations. In fact it is mandated by DID 81650 on government procurement projects. Anyone NOT considering the underlying probabilistic nature of a network of project activities is not actually "managing" the project, but simply observing the project pass them by and most likely go directly to the ditch.
    So why is Reasoning about Uncertainty interesting?

• It describes the underlying processes of "many worlds" - this is the many possibilities that may occur in a single collection of activities. The probabilistic nature of nature.
• Measuring the attributes of this probabilistic nature is simple at first. But then Mr. Bayes is introduced and the world get more complex. Much closer to reality.

    What struck me most was the concept that "simple minded" approaches to probability and statistics work for "simple minded" problems. The real world of course is not "simple minded." This book along with Terry Williams' Modelling Complex Projects, John Wiley & Sons are good starting points for programmatic risk analysis that is the foundation of large complex projects in construction, aerospace and enterprise systems.
    For example:

• "What is the completion confidence for flying to the International Space Station in 2014?"
• "What is the cost confidence in rolling out 54 SAP sites across the enterprise for under $235M?"
• "What is the probability that we won't need to have a second set of engineering development units for the light rail systems between Boulder and Denver?"

Now these all sound very esoteric for many in the code development business. But how about:

• What is the confidence that the features needs in the first release will earn their keep in the business 90 days after they are available to the order entry staff?
• What is the confidence that we'll spend less than $250,000 on the needed capabilities described in the Statement of Work for the sales order entry system?

The traditional agile approach is to push these questions to the customer, then use velocity or some other "driving in the rear view mirror" approach to inform the customer how much money has been spent and how many features made it into the current release. All interesting things to cost accountants. But not that useful for forecasting the credibility of the plan, compliance with the "not to exceed" budget and the probability of fulfilling the business case on the "go live" date.

Mean or Median

Seth Godin points out that 1/2 of every profession is above the mean and 1/2 are below.

This is only true if the statistical distribution of metric for the profession is symmetric. Then the Mean = Median.

The Median describes the "middle" of the distribution in value terms - 1/2 above, 1/2 below.

The Mean represents the total of all the values divided by the number of values. If skewed, the Mean can be far removed from Median - in either direction.

In the project management domain, this confusion is critical to understanding the probabilistic aspects of duration or cost. The Most Likely value used in PERT and Monte Carlo is the Mean - the value the occurs most often when the task duration or cost is observed a number of times.

Richard Rodger has a small error as well. The item marked Median "could" actually the Mode of the distribution, depending on its shape. The value that occurs most often. The Median is the "middle most" value.

Riding Fast

A recent Blogger described a ride to work on a mountain bike at 30MPH. As a moderately experienced cyclists (road not mountain) it got me thinking. Our weekend training rides in our club peloton of 10 to 15 provides a nice comfortable 22 MPH on the flats and a shield from the wind once I'm completely whipped on the way home.

I got to thinking about the 30MPH. Not knowing the details of the route, the rider or the weather, there is a relatively simple calculation using 30MPH, a 190 pound rider (me for example), no wind, and a flat riding surface with minimal friction gives a total wattage of around 317 watts needed to overcome the ~24 Newtons of drag, which translates into about 4.5 calories per minute of physical energy - at great way to loose weight. BTW Lance puts out around 400 to 450 watts while climbing a steep grade.

Speed is King

Just to get calibrated, Dave Zabriskie (team CSC) won the 2005 19KM Tour time trial with ~55 km/h, which is around 34 MPH. The "Hour" record was 49.7 km/h held by Chris Boardman. This is a simple race, ride on a closed course for one hour, measure your distance. It started in 1893. My Eddy Merckx Motobecane le Champion (long retired) was the signature machine for the previous record holder in Mexico in 1972.

Sosenka (Russia) broke Boardman's record in 2005 at 49.7000 (average speed) which is around 30.9MPH.

So my gut reaction was, this must have been a typo on the authors part, since I could assume he was not a Zabriskie grade cyclist commuting to work. But the more I think about it, it's a misunderstanding of what is really going on when riding a bike. The "feeling" of moving real fast is an illusion that becomes very disappointing when a speedometer is attached to the bike - properly calibrated for wheel size.

And The Point Here Is?

Numbers are very easy to toss around. Having calibrated, verifiable, repeatable numbers is a bit harder. Great quotes are encountered all the time...

• We dramatically improved our business by applying our process, you can as well
• We improved 155% over our previous efforts by simple changes in how we arranged our work
• You'll have dramatic improvements when you apply these principles to your project
• Such and such firm improved their throughput 340%

Where's the baseline? Where's the causal connection between the improvement and the process? Can this improvement be sustained over time? Can it be sustained in the absence of the Guru managing the project? Are these improvements in real bookable units of measure (money is a good one) that can be audited by an Independent Non-aligned Review (INAR is what it is called here in aerospace, and they are an axe to grind)?

All good questions I would suggest...but not many good answers. At least not in the form of equations and physical measurements verifiable by an INAR team.

But man I wish I could ride that fast. My fastest effort still has me stuck in the high teens and very very low 20's when sucking the wheel of the guy in front of me. Heck even the CU womens team passes us on the way home.