The numbers that appear in projects — cost, schedule, performance — are all random variables drawn from an underlying statistical process. This process is officially called a non-stationary stochastic process. It has several important behaviours that create problems for those trying to make decisions in the absence of understanding how these processes work in practice.
The first issue is that all point estimates for projects are wrong, in the absence of a confidence interval and an error band on that confidence.
How long will this project take is a common question asked by those paying for the project. The technically correct answer is there is an 80% confidence of completing on or before some date, with a 10% error on that confidence. This is a cumulative probability number collecting all the possible completion dates and describing the cumulative probability - the 80% - of an on or before, since the project can complete before that final probabilistic date as well.
Same conversation for cost. The cost of the project will be at or below "some amount" with a 80% confidence.
The performance of products or services are the third random variables. By technical performance it means anything and everything that is not cost or schedule. This is the wrapper term for the old concept of scope. But in modern terms there are two general purpose categories of Performance with one set of parameters.
- Measures of Effectiveness - are the operational measures of success that are closely related to the achievements of the mission or operational objectives evaluated in the operational environment, under a specific set of conditions. The Measures of Effectiveness:
- Are stated in units meaningful to the buyer,
- Focus on capabilities independent of any technical implementation,
- Are connected to the mission success.
- Measures of Performance - are the measures that characterize physical or functional attributes relating to the system operation, measured or estimated under specific conditions. The Measures of Performance are:
-
Attributes that assure the system has the capability and capacity to perform,
-
Assessment of the system to assure it meets design requirements to satisfy the MoE.
- Key Performance Parameters - represent the capabilities and characteristics so significant that failure to meet them can be cause for reevaluation, reassessing, or termination of the program. Key Performance Parameters:
-
Have a threshold or objective value,
-
Characterize the major drivers of performance,
-
Are considered Critical to Customer (CTC).
These measures are all random numbers with confidence intervals and error bands.
So What's The Point?
When we hear you can't forecast the future, that's not true. The person saying that didn't pay attention in the High School statistics class. You can forecast the future. You can make estimates of anything. The answers you get may not be useful, but it's an estimate all the same. If it is unclear on how to do this, here's a reading assignment for the books we use nearly every month to make our estimates at completion and estimates to complete for software intensive project, starting with the simplist:
- How Many Licks, Or How To Estimate Damn Near Anything, Aaron Santos - this straightforward book shows how to start estimating with confidence.
- How to Think About Statistics, 5th Edition - a survey of how good statistical thinking is needed to make decisions. And it is decision making we're after.
- How to Lie with Statistics, Darrell Huff - this is a mandatory book for anyone wanting to manage a project. The famous quote "Lies, Damn Lies, and Statistics," is an example of how to lie with statistics.
- How to Measure Anything, Douglas Hubbard - shows how to do just that, with a bit more mathematics than the first 3 books.
- Introduction to Stochastic Models, 2nd Edition - starts down the path of real mathematics used to model cost, schedule, and performance of complex systems. And Dr. Goodman's Statitics page
While on the topic of books, here are some books that should be on your shelf that put those probability and statistics to work.
- Facts and Fallacies of Software Engineering, Robert Glass - speaks to the common fallacies in software development. The most common is we can't possibly estimate when we'll be done or how much it will cost. Read the book and start calling BS on anyone using that excuse to not do their homework. And a nice update by Jack Atwood, founder of Stack Exchange.
- Estimating Software-Intensive Systems, Richard Stutzle - this is the book that started the revolution of statistical modeling of software projects. When you hear oh this is so olde school, that person didn't take the HS Stats class either.
- Software Engineering Economics, Barry Boehm - is how to pull all this together. And when you hear this concept is olde school, you'll know better as well.
There are several tools that make use these principles and practices:
Here's the End
- Learn to estimate.
- Teach others to estimate.
- When the Dilbert boss comes around, you'll have to tools to have a credible discussion about the Estimate to Complete number he's looking for is bogus. He may not listen or even understand, but you will.
And that's a start in fixing the dysfunction of bad estimating when writing software for money. Start with the person who can actually make a change - You
