Phillip Armour has a classic article in CACM titled "Ten Unmyths of Project Estimation," Communications of the ACM (CACM), November 2002, Vol 45, No 11. Several of these Unmyths are applicable to the current #NoEstimates concept. Much of the misinformation about how estimating is the smell of dysfunction can be traced to these unmyths.
Mythology is not a lie ... it is metaphorical. It has been well said that mythology is the penultimate truth - Joseph Campbell, The Power of Myth
Using Campbell's quote, myths are not untrue. They are an essential truth, but wrapped in anecdotes that are not literally true. In our software development domain a myth is a truth that seems to be untrue. This is Armour's origin of the unmyth.
The unmyth is something that seems to be true but is actually false.
Let's look at the three core conjectures of the #Noestimates paradigm:
- Estimates cannot be accurate - we cannot get an accurate estimate of cost, schedule, or probability that the result will work.
- We can't say when we'll be done or how much it will cost.
- All estimates are commitments - making estimates makes us committed to the number that results from the estimate.
The Accuracy Myth
Estimates are not numeric values. they are probability distributions. If the Probability Distribution below represents the probability of the duration of a project, there is a finite minim - some time where the project cannot be completed in less time.
There is the highest probability, or the Most Likely duration for the project. This is the Mode of the distribution. There is a mid point in the distribution, the Median. This is the value between the highest and the lowest possible completion times. Then there is the Mean of the distribution. This is the average of all the possible completion times. And of course The Flaw of Averages is in effect for any decisions being made on this average value †
“It is moronic to predict without first establishing an error rate for a prediction and keeping track of one’s past record of accuracy” — Nassim Nicholas Taleb, Fooled By Randomness
If we want to answer the question What is the probability of completing ON OR BEFORE a specific date, we can look at the Cumulative Distribution Function (CDF) of the Probability Distribution Function (PDF). In the chart below the PDF has the earliest finish in mid-September 2014 and the latest finish early November 2014.
The 50% probability is 23 September 2014. In most of our work, we seek an 80% confidence level of completing ON OR BEFORE the need date.
The project then MUST have schedule, cost, and technical margin to protect that probabilistic date.
How much margin is another topic.
But projects without margin are late, over budget, and likely don't work on day one. Can't be complaining about poor project performance if you don't have margin, risk management, and a plan for managing both as well as the technical processes.
So where do these charts come from? They come from a simulation of the work. The order and dependencies of the work. And the underlying statistical nature of the work elements.
- No individual work element is deterministic.
- Each work element has some type of dependency on the previous work element and the following work element.
- Even if all the work elements are Independent and sitting in a Kanban queue, unless we have unlimited servers of that queue, being late on the current piece of work will delay the following work.
So what we need is not Accurate estimates, we need Useful estimates. The usefulness of the estimate is the degree to which it helps make optimal business decisions. The process of estimating is Buying Information. The Value of the estimates, like all value is determined by the cost to obtain that information. The value of the estimate of the opportunity cost, which is the different between the business decision made with the estimate and the business decision made without the estimate. ‡
Anyone suggesting that simple serial work streams can accurately forecast - an estimate of the completion time - MUST read Forecasting and Simulating Software Development Projects: Effective Modeling of Kanban & Scrum Projects using Monte Carlo Simulation, Troy Magennis.
In this book are the answers to all the questions those in the #NoEstimates camp say can't be answered.
The Accuracy Answer
- All work is probabilistic.
- Discover the Probability Distribution Functions for the work.
- If you don't know the PDF, make one up - we use -5% + 15% for everything until we know better.
- If you don't know the PDF, go look in databases of past work for your domain. Here's some databases:
- http://www.nesma.org/
- http://www.isbsg.org/
- http://www.cosmicon.com/
- If you still don't know, go find someone who does, don't guess.
- With this framework - it's called Reference Class Forecasting - that is making estimate about your project from reference classes of other projects, you can start making useful estimates.
But remember, making estimates is how you make business decisions with opportunity costs. Those opportunity costs are the basis of Microeconomics and Managerial Finance.
Cone of Uncertainty and Accuracy of Estimating
There is a popular myth that the Cone of Uncertainty prevents us from making accurate estimates. We now know we need useful estimates, but those are not prevented by in the cone of uncertainty. Here's the guidance we use on our Software Intensive Systems projects.
Finally in the estimate accuracy discussion comes the cost estimate. The chart below shows how cost is driven by the probabilistic elements of the project. Which brings us back to the fundamental principle that all project work is probabilistic. Modeling the cost, schedule, and probability of technical success is mandatory in any non-trivial project. By non-trivial I mean a de minimis project, one that if we're off by a lot it doesn't really matter to those paying.
The Commitment Unmyth
So now to the big bug a boo of #NoEstimates. Estimates are evil, because they are taken as commitments by management. They're taken as commitment by Bad Management, uninformed management., management that was asleep in the High School Probability and Statistics class, management that claims to have a Business degree, but never took the Business Statistics class.
So let's clear something up,
Commitment is how Business Works
Here's an example taken directly from ‡
Estimation is a technical activity of assembling technical information about a specific situation to create hypothetical scenarios that (we hope) support a business decision. Making a commitment based on these scenarios is a business function.
The Technical “Estimation” decisions include:
- When does our flight leave?
- How do we get there? Car? Bus?
- What route do we take?
- What time of day and traffic conditions?
- How busy is the airport, how long are the lines?
- What is the weather like?
- Are there flight delays?
This kind of information allows us to calculate the amount of time we should allow to get there.
The Business “Commitment” and Risk decisions include:
- What are the benefits in catching the flight on time?
- What are the consequences of missing the plane?
- What is the cost of leaving early?
These are the business consequences that determine how much risk we can afford to take.
Along with these of course is the risk associated with the uncertainty in the decisions. So estimating is also Risk Management and Risk Management is management in the presence of uncertainty. And the now familiar presentation from this blog.
Wrap Up
Risk Management is how Adults manage projects - Tim Lister. Risk management is managing in the presence of uncertainty. All project work is probabilistic and creates uncertainty. Making decisions in the presence of uncertainty requires - mandates actually - making estimates (otherwise you're guess your pulling numbers from the rectal database). So if we're going to have an Adult conversation about managing in the presence of uncertainty, it's going to be around estimating. Making estimates. improving estimates, making estimates valuable to the decision makers.
Estimates are how business works - exploring for alternatives means willfully ignoring the needs of business. Proceed at your own risk
† This average notion is common in the No estimates community. Take all the past stories or story points and find the average value and use that for the future values. That is a serious error in statistical thinking, since without the variance being acceptable, that average can be wildly off form the actual future outcomes of the project
‡ Unmythology and the Science of Estimation, Corvus International, Inc., Chicago Software Process Improvement Network, C-Spin, October 23, 2013
