The current issue of ICEEAWorld, has an article on estimating on agile projects.
To build a credible estimate for any project, in any domain, to produce a solution to any problem, we need to start with a few core ideas.
If you're missing any of the items in this picture, it's going to be a disappointing effort. Some may even call it a waste to estimate. But not for the reasons you think. It is a waste to estimate if you don't know how to estimate. But estimate are not for you, unless you're the one providing the money. They're for those providing the money, expecting the outcomes from that expense show show up on some need date, with the needed value that provide them with the ability to earn back the money.
Rally Software is a local firm providing tools for the management of agile project. Project Managers provide the glue for all human endeavors involving complex work processes. Rally has those tools as do many others. Rally also has message that needs to be addressed by the project management community. Organizing, planning, executing social projects is one of the roles projects managers can contribute to.
Mike Cohn of Mountain Goat Software has a collection of 101 Agile Quotes.
There are few I have heart burn with, but the vast majority are right on.
Some of my favorite:
When we read on a blog post that estimates are not meaningful unless you are doing very trivial work, † I wonder if the poster has worked on any non-trivial software domain. Places like GPS OCX, SAP consolidation, Manned Space Flight Avionics, or maybe Health Insurance Provider Networks. Because without some hands on experience in those non-trivial domains, it's be hard to actually knowing what you're talking about when it comes to estimating the spend of other peoples money.
Maybe some background on estimates for nontrivial work will shed light on this ill informed notion that only trivial projects can be estimated.
These are a small sample of papers from one journal on software estimating for misison critical, some times National Asset projects.
Go to Cross Talk, The Journal of Defense Software Engineering, and search for "estimating" to get 10 pages of 10 articles on this topic alone. This notion of estimating in non-trivial domains is well developed, well documented, and many examples of tools, processes, and principles.
If Do Your Homework and the Test is much easier.
It could be that the original poster has little experience in mission critical, national asset, enterprise class, software intensive systems. Or it could be the poster simply doesn't know what making estimates for project that spends other peoples money, many times significant amounts of money, is all about.
And of course most of the problems describes as the basis for Not Estimating - the illogical notion that if we can't do something well, let's stop doing it - starts with not knowing what Done Looks Like in any units of measure meaningful to the decision makers.
So start here with my favorite enterprise architect blog amd his list of books when you follow the link to the bottom.
So when you have some sense of what DONE looks like in terms of capabilities, the estimating process is now on solid ground. From that solid ground you can ask have we done any like this before? Or better yet can we f ind someone who has done something like this before? Or maybe can we look around to see what looks like our problem and figure out how long it took them by simply asking them? I
If the answer to any of those questions is NO and you're NOT working in a research and development domain, then don't start the project because you're not qualified to do the work, you don't know what you're doing and you're going to waste your customers money.
† Scroll to the bottom of http://zuill.us/WoodyZuill/category/estimating/ and search for "A Thing I Can Estimate," to see the phrase, and remember the questions and the answers above. If you're not answering those in some positive way, you're now on a death march project starting day one, because you don't know what done looks like for the needed capabilities. Not the requirements, not the code, not the testing - that's all straight forward. Without some notion of what the system is supposed to do, you're never recognize it if it were ever to come into view. And since the customer doesn't know as well, all the money they're spending to find out has to be written off as IRAD or flushed down the toliet as a waste of time and effort in the end. And then you'll know why Standish (improperly) reports projects fail.
In a recent email exchange, the paper by Todd Little showing projects that exceeded their estimates was used as an example for how porrly we estimate, and ultimately one of the reasons to adopt the #NoEstimates paradigm of making decisions in the absence of estimates of cost, schedule, and the probability that the needed capabilities will show up on time and be what the customer wanted.
Sherlock here had it right. This picture by the way is borrowed from Mike Cohn's eBook of 101 quotes for agile.
I've written about Little's paper before, but it's worth repeating.
It's very sporty to use examples of bad mathematics, bad management, bad processes, bad practices as the basis for something new. This is essential the basis the book Hard Facts, dangerous Half-Truths: Profiting from Evidence-Based Management. When we start talking about something new and disruptive in the absence of the data, facts, root causes, and underlying governance and principles, we're treading on very thin ground in terms of credibility.
Here's the core principle of all software development
Customers exchange money for value. The definition of value needs to be in units of measure that allows them to make decisions about the future value of that value. That value is exchanged for a cost. A future cost as well.
Both this future cost and future value are probabilistic in nature, due to the uncertainties in the work processes, technologies, markets, productivity and all the ...ilities associated with project work. In the presence of uncertainty, nothing is for certain - a tautology. There are two type of uncertainty - reducible and irreducible. Reducibe we can do something about. We can spend money to reduce the risk associated with the uncertainty. Irreducible, we can't. We can only have margin, management reserve, or a Plan B.
To make decisions in the presence of these uncertainties - reducible and irreducible - we need to estimate the uncertainty, the cost of handling the uncertainty, and the value produced by the work driven by these uncertainties. When we fail to make these estimates, the uncertainties don't go away. When we slice the work into small chunks, we might also slice the uncertainties into small chunks - this is the basis of agile and the paradigm of Little Bets. But the uncertainties are still there, unless we've explicitly bought them down or installed margin and reserve. They didn't go away. And what you don't know - or choose to explicitly ignore - can hurt you.
So Sherlock is right don't put forth a theory without the data.
On a twitter discussions and email exchanges there is a notion of populist books versus technical books used to address issues and problems encountered in our project management domains. My recent book Performance-Based Project Management® is a populist book. There are principles, practices, and processes in the book that can be put to use on real projects, but very few equations and numbers. It's mostly narrative about increasing the probability of project success. But the to calculate that probability based on other numbers, processes, and systems is not there. That's the realm of Technical books and journal papers.
The content of the book was developed with the help of editors at American Management Association, the publisher. The Acquisition Editor contacted me about writing a book for the customers of AMA. He explained up front AMA is in the money making business of selling books. And that although I may have many good ideas, even ideas that people might want to read about, it's an AMA book and I'll be getting lots of help developing those ideas into a book that will make money for AMA.
The distinction between a populist book and a technical book are the differences between a book that addresses a broad audience with a general approach to the topic and a deep dive book focused on a narrow audience.
But one other disticntion is for most of the technical approaches, some form of calculation takes place to support the materials found in the populist material. One simple example is estimating. There are estimating articles and some books that lay out the principles of estimates. We have those in our domain in the form of guidelines and a few texts. But to calculate the Estimate To Complete in a statistically sound manner, technical knowledge and the underlying mathematics of non-linear, non-stationary, stochastic processes (Monte Carlo Simulation of the projects work structure) is needed.
Two examples of populist versus technical
Two from my past two from my current work.
These two books are about the same topic. General relativity and its description of the shape of our universe. One is a best selling popularization of the topic, found in many home libraries of those interested in this fascinating topic. The one on the left is on my shelf from a graduate school course on General Relativity along with Misner, Thorne, and Wheeler's Gravity.
Dense is an understatement for the math and the results of the book on the left. So if you want to calculate something about a rapidly spinning Black Hole, you're going to need that book. The book on the right will talk about those Black Holes in non-mathematical terms, but no numbers come out from that description.
The book on the left is about probabilistic processes in everyday life that we misunderstand or are biased to misunderstand. The many cognitive biases we use to convince ourselves we are making the right decisions on projects are illustrated through nice charts and graphs.
We use the book on the left in our work with non-stationary stochastic process of complex project cost and schedule modeling. Making these decisions is critical to quantifying how technical and economics risk may affect a system's cost. This book is a treatment of how probability methods are applied to model, measure, and manage risk, schedule, and cost engineering for advanced systems. Garvey's shows how to construct models, do the calculations, and make decisions with these calculations.
Here's The Point - Finally
If you come across a suggestion that decisions can be made in the absence of knowing anything about the future numbers or about actually doing the math, put that suggestion in the class of populist descriptions of a complex topic.
If you can't calculate something, then you can't make a decision based on the evidence represented by numbers. If you can't decide based on the math, then the only way left is to decide on intuition, hunchs, opinion, or some other seriously flawed non-analytical basis.
Just a reminder from Mr. Deming stated in yesterday's post
If it's not your money, there's likley an expectation that those providing the money are intestered in the calculations needed to make those decisions.
All ideas require credible evidence to be tested, suspect ideas require that even more so - Deep Inelastic Scattering, thesis adviser, University of California, 1978
When the response to questions about the applicability of an idea is push back with accusations that those asking the questions in an attempt to determine the applicability and truth of the statement are somehow afraid of that truth, suggests there is little evidence as a test of those conjectures.
When there are proposals that ignore the principles of business, microeconomics, control systems theory, and are based on well know bad management practices, with well know and easy to apply corrective actions - there is no there, there.
So without a testable process, in a testable domain, with evidence based assessment of appliability, outcomes, and benefits, any suggestion is opinion at best and blather at worse.
In a sufficiently complex project we need measures of progress to plan beyond burning down our list of same sized stories, which by the way require non-trivial work to make same sized and keep same sized. And of course if this same size-ness does not have a sufficiently small variance all that effort is a waste.
But let's assume we're not working on a sufficiently small project where same sized work efforts can be developed, we need measures of progress related to the Effectiveness of the deliverables and the Performance of those deliverable in producing that effectiveness for the customer.
Here's a recent webinar on this topic.
Much has been written about the Estimating Problem, the optimism bias, the planning fallacy, and other related issues with estimating in the presence of Dilbert-isk style management. The notion that the solution to the estimating problem is not to estimate, but to start work, measure the performance of the work, and use that to forecast completion dates and efforts is essentially falling into the trap Steve Martin did in LA Story.
Using yesterday's weather becasue he was too lazy to make tomorrow's forecast
By the way each of those issues has a direct and applicable solution. So next time you hear someone use them as the basis of a new idea, ask if they have tried the known to work solution to the planning fallacy, estimating bias, optimism bias, and the myriad of other project issues with knowing solutions?
All measuring performance to date does is measure yesterday's weather. This yesterday's weather paradigm has been well studied. If in fact your project is based on Climate then yesterday's weather is likely a good indicator of tomorrow's weather.
The problem of course with the yesterday's weather approach, is the same problem Steve Martin had in LA Story when he used a previously recorded weather forecast for the next day.
Today's weather turned out not to be like yesterday's weather.
Those posting that stories settle down to a rhythm assume - and we know what assume means - that the variances in the work efforts are settling down as well. That would mean the word assume comes true Ass out of U and Me. That's a hugely naive approach, without actual confirmation that the variances are small enough to not impact the past performance. When you have statistical processes looking like this, from small sampled projects in the absence of actual reference class - in this case self-reference class - you're also being hugely naive about the possible behaviours of stochastic processes.
Then when you slice the work to same sized efforts - this is actually process used in the domains we work: DOD, DOE, ERP - you're actually estimating future performance base on a reference class and calling it Not Estimating.
So when you hear examples and Bad Management, over commitment of work, assigning a project manager to a project that is 100's of time larger than that PM has ever experienced and expecting success, getting a credible estimate and cutting it in half, or any other Dilbert style management process - and you start with dropping the core process needed to increase the probability of success.
This approach is itself contrary to good project management principles, which are quite simple:
If we start with a solution to a problem of Bad Management, before assuring that the Principles and Practices of Good Management are in place, we'll be paving the cow path as we say in our enterprise, space, defense domain. This means that the solution will have not actually fixed the problem. It will have not treated the root cause of the problem, just addressed the symptoms.
There is no substitute for Good Management.
And when you hear there is a smell of bad management and there is no enumeration of the root causes and the corrective actions to those root causes, remember Ingio Montoya's retort to Vizzini's statement
You keep using that word. I do not think it means what you think it means.
That word is dysfunction, smell, root cause - all of which are missing the actual innumerated root causes, assessment of the possible corrective actions, and resulting removal of the symptoms.
I speak about this approach from my hands on experience working the Performance Assessment and Root Cause Analysis on programs that are in the headlines.
With the plethora of opinions on estimating - some informed, many uninformed - here's my list of books and papers that inform our software estimating activities for Software Intensive Systems. These books range for hard core engineering to populist texts
is not actually true after you have read the book. So please read the book and see how McConnell provides step-by-step actions for producing credible estimates.
Estimating software development starts with understanding what the software system is supposed to be doing and how we're able to measure that. This process is based on defining the needed capabilities, the measures of Effectiveness, Measures of Performance, Key Performance Parameters, and Technical Performance Measures all needed for the ultimate success of the project. Along with a Plan showing the increasing maturity of the delivered capabilities. If we don't these in some form, it's going to be a disappoiintment for those payinig for our efforts when they get to the end and the outcomes are what they were expecting.
Capabilities are not Requirements. Requirements implement Capabilities. Capabilities are pretty much fixed while the Requirements evolve. Capabilities Based Planning is the basis of project management in many Software Intensive Systems.
With the project's capabilities defined to a level needed to start the project - failing to do this results in a Death March at worse, and spending the customer's money to discover what should have been discovered before starting. With the capabilities, the project needs to be managed in a way that will increase the probability of success.
So when you hear of some new approach to project management, ask if there is any connection to a domain and a context in that domain. Because there any many ideas about how to improve the probability of project success. But without a domain and context it'll be hard to assess if they are applicable to your specific situation. Here's one way to think about this domain dependency. From solo projects to national assets, methods, processes, tools are different as is the value at risk.
In the mathematics of physics, there are two essential types of values in all calculations - Scalars and Vectors.
Scalars are isolated values, with no outside context. Indeed They remain the same regardless of any context. A common example would be mass. An object has a mass of 1 Kilogram no matter where it is, or how much physical space it occupies. The context of the object cannot change the scalar value of its mass.
Vectors are contexted values, and can change depending on that context. An object has weight, dependent on both the mass value and gravity context of the object. An object with high mass, may still have no weight in the corresponding context of gravity.
But most specifically, vector values allow the calculations of change over time. Numbers (scalars) without context (vectors) are not metrics.
It is meaningless to say that cost to operate the IT Service Desk has doubled within the last ten years, without also showing how the number of employees has tripled in the same time.
It is meaningless to say a self-selected 120 projects exceeded their estimated cost and duration without an assessment of the credibility of that original estimate and the determination of the Root Cause of that overage.
It is meaningless to say the end date and cost can be forecast with saying something about the underlying uncertainties in effort size, risk, inter-dependencies, changing requirements, defect rates, labor absorption rates, integration issues, performance issues, and complexities of emerging behaviours once the system starts to come together and applied to fulfill the needed capabilities.
When we hear about small sample sized forecasts of same sized work activities, or selecting the next priority item to work on without considering the inter-dependencies of past work items or future work items - we're speaking about Scalar numbers, not Vector numbers.
Vectors state magnitude and direction. Open Loop control only states magnitude. Use it at your own risk.
Much of the noise around agile is based on platitudes and words in the absence of a domain and a context in that domain. Here's a possible anchoring processes
Speaking at the Integrated Program Management Conference in Bethesda MD this week. The keynote speaker Monday was Katrina McFarland, Assistant Secretary of Defense (Acquisition)(ASD(A)), the principal adviser to the Secretary of Defense and Under Secretary of Defense for Acquisition.
During her talk she spoke of the role of Earned Value Management. Here's a mashup of her remarks...
EV is a thoughtful tool as the basis of a conversation for determining the value (BCWP) produced by the investment (BCWS). This conversation is an assessment of the efficacy of our budget.
We can determine the effecacy of our budget through:
- Measures of Effectiveness of the deliverables in accomplishing the mission or fulfilling the technical and operational needs of the business.
- Measures of Performance of these deliverables to perform the needed functions to produce the needed effectiveness
- Technical Performance Measures of these functions to perform at the needed technical level.
These measures answer the question of what is the efficacy of our budget in delivering the outcomes of our project.
The value of the project outcomes must be traceable to a strategy for the business or mission. Once this strategy has been identified, the Measures of Effectiveness, Performance, and Technical Performance Measures can be assigned to the elements of project. These are shown in the figure below
This approach is scalable up and down the complexity of projects based on five immutable principles of project success.
Without credible answers to each of these questions, the project is on the path to failure on day one.
The most unsuccessful three years in the education of cost estimators appears to be fifth-grade artihmetic - Norman R. Augustine
Augustine is former Chairman and CEO of Martin Marietta. His seminal book Augustine's Laws, describes the complexities and conundrums of today's business management and offers solutions. Anyone interested in learning how successful management of complex technology based firms is done, should read that book. As well, read McConnell's book and see if you can find where
Because I sure can't find that proof or any mention that estimates don't work, other than for those who failed to pay attention in the 5th grade arithmetic class.
Hugh McCleod's art for Zappo's provides the foundation for trust in that environment
If I'm the head of HR, I'm responsible for filling the desks at my company with amazing employees. I can hold people to all the right standards. But ultimately I can't control what they do. This is why hiring for culture works. What Zappos does is radical because it trusts. It says "Go do the best job you can do for the customer, without policy". And leaves employees to come up with human solutions. Something it turns out they're quite good at, if given the chance.
Now let's take another domain, one I'm very famailar with - fault tolerant process control systems. Software and support hardware applied to emergency shutdown of exothermic chemical reactors - those that make the unleaded gasoline for our cars, nuclear reactors and conventional fired power generation, gas turbine controls, and other must work properly machines. And a similar domain of DO-178c flight control systems, which must equally work without fail and provide all the needed capabilities on day one.
At Zappos, the HR Diector describes a work environment where employess are free to do the best job they can for the customer. In the domains above, employees also work to do the best job for the customer they can, but flight safety, live safety, equipment safety are also part of that best job. In other domains we work, doing the best job for the customer means processing with extremely low error rates, transactions for 100's of millions of dollars of value in the enterprise IT paradigm. Medical insurance provider services, HHS enrollment, enterprise IT in a variety of domains.
Zappo's can recover from an error, other domains can't. Nonrecoverable errors mean serious loss of revenue, or even loss of live. In the other domains, failure is similar consequences. I come from those domains, they inform my view of the software development world - where software fail safe and fault tolerance is the basis of business success.