This presentation is showing up again. When it first appeared, I thought, the author needed to do some more research because most of the principles presented here are erroneous in principle. And the practices are not connected to any principle.
Now that's it's coming back, time to deconstruct the Principles one at a time ...
Principle #1 - Trust Your Process, or Change Your Process
This is simply good process development and business management. Can't go wrong here, except for one critical issue.
What is the Root Cause of the process not being trusted? Have you done a Root Cause Analysis? Or are you just taking observations and deciding that the process is broken without first finding the actual cause of the undesirable outcomes? And only then deciding what preventive or corrective actions should be take to remove to cause which in turn would remove the symptom.
If you haven't encountered the notion of Root Cause Analysis, start with the Apollo Root Cause Analysis method. No matter what method you use, you should never take any action to fix some problem - which is most likely a symptom of some deeper problem - without first determining the cause of the problem.
It's the basis of the #NoEstimates advocates to take the symptoms of a problem and not consider the cause. Then conjecture that NOT estimating will fix that symptom. This, of course, is simply poor process improvement and a fallacy, since without the root cause the symptom cannot be fixed and will return.
The notion of waterfall development on slide 9 as actually prohibited in our domain. This is called Doing Stupid Things on Purpose (DSTOP). Then conjecturing (here) that Not Estimating will somehow, through some unstated process of correction or prevention, fix the problem of when you are DSTOP. Not likely.
Principle #2 - Shorten the Feedback Cycle
Yes, this is the basis of any closed loop control system from keeping the room temperature for a specific setting, to the speed control on your car, to managing software development projects in the presence of uncertainty, using any software method you choose.
The sampling rate to maintain control must be matched with the processes change rate. So the first thing to do is answer the question...
How long are you willing to wait before you find out you are late?
The very technical answer to this question is the sample rate can be determined by a Nyquist sample rate analysis of the underlying non-stationary stochastic process of any typical software development project.
But the simplest way to answer the question of how long is
We assess physical percent complete at close business every single day, with the update of the TO DO field in Rally at the Task level.
At a minimum, at the end of every week, a Scrum team assess physical percent complete and updates their estimate to complete in some field in the Scrum development system. TO DO in Rally, Remaining Estimate in Jira, and other fields in Team Foundation Server and VersionOne. This can be done even with sticky notes on the wall.
The supposed evidence in the presentation, that estimating does not work starts with the Standish Chaos Report. That report has been long debunked for a variety of very good reasons. Seems the author missed those debunking articles and failed to understand the breathlessly poor statistical sampling techniques done by Standish.
Read here for more background ...
- The Myth of Standish Reports - Update Number 2
- Standish Report and Naive Statistics
- Standish Number
- Statistical Significance
Same debunking from the Gartner Survey - no Root Causes of those poor performance numbers, undefined sampling processes, not defined population statistics.
So when you read some numbers like those on slide 16, and there is no Root Cause for those late or failed examples, there can be NO way that the conjecture of NOT Estimating can be connected with any corrective or preventive action.
This is like reading about some miracle cure for a medical disorder, with no clinical evidence what you just paid for will do anything other than decreasing your bank account. But the #NoEstimates advocates have learned to spin the words in the same way the diet hucksters all the way to the cure for cancer hucksters have.
- Let me point out an obvious problem - software projects go over budget and are late
- Make an unsubstantiated claim that if you use my method, that problem will be removed - #NoEstimate means never being late
It works for diets and seems to work for unsuspecting developers looking to get out from under dysfunctional bad management and bad managers.
Then you're presented another slide on page 21 saying.
Of the large systems that are completed, about 66% experiencece schedule delays and cost overruns
Here's the actual article Project Management Tools and Software Failures and Successes, so read that last paragraph to see that cause. Then read the other Capers Jones books to learn how actual analysis of the problems with software cost and schedule are addressed outside the personal anecdotes. Then read the 157 papers Capers has written on this critically important topic of Sofware Development performance and the Root Cause analysis of the failure of software development.
So for this principle, we're back to making unsubstantiated claims, with no research, no evidence for the claim other than personal anecdotes.
Same for the Ambler surveys. Send out a survey, ask some questions about project failure rates, ask nothing about the root cause of those failures. Collect the responses. Total up the numbers.
This approach gets you a D in the statistical assessment class. With no answers to the question.
- What was the population the received the survey?
- What was the population that returned the survey?
- What is the population of projects in specific domains?
- Do the returned surveys have any correlation with the underlying population?
This is a sampling error of the first order and used in almost every example from the author, where he quotes some failure numbers.
Finally, the McKinsey report shows that 17% of large IT projects go so badly they threaten the existence of the company. This McKinsey project is ongoing, with 5,400 IT projects in the database as of 2012. But if you read the actual article from McKinsey they tell you how to fix the source of the problems, in "Delivering large-scale IT projects on time, on budget, and on value."
The author then claims he has more data coming. But before you accept any data ask if he's read anything on the development of sample surveys?
You can start with "Guide to Statistical Information Sampling for Surveys," and be ahead of not only this author but anyone making a claim about anything resulting from a survey.
And then the works in the bibliography of that paper as well as:
Dozens of other guides can be found on Google. There is NO excuse for this type of conjecture. Anyone reading this presentation or any other presentation containing statistics needs to ask the simple question - is this data credible? and if so what is the evidence of this credibility?
Principle #3 - Believe the Data, Not The Estimates
Estimates are data
This is one of those toss-off phrases that without an understanding of probability and statistics used to make a decision in the presence of uncertainty sounds good. But it's pure bunk.
But let's start with the opening slide from S.D. Conte. The phrase from McConnel's books is often repeated by #NoEstmates advocates without actually reading the original book. What is repeated by NE advocates is not actually what the book and the derived papers say.
Read "Estimating Accuracy Mathematics" to see why this author and many others have failed to understand the underlying mathematics of how credible estimates are assessed. The link to the quote in the original book can be found here, Software Engineering and Metrics. For $6.00 you can read what the author seemed to have missed. This is the underlying math behind the statement that is misused by the #NoEstimates advocates.
Estimates are based on data or on models that produce data. This data can be actual data from the past. But they can be data produced by a model in an attempt to model the future based on the past. Weather models, climate models, cost and schedule model, stock market models all work this way.
Monte Carlo Models. Method of Moments model. A parametric model. Here's a possible starting point for the formal principle of estimating.
This is likely not very useful on agile software development processes. But here's a reading list for some works that can be useful.
Here are some other resources for the agile domain
- Agile Cost Estimation
- Cost Estimation in Agile Software Development: Utilizing Functional Size Measurement Methods
- A Model for Estimating Agile Project Process and Schedule Acceleration
- Estimating Agile Software Development
And more when you google "agile software estimating"
Principle #4 - Use Alternatives to Estimate-Driven Decision Making
This principle opens with observations that have no root cause, but it is conjectured that Not Estimating will fix the problems that created the delays.
The state test for value is good. But the unanswered question starts with
What are the alternatives of an estimate-driven decision-making process, in the presence of uncertainty? The answer is - there is none if you are subject to the principle of microeconomics of software development.
Microeconomics is a branch of economics that studies the behavior of individuals and small impacting organizations in making decisions on the allocation of limited resources.
All engineering is founded on constrained optimization. How do we take the resources we've been given and deliver the best outcomes? That's what microeconomics is. Unlike models of mechanical engineering or classical physics, the models of microeconomics are never precise. They are probabilistic, driven by the underlying statistical processes of the two primary actors - suppliers and consumers.
This principle is immutable, no matter the software development method
If we look at the discipline of software engineering, we see that the microeconomics branch of economics deals more with the types of decisions we need to make as software engineers or managers.
Throughout the software life cycle, there are many decision situations involving limited resources in which software engineering economics techniques provide useful assistance.
Economic principles underlie the overall structure of the software lifecycle, and its primary refinements of prototyping, incremental development, and advancemanship.
The primary economic driver of the life-cycle structure is the significantly increasing cost of making a software change or fixing a software problem, as a function of the phase in which the change or fix is made.- Boehm, Barry W. "Software engineering economics." IEEE Transactions of Software Engineering, 1 (1984): 4-21.
As an aside, the author like to quotes books about Macroeconomics. Software development is Microeconomics.
- Economics is the study of how people make decisions in resource-limited situations.
- Macroeconomics is the study of how people and institutions make decisions in resource-limited situations on a national or global scale. It deals with the effect of decisions that national leaders make on such issues as tax rates, interest rates, foreign and trade policy. Macroeconomics also deals with markets - stock markets, bond markets, commodity markets.
- Microeconomics is the study of how people (not institutions) make decisions in resoruce-liited situtations on a personal scale. It deals with the decisions of individuals and organizations make on such issues as how much insutance to buy, what word processor software to buy, or what prices to charge for products and services.
If we look at the discipline of software development, we see that microeconomics deals more with the types of decisions we need to make as software developers or managers. When you hear the author or others quote macroeconomics texts, they've got the wrong paradigm.
Principle #5 - Test for Value First, the Test for Functionality
Testing for value is a business process, not a developer process. The core principle of Scrum is the Product Owner defines the priority of the work of the team, what the expected Value needs to be for the work - NOT the development team. The development teams role is to produce the software that meets the PO's needs. #NoEstimates seesm to willfully ignore the principles of Scrum.
The test for value needs to start with the strategy for why a Feature is needed. If the work is a project, then that answer comes from the plan for the outcomes of the project. Of the work is a product, it is most likely the Product Owner will be connected at the hip with the Product Manager and they together will have a product roadmap complete with release dates for the product to be used by a set of target customers, their needs, and the business model for how to recover the cost of developing any specific product feature. Along with the market analysis of what the feature is worth in terms of Price since price and cost are not the same.
For projects, there can be also a product roadmap and release plan, but every project I work has not just a set of minimum viable products, but a set of mandatory features needed for the project to go live. On the needed date, for the needed cost.
We are always testing that these features are ready for production.
But ...
These features that produce the Value had better Function as required
So it can't be one before the other, but both in parallel.
So seems to be the background on the sample company that bids for contracts with #NoEstimates. It may well be that the example firm that produces the Sitegeist mobile app didn't need to estimate. Maybe apps for IOS can be built without estimating the cost, duration, and features needed to go live.
But it turns out Sitegeist seems to have encountered troubles
Principle # 7 - Measure Progress only with Validated, Running Software
Yes, this is the foundation of Earned Value Management in any domain, including the agile software development domain, where EVM and Agile are the standard processes in our Software Intensive System of Systems domain is space and defense. Here's the current version of NDIA IPMD Guide for Agile on Earned Value Programs, which I am a contributor to.
The challenge is what are the units of measure for working software? The platitude of working software is too broad. Working for whom? Working under what conditions? Working at what cost? Working on what date?
This is one of those non-actionable principles that sounds good. But without units of measure, it is meaningless.
The picture on slide 37 is one of those meaningless charts as well. Yes, that actual completion date is later than the planned completion date. Why? Any Root Causes for that condition? As well when you see the term average you should immediately ask and what's the variance on that average? Without that variance that average number is worthless for any decision making. Please read The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty.
So why did all those projects come in late? The killer question is then ...
Will NOT Estimating have resulting in those projects coming in on time?
What would be the mechanism of NOT Estimating that would enable those projects to be successful? How could that be, when operating in the presence of uncertainty?
Page 40 is another example of how to lie with statistics. The average of the stories is 2.76 the variance in that average is 1.03. That's a 37% variance. Do you want to bet that your project will be OK with a 37% variance in the outcome? The answer starts with the value at risk. For small projects may be so, for non-trivial projects probably not. The subtitle to The Flaw of Averages is
A must read for anyone who makes business decisons that have a major financial impact
The author seems to have not read this book and the many others providing advice on how to make decisions in the presence of uncertainty while spending other people's money.
Page 41 is a good example of Doing Stupid Things on Purpose except for number 4, which is a core principle of agile.
Principle #8 - The System where you work has Predictable Outputs
The claim #NoEstimates delivers! has no testable evidence. It's a personal anecdote.
The whole argument between Story Points and Stories is a house built on sand unless you've assessed the correlation coefficient factor - p-value - between the Stories and Story Points.
A correlation coefficient is a number between –1 and 1 that determines whether two paired sets of data. The closer to 1 the more ‘confident’ we are of a positive linear correlation and the closer to –1 the more confident we are of a negative linear correlation (which happens when, for example, one set of numbers tends to decrease when the other set increases as you might expect if you plotted a person’s age against the number of toys they possess).
When the correlation coefficient is close to zero there is no evidence of any relationship.
Confidence in a relationship is formally determined not just by the correlation coefficient but also by the number of pairs in the data set. If there are very few pairs then the coefficient needs to be very close to 1 or –1 for it to be deemed ‘statistically significant’, but if there are many pairs then a coefficient closer to 0 can still be considered ‘highly significant.'
The standard method that statisticians use to measure the significance of their empirical analyses is the p-value. Suppose we are trying to determine if the relationship between height and intelligence of people is significant; then we start with the null hypothesis which, in this case, is the statement height and intelligence of people are unrelated. A p-value is a number between 0 and 1 representing the probability that this data would have arisen if the null hypothesis were true. In medical trials, the null hypothesis is typically of the form that the use of drug X to treat disease Y is no better than not using any drug. The #NoEstimates advocates have no null hypothesis to test. They just state an unsubstantiated conjecture and want to agree with it without questions the logic behind it.
Without actual data to assess, no credible claim can be made.
All Page 55 through 59 tell you is there a high correlation between Story Points and Stories. What do those charts look like when there is not that correlation?
Principle #9 - Use Methods with a Proven Track Record
So where is the proof that Not Estimates has a track record of success. All the examples are personal anecdotes in unstated domains. Much of the data published by the author and other #NoEstimates has very wide statistical variances. Sometimes up to ±40% from the mean. Do you want to bet on the success of your project using a method whose examples are essentially flipping a coin?
The End
This presentation, which is held up as the example of how to run a project with #NoEstimates, violates several established principles of project management, business management, finance management, and probabilistic decision making.
There is no principle of Managerial Finance, Probabilistic Decision‒Making, or Microeconomics of software development in presence of reducible and irreducible uncertainty, by which a credible decision can be made when using scarce resources, while spending other people’s money, without estimating the outcome of that decision and its impact on the probability of success of the project.
