One's originality comes in your inability to emulate your influencers - Dale Watson
Principles, Practices, and Processes to Increase Probability of Success
Great equations change the way we perceive the world. They reorchestrate the world - transforming and reintegrating our perception by redefining what belongs together with what. Light and waves. Energy and mass. probability and position. And they do so in a way that often seems unexpected and even strange.
- Robert P. Crease, The Greatest Equations Ever, Physics Web 2004, in The Mathematics Devotional: Celebrating the Wisdom and Beauty of Mathematics, Clifford A. Pickover
The impression that "our problems are different," is a common disease that afflicts management to work over. They are different, to be sure, but the principles that will help improve the quality of product and service are universal in nature - W. Edwards Deming
So when we hear some new and controversial conjecture is the solution to the smell of dysfunction, ask for tangible, testable, verifiable evidence that this new approach is not a solution looking for a problem to solve.
Any effectively generated theory capable of expressing elementary arithmetic cannot be both consistent and complete. On particular, for any consistent, effectively generated formal theory that proves certain basic arithmetic truths, there is an arithmetical statement that is true, but not provable in the theory
Godel, First Incompleteness Theorm
Complexity and chaos are not the same. Complexity requires a higher degree of order and works against chaos. To construct a complex system, work is required - resisting or decreasing entropy. Chaos increases entropy and is a natural process of the universe. - Alexander B. Alleman, PhD student, Montana State University
Managing projects in the presence of uncertainty requires energy be put into the system to maintain it's equilibrium and stability. How much energy, for how long, in what order, in what type is the role of management. To know the answers to these question of how much, when, where means make estimates of those quantities and the resulting outcomes that reduce chaos.
In other domains, this principle is called the 2nd Law of Thermodynamics. In project work, this principle is also applicable, since the system of work is a dynamic coupled collection of random processes, interacting with each other in non-linear, non-stationary ways.
The Law of Entropy is expressed in this 2nd law. The law predicts that the natural state of all things—from the tiniest atoms to the largest of galaxies—is that of disorder. This means, without appropriate systems or balances in place, everything wants to fall into chaos. The management of this naturally occurring (statistical) - as well as the probabilistic - set of processes is the role of Risk Management. If there no uncertainties on your project the 2nd Law would not be applicable. Since all project work is uncertain there is ALWAY risk associated with those uncertainties. And of course managing in the presence of these uncertainties Mandates making estimates about the impact of managerial actions on the outcomes of those decisions.
A nice background on the 2nd Law of Thermodynamics of Project work comes from Shim Marom. This topic is little understood in some SW and IT domains. In Engineering and Construction and Software Intensive Systems is it a well known process.
The entire Universe and everything in it is a collection of 2nd order non-linear partial differential equations, all obeying the 2nd Law of Thermodynamics.
General notions are generally wrong - Letter to Mr. Wortley Montegu, 28th March 1710
This reminds me of many suggestions that new and supposed innovative ideas can be applied outside the personal anecdotal experience of the person making the suggestion - conjecture actually.
It is a capital mistake to theorize before one has data - Sir Arthur Conan Doyle
When we hear about a new idea that will undo all the established principles and practices . Ask first you got any data to show your cocka mammy idea has any hope of being correct?
Taking Three as the subject to reason about - A convenient number to state - we add Seven, and Ten, and multiply out buy One Thousand diminished by Eight. The result we proceed to divide, as you see, by Nine Hundred and Ninety Two: then subtract Seventeen, and the answer be Exactly and perfectly True - The Hunting of the Snark
Lewis Carol denied knowing the meaning of his poem about the hunting of a mythical creature. The poem belongs to a family of narratives called romance quest, where a band of comrades sets out on a journey into a perilous domain on a quest for something of value. Often the exact nature of the object of the quest is a mystery. In the perilous domain, the comrades encounter obstacles that test their heroic virtue.
In medieval literature the knights of King Arthur’s Round Table engaged in a quest for the Holy Grail. Best represented in modern times by Month Python's version.
The world of the Snark is surreal and absurd. The poem’s central concern is being and nothingness. French surrealist Louis Aragon translated Carroll’s poem into French and believed that such nonsense writings were a subversive political protest against the rigid social and economic structures of Victorian England.
The Plot of the Poem goes like this:
After crossing the sea guided by the Bellman's map of the Ocean—a blank sheet of paper—the hunting party arrives in a strange land. The Baker recalls that his uncle once warned him that, though catching Snarks is all well and good, you must be careful; for, if your Snark is a Boojum, then you will softly and suddenly vanish away, and never be met with again.
With this in mind, the hunting party splits up to hunt. Along the way, the Butcher and Beaver -previously mutually wary for the Butcher's specialty in preparing beavers- become fast friends, the Barrister falls asleep and dreams of a court trial defended by the Snark, and the Banker loses his sanity after being attacked by a frumious Bandersnatch. At the end, the Baker calls out that he has found a Snark; but when the others arrive he has mysteriously disappeared, 'For the Snark was a Boojum, you see'.
The poem is a class of surreal and absurd literature popular in the 19th century, along with art and music. When translated into French by Louis Aragon, It was believed that such nonsense writings were a subversive political protest against the rigid social and economic structures of Victorian England.
The poem reminds me of the quest to remove the need to estimate the outcomes of our decisions in the presence of uncertainty. Along with the Snipe, the Snark is a mythical creature searched for by a band of comrades bent on a quest to over throw the establishment - as Carol suggest in Victorian England - and establish a new order in the absence of external authorities, governance (government for Carol), with the freedom to roam the world free of existential angst of looking after other peoples money.
Thanks to our daughter for suggested readings, who was a Classics Major. She is actually educated, while the rest of the family is just trained.
 http://www.literature.org/authors/carroll-lewis/the-hunting-of-the-snark/ Poem as presented by Literature.org
 Sidney Williams and Falconer Madan: Handbook of the Literature of the Rev. C.L. Dodgson, as quoted in Martin Gardner: The Annotated Snark, Penguin Books, 1974 1981 Edition published by William Kaufmann, ISBN 0-913232-36-X
 "The Consumption of the Snark and the Decline of Nonsense: A Medico-Linguistic Reading of Carroll’s ‘Fitful Agony’". Fernando Soto (Autumn 2001). The Carrollian (8): 9–50. ISSN 1462-6519
Phillip Armour has a classic article in CACM titled "Ten Unmyths of Project Estimation," Communications of the ACM (CACM), November 2002, Vol 45, No 11. Several of these Unmyths are applicable to the current #NoEstimates concept. Much of the misinformation about how estimating is the smell of dysfunction can be traced to these unmyths.
Mythology is not a lie ... it is metaphorical. It has been well said that mythology is the penultimate truth - Joseph Campbell, The Power of Myth
Using Campbell's quote, myths are not untrue. They are an essential truth, but wrapped in anecdotes that are not literally true. In our software development domain a myth is a truth that seems to be untrue. This is Armour's origin of the unmyth.
The unmyth is something that seems to be true but is actually false.
Let's look at the three core conjectures of the #Noestimates paradigm:
The Accuracy Myth
Estimates are not numeric values. they are probability distributions. If the Probability Distribution below represents the probability of the duration of a project, there is a finite minim - some time where the project cannot be completed in less time.
There is the highest probability, or the Most Likely duration for the project. This is the Mode of the distribution. There is a mid point in the distribution, the Median. This is the value between the highest and the lowest possible completion times. Then there is the Mean of the distribution. This is the average of all the possible completion times. And of course The Flaw of Averages is in effect for any decisions being made on this average value †
“It is moronic to predict without first establishing an error rate for a prediction and keeping track of one’s past record of accuracy” — Nassim Nicholas Taleb, Fooled By Randomness
If we want to answer the question What is the probability of completing ON OR BEFORE a specific date, we can look at the Cumulative Distribution Function (CDF) of the Probability Distribution Function (PDF). In the chart below the PDF has the earliest finish in mid-September 2014 and the latest finish early November 2014.
The 50% probability is 23 September 2014. In most of our work, we seek an 80% confidence level of completing ON OR BEFORE the need date.
The project then MUST have schedule, cost, and technical margin to protect that probabilistic date.
How much margin is another topic.
But projects without margin are late, over budget, and likely don't work on day one. Can't be complaining about poor project performance if you don't have margin, risk management, and a plan for managing both as well as the technical processes.
So what we need is not Accurate estimates, we need Useful estimates. The usefulness of the estimate is the degree to which it helps make optimal business decisions. The process of estimating is Buying Information. The Value of the estimates, like all value is determined by the cost to obtain that information. The value of the estimate of the opportunity cost, which is the different between the business decision made with the estimate and the business decision made without the estimate. ‡
Anyone suggesting that simple serial work streams can accurately forecast - an estimate of the completion time - MUST read Forecasting and Simulating Software Development Projects: Effective Modeling of Kanban & Scrum Projects using Monte Carlo Simulation, Troy Magennis.
In this book are the answers to all the questions those in the #NoEstimates camp say can't be answered.
The Accuracy Answer
But remember, making estimates is how you make business decisions with opportunity costs. Those opportunity costs are the basis of Microeconomics and Managerial Finance.
Cone of Uncertainty and Accuracy of Estimating
There is a popular myth that the Cone of Uncertainty prevents us from making accurate estimates. We now know we need useful estimates, but those are not prevented by in the cone of uncertainty. Here's the guidance we use on our Software Intensive Systems projects.
Finally in the estimate accuracy discussion comes the cost estimate. The chart below shows how cost is driven by the probabilistic elements of the project. Which brings us back to the fundamental principle that all project work is probabilistic. Modeling the cost, schedule, and probability of technical success is mandatory in any non-trivial project. By non-trivial I mean a de minimis project, one that if we're off by a lot it doesn't really matter to those paying.
The Commitment Unmyth
So now to the big bug a boo of #NoEstimates. Estimates are evil, because they are taken as commitments by management. They're taken as commitment by Bad Management, uninformed management., management that was asleep in the High School Probability and Statistics class, management that claims to have a Business degree, but never took the Business Statistics class.
So let's clear something up,
Commitment is how Business Works
Here's an example taken directly from ‡
Estimation is a technical activity of assembling technical information about a specific situation to create hypothetical scenarios that (we hope) support a business decision. Making a commitment based on these scenarios is a business function.
The Technical “Estimation” decisions include:
This kind of information allows us to calculate the amount of time we should allow to get there.
The Business “Commitment” and Risk decisions include:
These are the business consequences that determine how much risk we can afford to take.
Along with these of course is the risk associated with the uncertainty in the decisions. So estimating is also Risk Management and Risk Management is management in the presence of uncertainty. And the now familiar presentation from this blog.
Risk Management is how Adults manage projects - Tim Lister. Risk management is managing in the presence of uncertainty. All project work is probabilistic and creates uncertainty. Making decisions in the presence of uncertainty requires - mandates actually - making estimates (otherwise you're guess your pulling numbers from the rectal database). So if we're going to have an Adult conversation about managing in the presence of uncertainty, it's going to be around estimating. Making estimates. improving estimates, making estimates valuable to the decision makers.
Estimates are how business works - exploring for alternatives means willfully ignoring the needs of business. Proceed at your own risk
† This average notion is common in the No estimates community. Take all the past stories or story points and find the average value and use that for the future values. That is a serious error in statistical thinking, since without the variance being acceptable, that average can be wildly off form the actual future outcomes of the project
‡ Unmythology and the Science of Estimation, Corvus International, Inc., Chicago Software Process Improvement Network, C-Spin, October 23, 2013
The door of a bigoted mind opens outwards. The pressure of facts merely closes it more snugly.
- Ogden Nash
When there are new ideas being conjectured, it is best for the conversation to establish the principles on which those ideas can be tested. Without this the person making the conjecture has to defined the idea on personality, personal anecdotes, and personal experience alone
Any process that does not have provisions for its own refinement will eventually fail or be abandoned
- W. R. Corcoran, PhD, P.E., The Phoenix Handbook: The Ultimate Event Evaluation Manual for Finding Profit Improvement in Adverse Events, Nuclear Safety Review Concepts, 19 October 1997.
I have a card on my desk with a cute quote
May the Sun Always Shine Down on You ...as a constant reminder that there is a giant nuclear-powered fireball in the sky just barely holding it together.
Simply said It's Not About You
Science is the great antidote to the poison of enthusiasm and superstition.
- Adam Smith Wealth of Nations
If you hear a conjecture or a claim that sounds like it is not what you were taught in school, doesn't seem to make sense in a common sense way, or appears to violate established principles of science, math, or business - ask for the numbers.
When you have mastered numbers, you will in fact no longer be reading numbers, any more than you read words when reading books. You will be reading meanings.
–W. E. B. Du Bois, American sociologist, historian, civil rights activist
As project managers, numbers and the information they convey are the glue that holds the process of delivering value to the customer to together.
These numbers are plans, estimates, measures of Effectiveness, Performance, Risk, Value. There units of measure must always be meaningful to the decision makers. This is a fundamental issue when the units are useful only to a specific group.
Story Points or Stories are not units of measure found on the balanced sheet. The primary unit of measure for the business decision maker is Dollars and Time. All deliverables have a value measured in some form in Dollars and the time when that value will be available to start accruing that value.
When we speak in units of measures not meaningful to the decision makers, we obfuscate conversation, and become frustrated when those we should be informing don't understand what we are trying to convey.
When we hear you don't understand what I am telling you, it is not the obligation of the listener to understand in the presence of meaningless units. It is the obligation of the speaker. Classroom teacher know this, flight instructors know this, baseball coaches know this. Any mentor, coach, advisor, or consultant knows this.