And the Principles, Practices, and Processes to Increase Probability of Success
Where You Stand Depends On Where You Sit
This notion of the basis of all good discussions. It is also the basis of discussions that get us in trouble. For example, I sit in a FAR 34.2/DFARS 234.2 Federal Procurement paradigm and a similar paradigm for Enterprise IT based in ISO 12207, ITIL V3.1, CMMI, or similar governance processes.
Both these domains are guided by a Governance framework for spending other people's money, planning for that spend, performing the work with that money, reporting the progress to plan for that spend to produce the needed value, and taking corrective actions when the outcomes don't match the plan to increase the probability that the needed value (Capabilities) will be delivered for the planned cost to keep the Return On Investment on track.
This paradigm is independent of the software development method - traditional or agile.
If you work where the customer has a low need to know the cost, schedule, or what will be produced for the money, then you likely sit somewhere else.
I work on Agile software development programs. Most everyone in Boulder Colorado works on Agile development programs. We meet once a month or so for coffee and talk about Agile. We have formal MeetUps hosted by local vendors - Rally, Scaled Agile, and other agile development shops.
The range of projects at our morning coffee clutch go from one man shows to multi-billion dollar space flight programs and back again. There are times when we get pissed off at each other for all the right reasons - this is the big ended littled ended argument of Gulliver's Travels that was actually an argument about the bit order in microprocessors when the 32 bit machines started with floating point calculations - Holy Wars and a Plea for Peace. When I was working on embedded systems and we had to choose a 32 bit machine for our new signal processing system, we got in huge fights about how the floating points software was going to be moved over to the floating point hardware.
So the question is what are the shared principles across a broad range of projects, business and technical domains. Here's some background on the domain I work and the application of Agile in that domain. I'm speaking at the Boulder Agile Meetup July 27th if anyone is interested in hearing about Agile in Government.
Here's some background on Agile in the domain I work
Typically these are also Software Intensive System of Systems. Here's some background on those
So when we hear something about exploring new approaches, ask first - in what domain have you tested that idea? By what Principles can that new idea be accepted into a domain outside your domain? Is there any evidence that your new idea can work outside your personal experience? How would I test your idea before spending my customer's money? What would be those test cases?
The latest thread in agile is ...
the continued paradigm of deadline-driven development is killing the benefits that Agile Software Development can bring.
It is suggested by Neil Killick that ...
... using genuine time constraints as a factor in the prioritisation of work rather than as a focus for its execution, the odds of meeting those "deadlines" are actually improved.
Not sure what a genuine time constraint is versus any time constraint. But the conjecture that Executives of software product and service companies always want stuff delivered faster, cheaper and better. Agile principles and methods are believed to be a way to achieve this ignores several fundamental principles of all work and especially software development work.
All project work has uncertanty. Reducible uncertanty and irreducible uncertanty.
Agile does not remove this uncertanty. Agile is NOT a risk management process. Genuine constraints don't remove this uncertanty., I've spoken many times in the past about how to Manage in the Presence of Uncertainty.
These principles are always in place. More so on Agile projects, where emergent requirements are encouraged, which in turn drive uncertanty further to the right in the Probability Distribution Function of the possible range of durations, costs, and technical performance shortfalls.
When those uncertainties are not considered and handled, any project is going to have an increased chance of being late, over budget, and have technical issues.
Setting genuine constraints may make this issue visible, but does not remove the risk to the project's probability of success. Only active risk management and the necessary margin can increase this probability.
The only protection for irreducible uncertanty is Margin and the only protection for reducible uncertainty is active risk management. Both of these activities require careful planning and execution of the plan, along with the estimate of the probability of occurrence of a reducible event and the statistical distribution of the naturally occurring variances and the probabilistic impact on the success of the project.
This is the Reason for Planning
It's been suggested I work in a unique domain, where deadlines and need dates are themselves unique. This is False.
No credible business, no matter the size, Doesn't have a need date for the Value produced by the software project. If there was no need date, the developers would show up whenever they wanted after spending whatever they wanted, with whatever they thought the customer needed.
Ignoring the simple time cost of money and the time phased Return of Investment of (Value-Cost)/Cost, any business that intends to stay in business is spending money for software - either developed or purchased - to provide some value to the business. Not having a need date for the production of that Value means the business is ignoring the core principle of retained earnings. Even non-profits and not-for-profit business (and I've worked there as well) have a time value of money economic model.
So if you're going to produce value for your customer, that value is most always time sensitive other wise it's de minimis value. If it's time sensitive, there is a deadline. If there's a deadline, reducible and irreducible uncertanty and the risk it produces must be handled.
Risk Management is How Adults Manage Projects - Tim Lister
† The naive notion that scrum teams are self contained and need no external support is only the case when there is little at risk for the resulting code. Cyber security, Database integrity, performance validation, operational integrity are external surveillance roles on any Software Intensive System of Systems. This is called Governance and guided by documents like ISO 12207, ITIL V3.1, COBIT, DoDAF, ToGAF.
A primary learning process is research. This process is part of all science, engineering, management processes. Here's a starting point for learning how to estimate in the presence of uncertainty on agile projects.
This is the start of a literature search. Anyone making suggestions about making decisions in the presence uncertainty on agile project can start here for establishing the basis of any arguments of how and why that suggestion (conjecture possibly) is based on some set of principles. Not just anecdotal opinion.
The conjecture of #NoEstimates starts with the first Tweet
This conjectures - (there are) ... ways to make decisions with No Estimates ... is not founded on any principle of business management, microeconomics of decision making, or principles of probability and statistics. It's a fallacy.
Let's start with a simple approach to exploring for anything?
The Hypothesis might well be (if there was one) is ... can we make a decision in the presence of Uncertainty without making an estimate of the impact or outcome of that decision?
Let's put aside for the moment the missing principles of managerial finance, probabilistic decision making, microeconomics of decision making, Real Options, Bayesian decision networks, and other decision making processes used in modern business when spending other people's money. And ask a simple question...
What would be the evidence that we could make decisions in the presence of uncertanty without estimating the impacts and outcomes of those decisions?
The Myths of No Estimates and the busting of them is one purpose of this blog post. Here are some books and papers that can provide you with all the tools needed to learn to estimate in the presence of uncertainty. As well these books and papers can show you the snake oil salesman's fallacy that estimates are hard, are a waste, and aren't needed to make decisions in the presence of uncertainty.
Before listening to any conjecture that estimates aren't needed to make decisions in the presence of Uncertainty for software development, please read these books. Ask the person making those conjectures if they have read the books. Ask to see the marked up, sticky noted, tabbed copy of the book and the notes they made from the content. If not, walk away. They are not informed by the principles of spending other people's money.
Before we start, let's look with the notion of estimation. A Guess is uninformed by experience, skill, or data. An estimate is.
This is a simple book that will show how to estimate most everything. Start here. Read the entire book,. Learn how to think as an estimator.
Take that experience and start making estimates for software projects. Then and only then ask yourself why do people claim estimates are wrong, why is estimating hard. And you'll know the answer - they have no experience, they have no skill, they have no knowledge of how estimates can be made. Then you'll know. It's not that estimates that are smell of dysfunction, it's the unskilled, inexperienced, untrained, uninformed, unknowledge people making those unsubstantiated claims.
The next learning opportunity is to realize how much nonsense is floating around about not estimating. Here's the start for that understanding.
Managing a business profitably is always hard work. There are intense pressures, incomplete information about what’s happening in the marketplace and an army of consultants, advisors and others who come along with new ideas every day. Under these conditions, it isn’t surprising managers sometimes fall victim to hype about “miracle” cures for management challenges or simply adopt the “best practices” of other successful companies. The result is sometimes poor-quality decisions are made which end up wasting time and money which are badly needed elsewhere.
This book was handed out by Jeff Sutherland at the State of Agile conference as an indication of how agile has come to represent sloppy thinking on behalf of many of its advocates.
Here's a start of actually estimating in a credible manner, using tools available to anyone.
Start here to learn how simple it is to estimate things. Things you have never seen before. Things you have never done before Using Enrico Fermi’s theory of approximation.
The Fermi estimate, or order estimation is an estimation problem, teaches dimensional analysis, approximation, using a back-of-the-envelope calculation. The estimation technique, named after physicist Enrico Fermi, makes good approximate calculations with little or no actual data. Fermi problems typically involve making justified guesses about quantities and their variance or lower and upper bounds.
There is no excuse for not learning how to apply this approach to software development.
With the basic concept that estimates are straightforward, this book shows the economics of managing iterative development. Here estimates are part of planning and measuring progress.
The book speaks to assessing the technical viability of the system and the overall cost to achieve that technical viability.
These are core business processes for the success of any investment. And of course based on making decisions in the presence of uncertainty.
Now that we've established with the above sources there are conjectures without basis, nonsense statements like estimates are the smell of dysfunction without ever stating what the dysfunction is or what could possibly be causing the dysfunction, here's the place to start for serious estimating for software intensive systems.
A de minimis software project rarely benefits from estimates. Willfully bad management rarely benefits from learning how to estimate. If you customer has no real value at risk, of has no real concern about showing up on time to start accruing the business value in exchange for the cost to produce that value, then estimates are unlikely to be needed.
This is a seminal book on estimating. It provides the background and the practices needed to estimate Agile projects.
Mike shows why traditional processes fail and why agile approaches work. It's standard Feature Story breakdown but with the reasoning behind the processes.
This is the other seminal book on estimating.
Between these two books, you'll find need to know to make credible estimates for the development of software using Agile. The book claiming to show hwo to improve your project performance by 10X is so riddled full of holes, it's worthless. Don't waste your money on it. For the same price as that book, you can own both Mike's and Steve's books with field proven examples rather than fictitious anecdotes of bad management practices.
All project work is probabilistic. Some probabilities are event based, some are statistically based. Cost is always a driving consideration for how products are built, delivered, and sustained. A critical success factor for all software project work is cost and the associated schedule. Showing up on or near the needed time at or near the planned cost is simple business. Anyone saying cost is not important is not accountable for the business success of the development. Ignore them, they have no seat at the business management table.
If we come out late with the product we lose revenue needed to meet the business plan. If we spend more than planned, we erode profit and fail to meet the business plan.
Both these variables - cost and schedule - along the the needed technical performance to meet the expected acceptance of the product are probabilistic. Estimates are needed to inform the decision makers of the Probability of Success of the product.
Setting the date before the project is planned is the oldest root cause of project failure. Not having some sense of the scope of work, the effort needed to produce that scope, and the capacity for the development to produce the needed features for that scope is in the same class of failure modes.
Ignoring the need to have estimates of effort, time, and cost is not only bad project development it is bad business management.
Any suggestion that decisions can be made in the presence of project uncertanty without estimates willfully ignores these principles.
Large Scale Scrum (Less) is a framework for scaling Scrum to the enterprise. In the Less method, estimates are part of the Product Backlog Refinement. This process:
Agile at Scale is not the same as small agile teams. Governance drives processes in Agile at Scale. Governance can be ignored or even flaunted for small self contained teams. Organizing for responsiveness to external business drivers at scale means additional cost must be absorbed to govern in the presence of these externalities.
Agile at Scale also means dealing with architecture - technical architecture,process architecture, business architecture. Managing in the presence of all these uncertainties mandates we estimate the impact of these decisions. At scale without estimating the work processes turn into chaos.
This is one of the books that started it all. In Highsmith's books he calls out explicitly the need for estimating and some of the steps to do it.
Any enterprise agile development operating inside a corporate governance process requires knowing something about the outcomes of the work effort. The cost, the expected delivery date, and the expected business value produce for the cost. The time cost of money is a foundation of business decision making.
O those paying you have no need to know the time cost of money, how much it will cost, what the possible business value will be and when you might be done, then estimating is not needed.
Anyone suggesting you can't estimate must read this book to learn that conjecture is false - patently false - and learn how to actually make estimates of anything.
What I've found is when there are statements like all estimates are wrong, we can't estimate, estimates are misused is that those making those conjectures aren't actually informed how good estimates are made.
So instead of learning about estimates, they fall into the fallacy of #Noestimates.
This is another book that if the #Noestimates advocates had read on day one of their exploring they would have found the destination and stopped spouting all the nonsense about the smell of dysfunction.
Troy combines all of my favorite topics - mathematics of project management, monte carlo simulation - the actual monte carlo simulation, not the bogus sampling of past performance advocated by some No estimator's, and a logical, clear, and concise approach to making estimates in the presence of uncertanty to inform the decision makers when spending their money.
There are many fallacies in the software development world. Many of these fallacies go unchecked, unaddressed, unchallenged.
Here's where to start learning about the fallacies and their facts.
This is a similar issue with #Noestimates. This starts with the missing principle by which a decision can be made in the presence of uncertanty without estimating. Attempts to question these missing principles, results in further fallacies being put forward, along with labeling anyone probing further for the missing principle as aggressive, rude.
This is the basis of software estimating. It's been updated for agile.
Read it first, then read the agile updates. Learn how to use numbers, models, evidence, tools to estimate in the presence of uncertanty.
This is how non-trivial projects are managed. Anyone suggesting this is olde school hasn't done their homework to earn the position to be critical of the contents.
Spending other people's money involves governance of those decisions.
Governance by its definition is about decision rights
Who gets to the decide what is needed, when it is needed, how much it could cost (affordability). These decision rights are almost alwasy assigned to those paying for the outcomes.
The #NoEstimates advocates would have those decision rights transferred to those who spend, by suggesting estimates are a waste. Without stating who they are waste to. It is very unlikely they are a waste to those provided the funds that enable the #NoEstimates advocates to produce the software needed by those providing the funds.
It's illogical to invert this relationship between those paying and those spending.
Now we're into the heavy lifting of estimating. Yes estimating is forecasting. Forecasting is an integral part of the decision making activities of management. The organization establishes goals and objectives to produce outcomes from its decisions. The need to forecast (estimate future outcomes) is based on management's need to decrease its dependence on chance and become more predictive in dealing with the uncertainty it encounters.
We're now down to the core processes of making decisions in the presence of uncertainty. This is how business operates. Anyone conjecturing that decisions can be made in the presence of uncertanty without estimating - forecasting is estimating outcomes in the future - needs to stop and learn more.
Learn how business actually works. Not just assume that some who make an unsubstantiated statement about estimates are the smell of dysfunction has any credibility when spending other people's money. It's time to call BS on the notion of #NoEstmates.
Along with these books here's a collection of papers and articles showing how to estimate on agile development projects and how to avoid the Snake Oil Sales Pitch of #NoEstimates
So stop listening to the fallacy that estimates aren't needed to make decisions. And start learning to estimate and be a proper steward of your customers money.
From Estimating Software Costs: Bringing Realism To Estimating, 2nd Edition.
Agile addresses 1, 2, 3, 4, 5, and 6 well.
So if these are the causes of project difficulties - and there may be others since this publication, what are the fixes?
There's a meme going around in some parts of agile that management is inhumane. This is an extreme view of course, likely informed by anecdotal experience with Bad Management, or worse lack of actual management experience.
Managing in the presence of Agile is not the same as managing in traditional domains. The platitude is Stewardship, but that has little actionable outcomes needed to move the work efforts along toward getting value delivered to customers in exchange for money and time.
One view of management in Agile can be informed by Governance of the work efforts. Here's a version of Governance, from "Agile Governance Theory: conceptual development,” Alexandre J. H. de O. Luna, Philippe Kruchten , and Hermano Moura.
There a popular notions in the agile development world that authors like Hayek and Taleb speak to how software development works. Let's look at one example. You Can’t Understand Agile Without Understanding Hayek. Let's look at Hayek's paper. "The Use of Knowledge in Society," F. A. Hayek, The American Economic Review, Vol. 35, No. 4, September 1945, pp. 519-530.
Let's look at the thesis of Hayek in light of software development and the decisions that must be made when spending other people's money in the presence of uncertainty. Below is from the paper, but download the paper and read the whole thesis. Hayek was a social scientist. He was not a program manager of engineered to order software intensive system of systems. His ideas have not held up well. Hayek and Modern Macroeconomics. So I'd suggest not spending your customers money before doing some more research.
What is the problem we wish to solve when we try to construct a rational economic order?
On certain familiar assumptions the answer is simple enough. If we possess all the relevant information, if we can start out from a given system of preferences and if we command complete knowledge of available means, the problem which remains is purely one of logic. That is, the answer to the question of what is the best use of the available means is implicit in our assumptions. The conditions which the solution of this optimum problem must satisfy have been fully worked out and can be stated best in mathematical form: put at their briefest, they are that the marginal rates of substitution between any two commodities or factors must be the same in all their different uses.
This, however, is emphatically not the economic problem which society faces. And the economic calculus which we have developed to solve this logical problem, though an important step toward the solution of the economic problem of society, does not yet provide an answer to it. The reason for this is that the "data" from which the economic calculus starts are never for the whole society "given" to a single mind which could work out the implications, and can never be so given.
The peculiar character of the problem of a rational economic order is determined precisely by the fact that the knowledge of the circumstances of which we must make use never exists in concentrated or integrated form, but solely as the dispersed bits of incomplete and frequently contradictory knowledge which all the separate individuals possess. The economic problem of society is thus not merely a problem of how to allocate "given" resources-if "given" is taken to mean given to a single mind which deliberately solves the problem set by these "data." It is rather a problem of how to secure the best use of resources known to any of the members of society, for ends whose relative importance only these individuals know. Or, to put it briefly, it is a problem of the utilization of knowledge not given to anyone in its totality.
First some questions:
Is there vague and unfolding knowledge, driven by externalities, of what the project should be accomplishing, with uncertain and variable funding? With members of the project team having differing views of the outcomes from those paying for the project?
If so, the project has already failed, the members of the project team are on a death march (Yourdan) and should start looking for work elsewhere, before it gets canceled.
So like the other misrepresentations of seminal works, the agile community adopts this attractive ideas with little or no consideration or understanding of their principles. And most of all it those principles are applicable to software development or if those principles are even applicable in today's understanding of who the world works. Macroeconomics is the dismal science - treat it as such when developing software.
So whenever there is some conjecture that someone outside the software development domain has a principle that can be applied to Agile, test that idea first before spending any time and money acting on it. Especially if that time and money is not yours.
A popular mantra in Agile is deliver fast, deliver often. In some domains this may be applicable. Where we work and where agile is becoming the norm, we have another view.
Deliver as planned
The Plan for the delivery of value is shown in the Product Roadmap and implemented in the Cadence Release Plan, or sometimes in the Capabilities Release plan. Fast is a term replaced by Planned. The Plan is based on a Capabilities Based Plan. This Plan shows the increasing maturity of the Capabilities produced by the project. These Capabilities are needed by the business to accomplish the business case or fulfill a Mission.
Showing up fast is defined by showing up when needed. The need is defined in the Capabilities Based Planning process.
A Capability is the ability to accomplish something of Value. Here's a sample of what a Capability sounds like.
These are mission and business case terms, defined by the owners of the mission or Business Case. If y9ou show up Fast, that also means you can't show up early. For a simple reason - early means you may not be able to put that Value to work. It may mean that Value is not needed yet, and that Value may have to change when we get ready to use that Value. This is the role of the Plan.
In what order do we need what Capabilities - and all their associated technical and operational requirements having been fulfilled - for the needed cost, effectiveness, and performance requirements as well.
A final example - one of my favorites is the notion of the intent of the commander as the basis of defining capabilities. I have a colleague, who was General Schwarzkopf's logistic person in the first Gulf war. She was an Army Colonel and one of a small number of women combatants at the time. There are many more now, but she was a pioneer. One of reasons the US Army was able to move up the coast prior to crossing into Iraq in rapid time was because of her and her staffs planning skills. The notion of agile is the basis of all military process, not just 5 coders in the room with their customer.
So this statement says it all in terms of needed Capabilities
So when you hear deliver early and deliver often. Ask a simple question - what are we delivering? Is that deliverable arriving in the right order for the end user - the customer, the warfighter? Are there any predecessors to that deliverable that have to be in place for the FAST deliverable to be of any use?
This is the role of a Capabilities Based Plan. If your project has no interdependencies, if everything that is produced can be used as a standalone deliverable - arriving in any order - than Capabilities Based Planning is not likely to be of much value. And that's fine. But when we enter the Agile At Scale domain - ERP, Enterprise IT, Software Intensive System of Systems - we've got a separate issue. Order does matter. Fast is no longer of much value. As planned and as needed are the Critical Success Factors.
And a final thought
If you're going to Deliver Fast, do you have a Plan for how to do that? No, then how in the world are you going to deliver fast of you don't know what you are going to deliver, when that delivery will be done, and how you are going to deliver that value? Without a plan of some sort, how can you assert the naive notion of deliver fast and deliver often can ever be executed? It's just a platitude, without any actionable outcomes without a Plan on how to do that.
Jurgen Appelo used the double pendulum on page 42 of his Management 3.0 book to explain the differences between simple, complicated, ordered, complex, and chaotic. He defines the term Chaotic as “very unpredictable.” Like many of Jurgen’s definitions, they are localized to suit the needs of the story line.
There is no universally excepted definition of chaos. But almost everyone would agree on the following ingredients:
Chaos is an aperiodic long-term behavior in a deterministic system that exhibits sensitive dependence on initial conditions.
In this context, the phrase aperiodic long-term behavior means that the motion does not settle down to a fixed point or a periodic orbit. Since the double pendulum loses energy to the environment, after some time the motion does become periodic and it eventually stops at a stationary fixed point. In this sense it is only the theoretical double pendulum without energy losses that would really be a chaotic system.
Please read that last sentence again. It is critical. As well the “sensitivity” to initial conditions is a parametric measure in itself. The starting angle of the pendulum is one parameter. Low starting angles result in different “sensitivities” than larger starting angles. This is an exercise for students in the introductory classical mechanics class as an undergraduate in Physics.
A deterministic system means that the system has no random or noisy inputs. The irregular behavior is intrinsic and arises from the system’s non-linearity rather than from any noisy driving forces.
Please read this last sentence again. It is critical to understanding the definitions needed to describe the behavior of the double pendulum.
Sensitive dependence to initial conditions means that nearby trajectories separate exponentially fast, i.e. two identical systems set up together in the same way such that the initial conditions are arbitrarily close together will have their trajectories rapidly diverge. To make this more concrete, consider two trajectories, where at some time t the trajectories are at position x(t) and x(t) + d(t), then the statement of chaos would be that d(t) ~ d(0) exp [ L t] , where the average value of L is called the Lyapunov exponent, and if this is positive it means that the two trajectories are quickly separating from each other.
Why is this an issue in the management of agile software projects? Good question?
Management 3.0 - and now #NoEstimates advocates - proffers a solution to a complex problem of managing the development of software. The book, while providing advice to managers on how to manage, mixes pseudo-scientific references and concepts – like the double pendulum – in support of essentially sound staffing and personnel management. I came to the book, through Jurgen’s himself. But on first reading I ran straight into what seemed like a collection of ideas that have no actual basis in fact. The double pendulum is just an example of this approach.
So here's the fix for these conjectures. There's a paper "Distilling Free-Form Natural Laws from Experimental Data," Michael Schmidt and Hod Lipson, Science, Vol. 324 3 April 2009, showing not only the equations of motion for the double pendulum, but a machine that can deduce these equations by observing the double pendulum in motion.
Here’s the core problem. When we can't get the analogies right, what else isn’t right in the foundational principles proposed by those suggesting we can't operate in the presence of uncertainty? If those analogies miss the mark on the underlying principles of these analogies, are the other suggested approached equally flawed? Maybe, maybe not, but for someone like me, trained and experienced in the application of approaches to solving complex problems, many of the fundamental approaches used in the book are simply muddled thinking. It’s too bad. A good editor, with experience in the analogies Jurgen uses could have established that they are just notional, analogies, or possible just anecdotal experiences. Instead Jurgen states them as the foundations of the principles of Management 3.0. In the same way the original posters of #NoEstimates state their case that decisions CAN be made without estimating, when in fact that violates microeconomics, managerial finance, and several other principles.
And of course, this plays directly into the #NoEstimates conjectures, based on even less credibility than Jurgen's management processes - minus the illformed analogies.
There is no principle stated to date by the advocates of #NoEstimates that supports the conjecture that decisions can be made in the presence of uncertainty with estimating the impact on the business of those decisions.
Plans are nothing, Planning is Everything- General Dwight D. Eisenhower
While Eisenhower didn't say this quote as an original. That was "No Battle Plan Survives Contact With the Enemy" - General Helmuth von Moltke.
The quote is used - and misused - by most agilest and in the Agile Manifesto. Of course those authors probably didn't have access to the original context. Eisenhower was speaking to the National Defense Executive Reserve Conference in Washington D.C. on November 14, 1957. The quote in full context reads:
I tell this story to illustrate the truth of the statement I heard long ago in the Army: Plans are worthless, but planning is everything. There is a very great distinction because when you are planning for an emergency you must start with this one thing: the very definition of “emergency” is that it is unexpected, therefore it is not going to happen the way you are planning.
We know this for sure - any forecast about the future is wrong. All forecasts are predictions about the future, so the forecast has to be wrong. Is the forecast worthless? Of Course not. But all Plans and Forecasts must have the ability to adapt to the emerging situation
The plan establishes a goal and provides an outline and a starting point for allocating resources, delivering outcomes in the needed order, determining conflicts and dependencies. Anyone making plans and forecasting the outcome from those Plans must understand the limitations of making decisions in the presence of uncertainty.
When the conditions change - as they must in the presence of uncertainty - who best can respond quickly and make adjustment the plan or forecast? Of course the people who built the plan!
Planning is Strategy Making. All strategies are hypotheses. Hypotheses must be tested with experiments. Experiments provide actual data to adjust the Hypothesis, which in turn adjusts the Strategies, represented in the plan.
So when we hear from the Agilest who quote the Agile Manifesto - Responding to Change over Following the Plan to mean we don't need a plan, because we're not going it anyway. They have made a fatal error in understanding.
To respond to change - as the manifesto says - means we have to have a Plan to know what we are responding it and how our responses will impact to goal of the project - which the user has stated (or should have stated as the Needed Capabilities). Otherwise everything is a change and the Product Backlog is just a pile of ideas - not a map of anticipated value to be delivered to the customer.
And of course - since the future is always uncertain, when we encounter the naturally occurring changes and the event based changes, we'll need to estimate to impact of those change on our Plan to meet the Goals for the project. And we'll need to estimate the impacts of the alternative choices to get back on the path toward our Goal.
Without making those estimates we're in an Open Loop Control System being whipped around by externalities , creating extra work, building technical debt, and increasing Work In Progress, and generally mismanaging the project.
Scrum is an Empirical software development management system
Although the Scrum Guide does not contain that term. Emperical processes by definition contain these elements:
These attributes did not start with Scrum, they are control system terms
Let's look at a non-software development empirical process control system
We want to design a process control for a hot water system for an industrial process. We have chosen two simple attributes - the level of the hot water feed tank - so we need a pumping system to keep the water level at the desired level. This in turn keeps the head pressure at the desired level for the users of the hot water. The temperature of the water when it arrives at the point of need. This type of closed loop control system is seen on many towns across the US, where Water Towers are used to produce the water pressure at the desired - usually 65 PSI - for the residents, by pumping the ground water to a height of the tank then releasing it on demand for the residents. Let's ignore the temperature aspects for the moment and just focus on keeping the tank full
We would like to model this system before we build it to avoid spending our customers money more than once. There are two simple ways of modeling the system.
The empirical model looks like this.
The water tank level is held constant by equalizing the water flow out and the water flow in. The flow out is changed nm a step wise manner and the level response is observed. A series of step wise changes are made and the response is observed. As the level changes - and acts like an integrating or ramp process - the average time delay and gain (in percent of level change per percent of water added) can be calculated to produce.
The analytical model looks like this.
What Does All This Mean for Sofware Estimating?
Managing software development is a Control System. We have a steering target - the tank level - in the form of needed capabilities, a planned delivery date, and a planned budget for the delivery of the needed capabilities at the needed time. If you have no planned budget, planned delivery date, or needed capabilities, you have an open loop control system - just keep spending till the money runs out or the customer tells you to stop. No need for a control system, no need to estimate, just code, someone else will look after the business side of your spending.
But let's also pretend for a moment that those providing the money have a fiduciary obligation to know something about how much it will cost to produce the needed capabilities (Value) and some need to know when those Capabilities will be ready for use, so they can start earning back their investment. This is standard Business 101 time phased Return on Investment. And let's pretend that the development of the software is not a deterministic system. That there are uncertainties in the system. Uncertainties from lots of sources, some you can control and some you can't.
Then what does a model of the control system look like? It can be empirical or analytical, or a combination of both.
It needs to be both for some simple reasons:
This of course is NEVER true in software development. If it were true developing software would be a mechanical process, done by machines. And it's not.
So while empirical data is useful, it's not the only thing needed to manage the project so you can show up at or near the time needed, when at or near the planned cost, and at or near the needed capabilities, so the business can fulfill its business model.
An analytical model is also needed, fed by empirical data.
Both are needed, full understanding that the future is never like the past - and analytical modeling must include the probabilistic aspects of the future - possibly derived from the past.
So when we hear - Scrum is an empirical process, (it is true) but think a bit more on what that means. When we hear #NoEstimates is an empirical process - which it's not because it's open loop, with no steering target (a set point in the control systems paradigm), and no consideration for the probabilistic aspects of writing code for money - someone else's money.
Ask a simple question - if you use the word empirical - do you have a notion of the control system model in which that empirical data is used? Or is the term empirical data just a word used without any meaning to the problem at had.
That problem at hand is how to forecast (estimate in the future) in the presence of uncertainty about things meaningful to the business paying for the software. This means cost, schedule, and technical outcomes.
How can that be done using empirical data and NOT call that estimating?
For some software focused background of control systems see this simple starting point
There is a popular notion is some circles of Scrum, that Sprints need to produce shippable software at the end of every Sprint. This of course depends as always on the domain.
How about ...
So yes, Sprints MUST produce tangible measurable outcomes that can be verified to be compliant with the needs of the customer and other stockholders,m including external stockholders, but
...but your governance framework should not stop you from building a potentially shippable product increment each sprint
needs a VERY broad definition of Potentially Shippable Product, a specific domain and the governance framework for that domain. Only then can that notion have any chance of being a useful process goal.
And of course all this planning, executing, testing, V&V, performance assessment, etc. etc. etc. needs to have estimates for the probability of arriving on planned completion date, at the planned cost, and the needed capabilities fully functioning as required in exchange for the money being paid for the resulting value.
I'm working two programs where Agile at Scale is the development paradigm. When we start an engagement using other peoples money, in this case the money of a sovereign, we make sure everyone is on the same page. When Agile at Scale is applied, it is usually applied on programs that have tripped the FAR 34.2/DFARS 234.2 levels for Earned Value Management. This means $20M programs are self assessed and $100M and above are validated by the DCMA (Defense Contract Management Agency).
While these programs are applying Agile, usually Scrum, they are also subject to EIA-748-C compliance and a list of other DID's (Data Item Description) and other procurement, development, testing, and operational guidelines . These means there are multiple constraints on how the progress to plan is reported to the customer - the sovereign.
These programs are not 5 guys at the same table as their customer exploring what will be needed for mission success when they're done. These programs are not everyone's cup of tea, but agile is a powerful tool in the right hands of Software Intensive System of Systems for Mission Critical programs. Programs that MUST, deliver the needed Capabilities, at the Needed Time, for the Planned Cost, within the planned Margins for cost, schedule, and technical performance.
One place to start to improve the probability that we're all on the same page is this reading list. This is not an exhaustive list, and it is ever growing. But it's a start. It's hoped this list is the basis of a shared understanding that while Agile is a near universal principle, there are practices that must be tailored to specific domains. And one's experience in one domain may or may not be applicable to other domains.
Like it says in the Scrum Guide.
Scrum (n): A framework within which people can address complex adaptive problems, while productively and creatively delivering products of the highest possible value.
And since Scrum is an agile software development framework, Scrum is a framework not a methodology. Scrum of Scrums, Agile At Scale, especially Agile at Scale inside EIA-748-C programs has much different needs than 5 people sitting at the same table with their customer with an emerging set of requirements where the needed capabilities are vague until they appear.
One of the classes every aspiring grad student has to take is research methods. This class teaches the PhD hopefuls (I didn't make the cut and got a consolation prize of a MS), all about doing research and preparing to be a real scientist. A topic in this class is literature search. This makes sure that your cleaver idea of a research topic, in case your advisor hasn't gotten around at actually talking to you, has already been taken, researched, and solved. This is one problem in the physics world - you need an original idea. Replicating old ideas doesn't get you very far.
Here's a start of a literature search on merging Agile at Scale with Earned Value Management. I haven't gotten to the European and Far East journals yet. Instead is a list, I'll just type this once and repurpose the resources here. This PDF is the Resources section of a briefing being used with our clients who are integrating Agile into EVM programs. Go to the LinkedIn Slideshare site - the LI logo in lower right, to open the PDF and follow the links.
A popular Agile phrase goes like this.
...it's important to know what Not to build
This begs the question, who's building things that other people don't want? Where's the process for assessing what people want? Seems pretty obvious that if you're building thing that you shouldn't be building you're not doing a very good job of requirements management.
Software requirements are a weak link in the chain of software engineering technologies. Requirements are usually incomplete and change at rates in excess of 2% per calendar month. For many years one common definition of quality has been “conformance to requirements.” However this definition ignores the fact that some requirements are hazardous or “toxic” and should not be included in software applications. Since clients themselves may not realize the dangers of toxic requirements, software engineers have a professional and ethical responsibility to point out the hazards of dangerous requirements and ensure that they are safely eliminated. An example of a “toxic requirement” is the famous Y2K problem which did not originate as a coding bug but rather as an explicit but dangerous user requirement.
Capers Jones, Vice President and CTO, Namcook Analytics LLC
So how is it decided if a requirement is toxic or not. Well it's pretty straight forward:
This is the role of a requirements elicitation and management process. One based on an actually process, not just some made up, some cockamany approach to writing things done on sticky notes and asking the next passing person what they think. An actual, true to life requirements engineering process.
This starts with a simple concept:
A requirement is a statement that identifies a capability, characteristic, or quality factor for a system for it to have value and utility to a customer or user.
If we don't know these capabilities are, their attributes, characteristics, or other factors are, we won't recognize them before the money runs out. Can we over specify the requirements? Perhaps we can. Can we under specify the requirements? It's done all the time to the detriment of the project. We need to find a way to specify the requirements in a sweet spot. Not too much, not too little.
Research shows the average investment in requirements elicitation and management is 2% to 3% of the total project cost. Research also shows this is wholly inadequate for success and is one of the root causes of failure on any non-trivial project. This research shows that projects that expended 2% to 3% on requirements experienced a 80% to 200% cost overrun. Those projects that expended 8% to 14% on requirements elicitation and management experienced 0% to 50% cost overruns.
We must be crystal clear here. Requirements may emerge, but the needed capabilities at the needed time are a critical success factor for any project, no matter the domain. As Yogi reminds us. If you don't know where you're going, you might not get there.
I'm not going to tell you how to develop requirements,. There are many ways, starting with the guidance listed below and endless other books articles, and papers. But if there is a notion that requirements are not needed - let's let them emerge and we'll start coding to see what comes out - it's going to be a rough ride before the money runs out.
 The Requirements Engineering Handbook, Ralph R. Young, Artech House, 2004
 Requirements Engineering: A Good Practice Guide, Ian Sommerville and Pete Sawyer, John Wiley & Sons, 1997
 Succeeding with Agile: Software Development Using Scrum, Mike Cohn Addison-Wesley, 2010.
 Project Management the Agile Way: Making it Work in the Enterprise, John C. Goodpasture, J. Ross, 2010.
 Agile Estimating and Planning, Mike Cohn, Prentice Hall, 2006
 Agile Project Management for Government, Brian Wernham, Maitland & Strong, 2012.
Both Agile development and Earned Value Management are coming together in the Federal Government. DOD and many Federal Civil agencies are adopting the use of Agile software development. At the same time, FAR 34.2 and DFARS 234.2 are flow down on procurement programs that implement Earned Value Management.
I've written about to benefits of this before, Here's some thoughts on the Dark Side
Risk Management is How Adults Manage Projects - Tim Lister
Tim's quote offends some people. Here's a collection of his presentations. But this not really the point of this blog. I'm working two programs where Agile (Scrum and SAFe) is integrated into the Earned Value Management System, complaint with EIA-748-C and the guidelines for deploying, using, and reporting progress to plan with that system. This integration is directed by FAR 34.2 and DFARS 234.2 on Software Intensive System of Systems programs.
These programs are similar to Enterprise IT programs, they have mission critical outcomes, they are expensive, they have high risk, they are not simple structure - by definition, and they have deadlines and budget goals set of the enterprise for maintenance of corporate performance.
Outside of the programs we work, we encounter much confusion around risk management, the source of risk, and the separation of the effective and non-effective process of managing risk in the presence of uncertainty. Let's start with a framework for making decisions in the presence of uncertainty. This briefing has been shown before here, many times. But it can't be shown too many times, because there is much misinformation about how to manage in the presence of uncertainty. This briefing is part of a much large set of charts developed over several years for an Office of Secretary of Defense for Acquisition, Technology and Logistics. These are the people that buy things for the warfighter and those who support them.
It may not appear to be applicable in all domains, but I'd strongly suggest it is. Because uncertainty is present on all projects. And the risk this uncertainty creates is present on all projects. Uncertainty and the resulting risk cannot be avoided it can only be managed.
When we hear Agile is Risk Management it's not true. Agile Software Development in the form of Scrum, XP, DSDM, Crystal, and now even Agile Prince2 is just that - a Software Development Method using the principles of agile.
In some domains by the way a few of those principles are violations of governance of other people's money. Many business people have day jobs and aren't going to be available on demand to work with the developers. standards - DODAF, ToGAF, 1553 Bus, ISO 12207 etc. define architectural processes.
But that aside for now, Agile provides information for risk management through frequent feedback and adaptability. But Agile is NOT risk management for a simple reason ...
Agile does not model the uncertainties that underly all project work.
Agile has no notion of margin. Agile has an informal notion of risk reduction of Reducible Risks (Epistemic uncertainties if you read the briefing). But those irreducible uncertainties (Aleatory) are not addressed by Agile. Research shows the irreducible uncertainties are the killer problems on all project work. These come from the natural variances of humans, processes, and machines.
Agile in the form of methods has no process model for addressing the reducible and irreducible uncertainties other than to provide data on performance to date to make decisions about mitigations, margin, and management reserve needed to MANAGE Risk in the presence of uncertainty.
And of course the final tag line
To Manage Risk in the presence of uncertainty we must Estimate not only the probability of occurrence of an Event based risk, but also Estimate the statistical processes of naturally occurring variances and then estimate the impact of those uncertainties creating risk, Estimate the cost and schedule to mitigate those risk, and finally Estimate the residual uncertainty after mitigation.
Yes, Risk Management is How Adults Manage Projects and of course Estimating is how Adults Manage Projects
The Holy Grail of all Agile discussions goes like this ...
We focus on value over cost
This is a mantra repeated by agilest and vendors of agile tools as well. The big question is ...
What are the units of measure of Value?
The units of measure of Cost are dollars. When we hear We Focus on Value over Cost, in what units of measure are these two variables being compared to one another? A better question is where is this Value being defined?. And another question how is this value being defined?
If we are going to move beyond the platitude of value over cost - which by the way is simply bad economics, since the ,,,
Value of something can't be determined unless you know the cost to acquire that value
but let's ignore this naïve concept for the moment. How can Value be defined and measured?
In our Software Intensive System of Systems world, System Engineering is the dominant paradigm for increasing the probability of program success. This is not the Systems Engineering of the IT server systems engineering. This is the INCOSE Systems Engineering, defined as:
Systems Engineering is an interdisciplinary approach and means to enable the realization of successful systems.
Systems Engineering focuses on defining customer needs and required functionality early in the development cycle, documenting requirements, then proceeding with design synthesis and system validation while considering the problems encountered for:
Systems Engineering integrates all the disciplines and specialty groups into a team effort forming a development process that proceeds from concept to production to operation. Systems Engineering considers both the business and the technical needs of all customers with the goal of providing a quality product that meets the user needs.
In this paradigm there are two primary measures that the product or service being produced satisfies the needs of those paying for the work:
So Now What?
We've got some high level definitions, but are no closer to the units of measure needed to compare Value with Cost.
The Measures of Effectiveness (MOE) are defined by the customer or user point of view. These are the customer's key indicators that the mission has been achieved in terms of performance, suitability, and affordability across the lifecycle of the product or service.
MOE's focus on the systems capabilities to achieve mission success, within the total operational environment. MOEs represent the customer's most important evaluation and acceptance criteria.
If the customer doesn't know the Measures of Effectiveness in some form, to some level of confidence, the software project is on a Death March and no software development method is going to fix this problem.
The Measures of Performance state the attributes considered important to ensure that the system has the capability to achieve the operational objectives. MOPs are used to assess whether the system meets design or performance requirements that are necessary to satisfy the Measures of Effectiveness. MOPs are derived from or provide insight to the MOEs or other user needs.
If the customer doesn't know the Measures of Performance in some form, to some level of confidence, the software project is on a Death March and no software development method is going to fix that problem.
In the End
When we hear value over cost and don't have a unit of measure for Value it's just a platitude. When we hear value over cost and don't know the cost to achieve that Value, it's just a platitude.
So don't fall for the platitude approach to spending other peoples money in the presence of uncertainty. Define the MOEs, MOPs and the cost to achieve them.
With the MOEs and MOPs there is a 3rd measure for the products or services that must also be connected with the cost for achieving them. Technical Performance Measures.
With all three (MOE,MOP, TPM) those paying for the work can monetize these to establish a common basis of measure with the Cost to produce the value.
Until those conjecturing you should focus on Value over Cost can produce units of measure for that comparison, consider their statements as just platitudes with no actionable outcomes.
Estimating is an informed assessment of an uncertain event. All project work operates in the presence of Uncertainty. Estimating means developing probabilistic confidence intervals for the possible outcomes of that work in the presence of uncertainty.
The numbers we use on projects come in two types
Story points are Ordinal numbers, Stories are Cardinal numbers. Story Points are relative measures of effort. They are not duration or cost. Story Points are measures of relative effort 
Story Points are arbitrary measures used by Scrum teams to determine the Relative (Ordinal) effort of the work. They tell the team how hard a story is, from it’s perceive complexity, risk, unknowns – each related to effort. These Relative (Ordinal) measures are the antithesis of Business Management measures of work planning and accomplishments, which are in Hours and their rated dollars for the direct labor needed to produce the outcome (assuming no material cost).
Story Points don't tell us the duration or cost of this relative effort. Story Points dont' tell us the absolute effort to performance the work. They aren’t normalized across work efforts, across teams, or across the program. Story Point effort estimates are not Calibrated across the project, but rather are developed for the work at hand. The calibrated units of measure for Story Points – can and will change – change as the program progresses.
Business operated in units of dollars and duration for the work needed to produce the needed capabilities in exchange of those dollars and time. Business does not operate in units of Story Points.
The killer question is what is a Story Point Worth to those paying for the work? Agile teams rarely produce comparable calibrated Story Points for dissimilar or even similar work. This is a key difference between Business estimates and Agile estimating. Most businesses have an external Basis of Estimate process to calibrate the cost and duration of planned work. Business teams working on different parts of the project, with different assessments of Effort, different story point values, and different project costs result in dissimilar units of measure for a Story Point.
When Agile teams have different approaches to applying Story Points, the physical effort can still be calculated for each team, and rolled up to the Total Story Point count for the project for an individual Feature Physical Percent Complete.
The program level budget can flowed down to the planned Work in the Product Backlog and connected with the Total Story Point count built bottom up from the Agile Planning process.
From there, all Physical Percent Complete calculations remain the same - units of Story Points and Dollars
With the proper application of Story Points, at the agile estimating level, the Business can produce a Cardinal estimate the cost of the work with some simple rules:
Measuring The Project with Stories and Their Completon Rate
There is a conjecyure that measuring Stories is better than measuring Story Points. Here's the simple answer to that conjecture
This can only be valid if the Stories are statistically similar enough that their individuals variances (range of actual effort versus the estimated effort) in the collection of Stories is de minimis. That is the Stories are statistically identical.
If this is not the case, then uses Stories rather than Story Points as the measure of effort and conversation of those measures into units meaningful to the business is a fools errand. It violates the principles of statistical process control since the unit of measures of plan and progress itself has statistical variance unaccounted for the the data received is bogus.
If you don't have statistically identical relative efforts for all stories, never use Stories, Only use Story Points.
As well, a second caution is the false assumption that the future is always like the past. I got a book for Christmas Fool Proof, Greg Ip and our false belief that the future will be like the past. On any non-trivial project this is never the case, so making the assumptions that all stories are of the same size turns us into the fool in Fool Proof.
 Scrum + Engineering Practices: Experiences of Three Microsoft Teams, Laurie Williams North Carolina University, Gabe Brown, Adam Meltzer, Nachiappan Nagappan, Microsoft Corporation