Principles, Practices, and Processes to Increase Probability of Project Success
Software development ranges from straight forward, with small a team with a continuous improvement effort. Say a web site or warehouse management application were the customer is happy to just keep the improvements coming. Every improvement or new feature can be put to use. Along the way new ideas enter into the mix and since there is no real deadline, the features can be deployed when they are ready.
On the other end of this wide spectrum of projects is the mission critical, business critical project. For example a Mergers and Acquisition strategy of two large firms that need to join their ERP systems before any benefits from the merger for the customers, operations, and cash flow can be realized. Below is an example of the Plan for such a project.
The first thing to put to rest is the naive and misinformed notion that planning is somehow not needed. The Plan above is a strategy for the integration of legacy systems with a new ERP system. The Planned date for this system is tied to business success. Long before this Plan was developed, the business strategy identified the needed capabilities of the integrated system, the planned savings from the integrated system, and a customer facing set of abilities of the result.
This is the fundamental Return on Investment equation. ROI = (Value - Cost)/Cost. We know the value, since the analysis of the value of the integrated system was done before the merger. The cost was estimated at the highest level from experience of integrating ERP in the past. Details come next, usually after the merger.
Each of the boxes provides a needed capability. End box is an incremental deployment of this capability, with feedback that informs the downstream capabilities. No project can have an end-to-end detailed schedule with any hope of that schedule remaining intact. But the Plan above describes the order of delivery of the needed capabilities. These are not arbitrary, these are not random, these are not subject to change. At least not without understanding the impact on the business merger process and the resulting cost savings and cash flow generation.
So What's The Point
If the value at risk is sufficient to call into question business failure, or at least major issues, if the project fails to meet its goals, then we need a Plan, an estimated cost, and an estimated delivery date. To ignore these business needs is to ignore the obligation to provide that needed information.
You project may not be a merger of two multi-billion dollar firms. But the question isn't when to estimate but when do we no need to estimate. The asnswer to that starts with the Value at Risk. The business providing the funding gets to say what the value is.
What is the business willing to Risk on the project without knowing that value before starting?
So here's an immutable principle of all business:
For a business to stay in business it must manage its costs. Without managing cost there can be no hope of profit. Profit is why a business is in business. Sure there is that "let's change the world" mission statement. Or the "world domination" statement. But without profit those glorious goals cannot be met - unless your a sovereign. So managing cost and knowing cost is a precondition for profit. Profit is a condition for staying in business.
We can know cost from the past - these are called actuals or estimated actuals if the invoice hasn't arrived yet. Knowing costs in the future is called forecasting. Forecasting involves estimating. Estimating future cost is part of the immutable process of staying in business. If we can't forecast future cost, it's going to be hard to hang on to those revenues customers are giving us in exchange for some value we've delivered.
Enough of Business 101. Now on to project based business processes. Overrun our project cost for an external customer on a non-government job and our profit margin is reduced.
Overrun our baseline cost on a government cost-plus job and you're going to have to explain why we're a bone head to some contracting officer.
Overrun our government contract by 25% or greater and we'll have to explain to congress why you're a bone head that breached the Nunn-McCurdy limit.
Overrun our spend plan for staff, and someone is going to come and ask why we have more people than we planned. Have no plan for that number of people, the balance sheet is going to show the overrun to people you may have never met and they'll be asking us questions we may not have answers for.
Project performance starts with cost management. It may be likely that our salary is a recoverable (allowable) cost. It may be our salary is wrapped into the bill rate to the customer. But no matter the source of funds that pay your salary, the light bill, the really nice capichino maker, the pool table, Income - Cost is retained earnings. If we overrun our Planned cost, we're going to erode what remains from Revenue - Cost = Profit.
Any one not agreeing must be working for a not-for-profit or a non-profit organization. Cost management translates directly into profit margin goals. Profit Margin is revenue minus direct cost plusoverhead, indirects, and all other non-recoverable costs. Profits are what keeps the lights on. Profit is Net Margin.
So Now What?
Cost is one of two numbers in the Return on Investment equation every managerial accounting class taught us.
ROI = (Value - Cost) / Cost
Unplanned growth in cost reduces ROI. When we hear It's all about delivering customer value, nothing but value, starts and ends with value, ask at what cost? Don't know the cost, you can't determine the NET Value. Simple managerial accounting again. Not developer accounting in story points, not drip accounting in budget fed to your project. Dead Presidents accounting from the bank customers pay into to your payroll account your pay your mortgage from.
Any business not knowing what things cost for the work they are performing, be it internal work or external work, is not going to be in business for long.
Cost is KING
Once we come to understand that, we can now talk to business people and talk to engineers. Because most engineers work on budget. And budget is not the same as funding. Funding can buy bread at the store, It's real money. Budget is a cell in Excel or a number in SAP, or a number written on the White Board.
So Now What, Really?
If we're on the customer side, or we're in the accounting managers office, or the product managers office, or the account executives office, we'll be having a conversation about the cost of the wonderful work we're doing for our internal or external clients or our customers we're selling our product to. This is how business is run.
There is a mythical notion that places like Google, Facebook, Twitter don't care about cost. I'd suggest you go look at the 10-K and search for the term cost. You'll see things like cost of revenue. How much do we have to spend to bring in every dollar in revenue. Business runs on cost. It also runs on products, service, cool things no one can produce except for them, stuff we never ever thought about but they did. But when paychecks go out every two weeks to Chase, dead presidents are removed from one account and transferred to another - you're bank account. The cost of building all those cool, unimagined things requires real money. Spend too much - more than planned - profits go down.
How Can The Business Know What Costs It's Obligated to Pay Next Week, Month, Quarter, Year?
How you ask? They make estimates. Now these estimates may never reach us in the development department. We may only see budget, funding allocation, release of funds, or just a direction to go forth and build cool stuff. But someone, somewhere in the building needs to know the cost of our wonderful work.
This cost can be flat spread - peanut butter spread as we say where I work. A fixed number of Full Time Equivalents with a base cost and burden (fringe, overhead, and indirects). That's the budget for all the people working here. What happens then is we need to know our capacity for work. What can we produce for that cost? Unless we're building transmissions at Toyota, we'll have a variable cost of production. Somethings we work will cost more per the value produced, some will cost less. Software for example is NOT a production line. So if we've fixed the cost of production, the value produced will be variable. How variable? Oh I don't know, let's estimate.
Estimate from past performance. That's the absolute best way. What of there is no past performance, or no applicable past performance? Then estimate anyway. You're seeing a trend here right?
We have to estimate our cost of production in order to know, with any confidence, our future earnings. Where earnings are Revenue minus Cost.
No cost estimating process at the business level is simply nonsense. Now we as a developers may not be involved in cost estimating. But someone is. Let's go find that someone and ask. Do you really not want to know my cost of being here?
Software intensive project work is indeterminate. Many times, customers don't know what they really want until they see something working. Development's capacity for work is not well understood early in the project. Technology emerges and impacts the effectiveness of the project. Reducible and irreducible uncertainties abound. These conditions are normal, they can't be wished away.
So what are the options? Let's start with some advice and see what we can do to improve the Probability of Project Success. Wise leaders from the past have words we can use to guide our efforts.
I know one thing: that I know nothing
The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt
— Bertrand Russell
The fool doth think he is wise, but the wise man knows himself to be a fool.
— William Shakespeare, As You Like It
Knowing nothing about the project is not good for increasing the probability of success. Knowing what capabilities are needed to fulfill a mission or business need is a starting point. Capabilities are not requirements, those come next. Capabilities allow the user to do something of value. This is the value spoken about, most times spoken about in the absence of any units of measure. These measures, the measure of capability is the Measure of Effectiveness. (The tool in the previous link is a systems engineering tool, that provides all project participants with clear, concise, and testable assessments of their understanding of what Done looks like. The notion that tools get in the way of success is ill-informed at best, and just plain wrong at worst).
Measures of Effectiveness are operational measures of success that are closely related to the achievements of the mission or operational objectives evaluated in an operational environment, under a specific set of conditions.
If we don't know what capabilities we want the project to deliver, it's going to be a low probability that the project is a success. We don't know what done looks like. Now we can get paid to discover these capabilities. Capabilities Based Planning is a process to discover these. Someone has to pay for this of course. It is usually the customer that pays. But someone has to pay.
With the needed capabilities in hand, we can make our first estimate for cost, schedule, and the probability of success. Here's an example. Frank Cepollina was speaking at the BAA (Broad Area Announcement) for the Hubble Robotic Service mission.
He stated his needed capabilities rather than in techncial requirements. Those always come after we know what done looks like. Notice these capabilities are mission capabilities. Get the job done descriptions.
How much does that cost and when can you have it ready to fly?
The three vendors needed to develop an estimate. The details of this process can be found in Assessment of the options for extending the life of the Hubble Space Telescope. This example can be extended to IT and software development projects. What do we want the system to do? Don't know? Go find out. Make that the first step in the project. Read Predictabilty: A Simple Approach to Creating Reliable Project Schedules, Steve Bockman for an example of how to think about this in a software development environment. No matter what our domain, size of project, we're going to need to understand something about what capabilities are needed for project success. We might be able just to start work and let these emerge. But our probability of success is unknown and that's not likely a good thing in the end.
So when we know something about the needed capabilities, we can start to discover the requirements. But first we're going to need some kind of measure for those requirements. These are performance measures.
These are measures that characterize physical or functional attributes relating to the system operation, measured or estimated under specific conditions. These are Measures of Performance, and are:
These are measure of how the requirements are going to perform in pursuit of the mission.
Final we have to discover the Technical Performance Measures. These are attributes that determine how well a system or system element is satisfying or expected to satisfy a technical requirement or goal. These:
Here another need for estimating. How fast do we have to process transactions to meet the needed capability? How much can we weigh and still have the capability to fly? How reliable (mean time between failure, mean time to first failure, mean time to recover from failure), and still meet the needed capabilities?
These estimates are focused on technical outcomes. Thing that fly away, things that provide services in support of the customer. Things.
Where are we going here?
We must learn to estimate all kinds of things on a successful project. The credibility of our skills as project managers is always tied up with our ability to estimate. Can't estimate, choose not to estimate, see estimating as some kind of dysfunction, it is doubtful those providing us with money are going to have much confidence in our ability to deliver that mystical value we are told is part of all projects.
Estimating is part of project work. To not estimate, means we won't know what done looks like until we arrive at the end. That's unlikley to be acceptable to those paying us to do work.
The next step is to determine what requirements will be needed to implement the needed capabilities. These are defined as Measures of Performance that characterize physical or functional attributes relating to the system of operation, measured or estimated under specific conditions.
Notice something important here. Software project success always depends on development of working software. But what software, in what order, with what Technical Performance Measures, Measures of Performance, and Measures of Effectiveness all supporting the needed Capabilities.
Success doesn't start with coding. It's about knowing the answers to the CBP, MOE, MOP, TPM attributes. The notion of emergent anything works only when we can state in some unit of measure what DONE looks like. Without that we're just wasting the customer money exploring, challenging everything, and looking for the mystical Unicorn hiding in the bushes.
So Here's the Punch LIne
When we spend other people's money on our project, private money, public money, government money, they have the expectation of knowing something about how much much and when we will be finsihed with the work in exchange for that money. This is pretty much the case for every project I've ever worked for the past 34 years. It's the basis of every successful business - knowing what your costs are for the operations. So when we hear about developing software for money and not having any concern about the cost of that effort it simply doesn't make any sense.
The post 8 Reasons Estmates are too Low, is one of those pieces of material that on the surface seems plausible but has series flaws. First is the restating bad management practices and then arguing against them. This seems all too common in the Agile domain for some reason.
A poster campaign at Rocky Flats in the late 90's for safety and safeguards, but usable everywhere is...
Don't Do Stupid Things on Purpose
If we ignore the red herring approach of doing stupid things on purpose and then tilting at the resulting windmill, let's look further for each idea in the post. The picture to the right is used when engaging in a conversation about making improvement, but starting with a credible baseline. This is called Dead Horse on a Stick. Thanks to the Master Systems Engineer on our program for this concept. He uses this when he starts a review and the ideas are dead before he got there. It's also appropriate for concepts that are dead on arrival, like suggesting that those paying for products don't have a need to know how much it's going to cost, before they start spending money.
So what's the point?
When we look at making improvements to anything from projects and sepnding other people's money, to better peddling of my road bike on century rides - start with
stop doing stupid things on purpose.
Only then can you make real and lasting improvements. If you don't do that, you are beating a dead horse.
The common picture of requirements elicitation looks like this. Which of course is an example of doing stupid things on purpose. When this picture is used as an example of not doing something because it doesn't turn out right, is a further example of doing stupid things.
Let's see where the gaps appear that results in the outcome in the last panel:
The wheels fall off when there is no description of DONE shared between the customer and the provider. If the customer doesn't know what DONE looks like, who does? The developers? Probably not. Is DONE emerging? Then the customer has to pay for that?
There is no way out of the need to know what DONE looks like in Measures of Effectiveness, Measures of Performance, Key Performance Parameters, and Technical Performance Measures. Without these the success of the project is in doubt from the start.
The notion that requirements emerge is will established. But the capabilities the customer needs to be stable enough to establish an Estimate At Complete and an Estimated Duration to Complete. Without these the customer has no understanding of the all in cost of the project. These change as the requirements change. That's part of the project management process. This can't be ignored if the customer is to have any confidence in the project providing value at the needed time at needed to be put to use.
So a reminder one more time...
You can't assess the value of the project outcome without knowing something about the cost to deliver that value and the time frame for that delivery.
No matter what anyone says, this is an immutable business principle. Anything is simply fooling yourself into believing that the customer doesn't care how you spend her money or when you spend it.
Focus on value, that's what the customer bought. Actually they bought a capability to do something to improve their business or mission. But ignoring the cost of delivering that value is ignoring the balance sheet of the business.
Projects are composed of three fundamental elements. Cost, Schedule, and Technical outcomes. The Technical Outcomes go far beyond the PMI-style scope terms. In this paradigm, the technical outcomes are at the end of a chain. Here's examples of that chain - Capabilities, Measures of Effectiveness, Measures of Performance, Key Performance Parameters (there are 5 in our domain), and Technical Performance Measures. At the TPM level is where things like quality live, traceable to the KPPs.
These three elements are coupled in dynamic ways. Their connections are springy, in that changes in one has an impact on the other two, But rarely is this impact linear and ridgid. The Iron Triangle notion is really a Three Body problem, in which all three element impact each other and at the same time respond to that impact.
All projects have these three elements, coupled in this way. Changes in one impact the other two. Changes in two impact each other and the third. Without knowing the dynamics of cost, schedule, and technical performance, we can't have any credible understanding of these variables.
Three Body Problem
The three body problem determines the possible motions of three point masses m1, m2, and m3, which attract each other according to Newton's law of inverse squares. It started with the perturbative studies of Newton himself on the inequalities of the lunar motion. In the 1740s there was a search for solutions (or at least approximate solutions) of a system of ordinary differential equations by the works of Euler, Clairaut and d'Alembert (with an explanation by Clairaut of the motion of the lunar apogee).
Developed further by Lagrange, Laplace, and their followers, the mathematical theory entered a new era at the end of the 19th century with the works of Poincaré and since the 1950s with the development of computers. While the two-body problem is integrable and its solutions completely understood, solutions of the three-body problem (Java 7 in 64 bit Browser needed) may be of an arbitrary complexity and are very far from being completely understood.
The forces between the bodies can be self attractive or they can be a central force - the restrictive three body problem. Or a combination of the two. This is the basis of complex systems, where multiple forces are applied to objects, which in turn change the forces. As an aside, the double pendulum and the three body problem are used as examples of complex systems. Without acknowledging that the underlying mathematics is deterministic since the Java example above draws the lines from an algorithm.
This is a common mistake by those unable to do the math, or who want to suggest the problems of the day are beyond solution.
Three Body Problem and Three Elements of Project Management
The three body problem uses gravity as the force between the masses. There is a simpler example of three masses connected with three springs. This model is found in chemistry and biology, at the molecular level. Gravity is not in effect of course, but electromagnetic force.
Consider a simplified model for the vibrations of an ozone molecule consisting of three equal oxygen atoms. The atoms are represented by three equal point masses in equilibrium positions at the vertices of an equilateral triangle. They are connected by equal springs of constant k that lie along the arcs of the circle circumscribing the triangle. Mass points and springs are constrained to move on the circle, so that, e.g., the potential energy of a spring is determined by the arc length covered.
Now to Projects
If we assume for the moment that cost, schedule, and technical performance are dynamic variables, with forces between them described by their functional equations. In our functional equations for the force between them ar not constant, but are relationships like this:
The interaction between the three core elements (cost, schedule, technical) is a two way interaction, so the spring analogy is not quite correct, since the spring force doesn't know which end it is pushing or pulling.
It's not the Iron Triangle, it's a springy triangle. The connections are non-linear and most importantly they are probabilistically driven by the underlying statistical processes of the project. Let's start with the picture below.
All project processes are probabilistic. They have behaviours that are not fixed. The notion that you can slice work into same sized chunks and execute these chunks with the same effort would violate the basic aleatory uncertainties of all work processes. With an understanding of the statistical processes, driven by either aleatory or epsitemic uncertainties, is followed by asking probabilistic questions. What's the probability that we'll complete on or before a date or what's the probability we'll compete at or below a cost.
With a probability and statistics foundation, we can now put together a credible plan, driven by the underlying stochastic. All work is connected in dependent ways. The work effort, it's duration, and outcomes is also statistically driven. This picture is typical of such a project.
In The End
So we can:
In a conversation on Twitter with Sridhar, there was mention that product development software projects are different than internal IT projects. And that how they are run is much different as well. This may be the case. It's not been the case in my experience in Enterprise IT for those internal projects. My experience in product development (FileNet, Triconex, Kontron) and software intensive program delivery for government, defense, space, and energy clients is that projects are expected to show up on or before the planned date, at or below the planned cost and with the agreed set of minimal capabilities to accomplish the business goals or fulfill the needed mission.
When internal IT projects lose their ability to show up on time, on budget, with the needed capabilities - all within the probabilistically defined bounds of those measures - then they lose their seat at the table. Internal IT becomes a cost. Sometimes a sunk cost. An expense. A necessary expense, but an expense all the same. And when you're only an expense, those making decisions tend to treat you differently. You have no tangible way to show your worth. In business worth is defined by revenue or cost savings. In both cases worth is defined in units of money. When you have only cash outflow you're likely not to be in the conversation about how to run the business.
There are endless books, articles, and academic papers on the topic of getting a seat at the table. Successfully delivering internal IT in support of the business. Keeping your seat the table on that business requires very specific behaviours, starting with three simple questions:
But the bigger question - the killer question is what are we building and why are we building it? Ignoring for the moment the problem of showing up late and over budget dilutes any value produced by the project, some times to ZERO, we must have a notion of what DONE looks like in units of measure meaningful to the decision makers. Those paying the bills for the project.
Below is a briefing of how to answer these governance questions about what and why. With that in hand the how much, when, and working are straight forward.
There are many books on Balanced Scorecard. Some I find best are:
At The End of the Day
If you can't say, with some degree of confidence, how much you plan to spend, over what period of time, to deliver some tangible value to those paying for your work, you're not in the room when those paying for the work ae deciding what they want you to do. You're considered an expense - possibly a valuable expense.
There are Principles, Practices, and Processes of project success. I have a book coming that describes them. These are independent of any method, any domain, any favorite tool, buzz words, or personal paradigm. Much research on Root Causes of project failure have led to these Principles, Practices, and Processes. The Principles don't change, can't change. The Practices need to be tailored to the needs of the project and its paradigm. The Processes are localized.
But around these are 7 Activities of any project, that if not performed in some way, the probability of success is reduced, sometimes to Zero.
Domain and Context
No matter if your building bridges across rivers, a flying machine to Mars, the next anti-virus drug, or writing software using Scrum for your internal web site, these activities must take place in some way.
Without the last 4 activities, you're managing the project open loop. This might be OK if your customer doesn't care how much it will cost or how long it will take to finish the defined work. But of they do, you'll need a target cost and a target completion date, the feedback that you're making progress to those values. And of course some confidence that those targets are credible, their degree of variance, and how they are changing as the project progresses.
So one final comment
When we hear that we don't need to estimate, there is no way to have a closed loop control system. Without on estimate of the cost, schedule, and technical performance goals, the only management control system is open loop. This is unlikley to lead to success for any non-trivial project.
There are three key elements of every project on the planet - Cost, Schedule, and the performance of the product or service produced by the project. Each of these has drivers. The connections between Cost, Schedule, and Technical Performance are not Iron as suggested in the Iron Triangle of a PMI view of the project. Instead the connections are elastic, springy, flexible. But they are connected.
Cost is driven by:
These costs are themselves variable, as a function of the project phase, external forces for labor, materials, and overhead. But the cost variable starts with these.
Schedule is driven by:
Technical Performance is driven by:
So What Does All This Mean?
But for project success we need to have several things in place. The random behavior has to be knowable. It can't be chaos. If it is chaos, the project will fail, because there is no corrective action.
The three elements need to be known to some degree of confidence.
Do know these to some degree of confidence, you don't know what DONE looks. The only measure of progress becomes the passage of time and consumption of money. It's unlikely any customer is going to be willing to pay you - at least for very long - to spend their money, without some understand of Cost, Schedule, and the resulting Technical Outcomes.
The variables in the project or the product development process, are random variables, the their reducible and irreducible uncertainties creating risk to the probability of project success. Answering the questions above and the many other questions encountered in the business of writing software for money, require we make estimates of the outcomes of any decision that impacts the project or product. With these estimates, decisions can be made about the best path to take to increase the probability of success. By best it means the best with the probabilistic knowledge at hand.
Increasing the Probability of Program Success(project or product) must be the goal of every process improvement effort. When we hear of some new and untested approach to developing software in the presence of uncertainty while spending other peoples money, ask how does this increase the probability of success in units of measure meaningful to the decision maker. Rarely is that decision maker the one spending the money.
Here's some background on how we think about this critical topic.
The only certainty is uncertainty - Pliny the Elder (Gauus Plinus Secundus) (Natural History)
When the cost of a future state is considered the decision makers ask What is the chance the system's cost will exceed a particular amount? Or what are the uncertainties and how do they drive cost? Cost, schedule, and technical uncertainty analysis provides the decision makers insight into these and other important questions.
These uncertainties come from inaccuracies in cost and schedule estimates. They come from the misuse, misrepresentation, or misinterpretation of estimating data, or misapplied estimating methods. They come from intentionally ignoring the probabilistic and statistical nature of all project work.
But these uncertainties do not remove the need for the decision maker to know about the probabilities associated with the project, to some level of confidence of the cost, schedule, technical performance, and probability of project success.
This knowledge is needed before and during the project. Without this knowledge the very notion of making decisions is uninformed by the raw data needed to decide.
To not know, not be able to know, or not want to know, means basing decisions on ignorance of the emerging situation of the project. And since all project management processes are about making decisions, to make those decisions we need credible information. Estimates are one of those decision making pieces of data. To not have an estimate is to intenntionally ignore a piece of information critical to the sucess of any project.
So before you listen to anyone suggesting we don't need to estimate cost, schedule, and technical performance, ask them to show you their marked up copy of the book below (one of several dozen hands on guide books for estimating project cost and schedule) or have them point to where not knowing the cost, schedule, risk, or technical performance to some degree of confidence, before starting the project or during the project, is in the best interest of those funding that project.
A Fallacy of Estimation piece has an interesting phrase.
The post goes on to say...
Here's an ongoing collection of fallacy of estimating commentary, some useful, some misinformed, some not even right let alone wrong. Moving beyond the personal opinion into the realm of actual processes, tools, and people who estimate for a living probably has value when you're assigned a project that spends non-trivial amounts of money.
Ignoring for the moment the uninformed notion that estimating is the smell of dysfunction, since no dysfunctions have been mentioned in that context, let alone corrective actions, other than to Not Do Estimates, there is in fact a critical issue about estimating in all domains.
Of course the last statement as well ignores time phase for estimates. Before the project starts, what's our a risk exposure for the cost of this project? The let's get started and find out how much this will cost is like Steve Levitt's Freakonomics description of the drug dealers - here just try this, I give it to you for free. Now of course Not Estimating has no connection to getting people kooked on Crack Cocain, but let's get started spending your money and we'll find out later how much you're going to have to commit to this project, sounds a bit like bait and switch of the 1970's with Cal Worthington in Southern California, when he'd advertise a car that didn't exist, get you to come down, then up-sell everything - ah the good olde days.
Just heard the story on NPR yesterday about the Spanish firm that will Stop Work on the widening of the Panama Canal because they've overrun by a Billion $.
The Fallacy of Estimating term is used many times without attribution. It starts with the Daniel Kahneman and Amos Tversky and the difficulties humans have in making estimates. Their current book Think Fast and Slow is a continuation of their thesis. But the core of the thesis is contained in a few critical papers, that must be read before drawing any conclusion, and most importantly be read before listening to anyone who has read them.
But there's another set of knowledge needed to be successful in the estimating business and that the acknowledgement that all estimates are probabilitic. This can't be said enough. The place to start is Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, Paul R. Garvey. This book is the anchor for everything we do in the cost, schedule, and techncial performance estimating business on software intensive programs. Mr. Garvey's work at Mitre is the basis and many of the tools and processes used in our domain.
The last paper is one you must have on your desk if you're actually interested in solving the fallacy of estimating.
How Did We Get Into This Estimating Fallacy Mess?
The original explanation for the estimating fallacy was that planners focused on the most optimistic scenario for the project, rather than using past performance, subject matter experts, or parametric models of how much time similar work would require given similar conditions. This is the Optimism fallacy. What could possibly go wrong? Well we know the answer to that now don't we? This is common in our defense and space procurement world and most other worlds where high stakes projects are driven by politics. It a fundamental axiom of life. No Guts No Glory. When the value at risk is high, conservative actions go by the wayside.
Another explanation - one found in our domain as well - is the Authorization Imperitative. If we want to get our program approved, we'd better not tell them how much it will cost. The James Web Telescope and Joint Strike Fighter are good examples of that. JWT is currently at something like $7B it started out less than $1B. Similar for JSF.
So What Next?
It's the olde saw Doctor, Doctor it hurts when I do this? Than stop doing that. This of course is utter nonsense when it comes to estimating. Doctor, doctor we can't make good estimates. Estimates are the smell of dysfunction (with none listed) OK, then stop estimating and start spending. Yea Right!
A few facts of life:
How Can We Get Better Estimates?
Let's assume for a moment that we understand why we need to estimate how much of other people's money we're going to spend.
How do we get better? The answer is simple - We're spending other peoples money and they likely want to know how much, when, and what are they getting. We start by looking to the tried and true, field proven, tractable approaches to estimating. This is not a platitude. We start by doing our homework, reading books and papers, looking at tools, asking others how do you do this? We do what any person learning to do something new would do. We look to others first.
What's Out There to Learn?
There are lots of sources for learning to estimate. But the first - field proven way - is Reference Class Forecasting. It's the basis of most estimating processes in use today
Bent Flyvbjerg has lots to say on this. But some care is needed, he tends to overstate the intent of planners and estimators making poor estimates as Lairs. Maybe that's the case at times, I know of a few programs, but it's overstating none the less. Calling people Lairs is fight'en words where I come from in the Texas Panhandle, home of T. Boone Pickens, Dog the Bounty Hunter and Randy Matson
Let's assume you actually want to improve your estimating skills, abilities, and probability of success. Where do you start. First you start by ignoring those who say it can't be done, because it can. Then ignore those who say estimates are the smell of dysfunction because estimates are part of any credible business process - period. OK, if you believe in Unicorns and Pixy Dust, you might belive that making estimates is a dysfunction. Making estimates for the wrong reason is dysfunction. But doing anything for the wrong reason is a dysfunction. Learn to do things for the right reason and as the poster campaign at Rocky Flats said:
Don't do stupid things on purpose
By the way, the notion of drip funding is fine. But it does not answer the question what is our estimate at complete? Drip Funding is also called Time Boxed Scheduling been around for decades. Here's a small amount of money and a list of things I want you to do. Go do them, come back and we'll talk more. If you did them for more or less the money provided, good. If not, you've now got information to calibrate the future capacity for work. This called Reference Class Forecasting.
If you can't estimate credibly you won't be in business long. From the lawn care guy who cut our grass every to the builder of Joint Strike Fighter. OK, they're still in business ;<(
So the first place to start is to inform yourself how other do credible estimating. Let's start with How to Estimate if You Really Want To. But those aren't enough, you'll need some tools. And before you listen to anyone telling you tools get in the way of innovation and understand, ask if you can do non-stationary stochastic modeling of Monte Carlo Markov chains of dependent process flows in you head or by exchanging words between people? OK, back to the problem at hand.
There are many starting points for probabilistic estimating, but they all have one thing in common. We need to know what the problem is. For software we need to know what capabilities the customer would like to possess when the project is DONE. This is called Capabilities Based Planning. Capabilities aren't requirements. Capabilities reveal the requirements, and requirements enable the capabilities to exist. Here's an example of a set of evolving capabilities for a health insurance ERP system
So once we have something like this, we can start to decompose the parts into bite-sized chunks, just like those suggesting drip funding need for success. These are Drips of work.
Next you'll need to suspend belief, just like the Unicorns. The suspension goes like this.
For the vast majority of commercial and a whole lot of military and space software systems, there is nothing new under the sun.
You may not personally have had experience with this new requirement. You may have not even heard of such a thing. But help is at hand - Google. Start there, someone, somewhere, somehow has built something similar. Find it, ask them, do your homework, build a Reference Class. Can't build the Reference Class? OK, then spend some money to build a prototype. Charge the customer for this exploratory effort. This by the way is called agile development. Try a little, learn a little. Try some more, learn some more. Improve your probability of success with direct experience. But make sure you get paid for this. It's part of the project. Exploring like this on other people's money with out them knowing it or without them paying you for it is really bad business. That's a true dysfunction.
Now For Some Tools
Here's my list of favorites. They're favorites because I use them or know people who do:
There are likely others, so if you have one send me a note.
In the End
It's not your money, behave appropriately
Let's Start With End In Mind
If you work on projects and get paid for that work, someone is paying for you. That can be a direct customer if you're working on a external project. Or it can be the firms customers, buying a product or service which provides the money to pay you. No matter, someone pays for you to do what you do best.
Project Accounting is a subset of Finance Accounting, Both Are Important to the Project Manager
As a person working on a project, you may not care about Project Accounting or Finanical Accounting. But it's important to know where your pay check comes from, and it's not the Bank of America.
Project Accounting, sometimes called job cost accounting, creates data that tracks the financial performance of projects. Project Accounting enables the firm providing project resources (labor and material) to monitor the progress of their projects from a financial point of view. This is separate from standard organizational accounting for departments, divisions or the firm.
There are several data values used in project accounting. The project manager and the business manager(s) funding the project expect a certain level of profitability to be maintained in each function of the organization workiing on the project. The cost associated with delivering the project is Budgeted by the firm - usually before the customer of the project receives an invoice for the work and sends Funds in the form of real money. Budget is not dead presidents. Budget is an authorization to spend dead presidents, but you can't take your budget to Star Bucks and buy a Vente Latte.
For external projects the separation of internal costs - both direct and indirect - is many times done by finance. Usually the project costs are wrapped or grossed up for billing purposes to the customer. For internal projects, the fringe, labor overhead, material overhead, and G&A costs are not usually recorded directly to the project, but recorded in the Project Ledger as the wrap rate for your labor. Fringe, labor overhead, material overhead, G&A and other indirects are paid with actual funds - dead presidents - so those costs can be recorded outside the project, since they are not material to the project's performance if managed properly. We see all the time, when those indirect costs get out of hand the firm itself reduces its profit and they come looking for savings from your project.
An important concept of project cost accounting is the ability to provide visibility to targeted revenue against the actual revenue, and the estimated costs to produce that revenue against actual cost to produce that revenue. Many firms separate these two items - revenue and costs. The Project Manager focuses on the cost side - direct labor, materials, services, etc. And the Financial Accounting department focuses the revenue forecast and actuals. Since those direct labor, materials, and services - whether internal or external - have to be paid in real money, someone in Accounts Payable needs to know how much will be coming due in the next period.
This By The Way is the role of Estimating. How much labor will be needed to produce the planned outcomes of the project next quarter, next release, or the next anything? Fixed labor pool? Simply add up the FTE (Full Time Equivalents), multiply by the Fully Burdened labor rate and send that to accounting. But of course that means with a fixed labor pool, you'll need to know what your capacity for work is. Have a fixed commitment to deliver some value? Then you'll need to know how many FTE's that takes. Either way you'll need an estimate for those outside your project paying the bills.
What Does Project Accounting Do Best?
Project cost accounting records the costs associated with a particular project or job. Project accounting collects all cost - invoices, time cards, other direct project charges - and assures they are recorded in the proper charge account. These charges can be direct billed to the customer, or assigned to a Budget Item in the project cost baseline, or recorded as non-billable. When costs are non-billable, they are also considered non-recoverable and are usually captured in some overhead account. It's the cost of business.
It's a simple matter of balancing the books. Money In minus Money Out = Retained Earnings (money left over). It's of course more complex than this, but this will do for now.
This must balance...
All Income in the form of Accounts Receivable = All Outgo in the form of payments in "real money"
For project accounting to add value, care is needed to record and track every project cost. This by the way is one primary role of the Work Breakdown Structure. The WBS defines the budget and captures the cost of each project element. On the project side expenses are represented by direct labor (this is why time cards are used on many external contracts), invoices submitted properly for non-labor project costs.
One important issue in project accounting is who carries the overhead?
Project managers are usually not aware - at least directly - of the overhead, fringe, and other indirect costs to their projects. In some domains these costs are wrapped in a multiplier to the direct labor and recorded on the Project Balance sheet without any detail. This Wrap Rate value is important to the project manager, since this cost is subtracted from the revenue generated from the external customer. For internal projects, this wrapped cost is part of the ROI calculation for the cost of delivering the value from the project to the internal business customer.
In the end someone has to pay the electric bill, those free lunches for the project team, those Star Buck cards we hand out for hard work, and the spot award checks for other actions of the team. Those really nice 1080P monitors sitting on the engineers desks? Someone has to pay for those. The Aeron chairs? Someone has to pay. In the end someone has to pay.
Our daughter (now grown an gone) came home one day from her High School Economics class and announced, Dad we learned today there is no such thing as free. Yep, dear, welcome to the real world. People need to know what the real cost is. When they say free checking, someone is paying for that, probably you.
On projects, all that infrastructure you enjoy - laptops, lunchroom, parking, etc. is paid for by someone. If not directly, than indirectly. That's accounted for in the Financial Accounting System, the one the CFO is interested in. As a project manager you may or may not know about the wrap rate or even care about the wrap rate. But those who pay you do. And since they do, you may want to have some concern about the wrap rate as well. Fringe benefits, labor overhead, material overhead, G&A, etc. are real costs and impact that ellusive bottom line for your project and your firm.
So In the End
If you work on projects and are not concerned with the wrap rate either directly or indirectly (a pun for all us project planning and controls geeks), then you're probably direct labor. Wonderful direct labor, hired for your irreplaceable skills, but accounted for as labor all the same. Recorded on the books by your direct rate (annual or hourly), the fringe, overhead and other indirects.
So if you don't care for the Overhead discussion, or don't want to perform your role as a project manager using Overhead, just remember, those who pay you do care. And since they care, you may want to care, or at least pretend you care, you're job may depend on it.
The separation of Project Accounting from Financial Accounting is common in many firms. Finanical Acounting is the domain of the CFO or equivalent and the finance and accounting department, where payments of invoices to customers and from suppliers ae processed.
Project Accounting, sometimes called Job Cost Accounting, is usually the domain of the Project Manager. Where budgets are assigned and records of work peformed and cost of that work reported. But actual money doesn't change hands.
In the Financial Accounting domain, all costs and all income are recorded as transactions on the General Ledger. These transctions are rolled out through accounts to produce a Balance Sheet of the cost versus value in some form. Sometimes that value is revenue. Sometimes, it's business value recorded as intangible asset. Managing of the intangible assets of a firm is the primary role of business leaders. Teh tangible assets are usualy managed in the accounting, by the depaertment with the same name.
I'm making this simple, and likely simple minded for the point that is coming. Who owns the overhead costs that are present in every single business? For larger complex projects here's one framework for doing that, but we'll keep it simple for now.
But before going simple here's a quick looks at FASB 86, the guidance for accouting for the development and use of software. But one quick notion that is missed by many in the agile world not working the arcane processes of public accounting ...
The Agile Alliance is working this issue.
So Back To Project Accounting Processes
Projects spend budget. Businesses spend Funds. If you work on a project, rarely does someone on the business side come to you with an envelope full of money - dead presidents - and say go hire some developers to write code for this project. Those authorizing you to hire people, or buy materials, do so with a Purchase Order of some sort (we have PCARDS that skip all the paper work), or some authorization from HR to hire. The employee or contractor then charges to a charge number or Employee Number to capture their expenses (labor or material).
That expense is then wrapped with fringe, overhead, and other indirects and recorded on the books as the Fully Burdened cost to the project.
Someone Has to Pay
No matter where you work in the organization, the all in cost for the project has to be paid. This includes your direct labor. What you get in your pay check, those fringe benefits - health insurance, matching 401(k), AMC movie passes, spot awards. Your firm has to pay. If you're a 1099 supplier, then it's you that has to pay. As a 1099, if you're not billing your customer for all that overhead than those costs are reducing your revenue. Same for a firm.
So if you're not concerned about overhead, don't manage the project knowing the overhead. Not a problem in principle. In practice though someone does, and if they care, you may want to care as well. If not, then you're likley working as labor on the project. That's in NO way a problem. We're all labor in some form to some one. But when it comes to managing projects and managing the business that projects use or produce, these indirect costs can make or break the project and or the business. Knowing about them, managing in their presence is simply the responsible thing to do. You know likey knowing what your project will cost when it's done.
Here's some background that have served me well over time as a Program Planning and Control manager.
It is popular to make a list of maxims for developing products, managing projects, or managing business processes. Some are based on experience, some based on surveys, some based on principles and practices of a profession.
Here's mine based on counter examples of the sole contributor paradigm. The sole (or small group) contributor paradigm means maxims gathered from personal experience from a person's engagement on the job. One example for the sole contributor, used without permission and with full attribution is Woody Zuill's list. There are others Five Project Maxims, 18 Maxims of Successful IT Consulting and other. But I like Woody's framework best, because his topics fit best with our processes on complex, mission critical, software intensive programs and the hands on integration with process. Although Woody would likely not agree, both technical skills and formal process frameworks are critical success factors in any sufficiently complex domain - both are needed.
Doing the work is guided by the Strategy and Performance Goals of the needed Capabilities.
Without a clear and concise understanding of what DONE looks like in Measures of Effectiveness for the needed capabilities, all the project work has no home. It's just a list of features or functions captured by the development team from the customer or product owner.
It's the capability to accomplish a business strategy that defines the mission and vision of the project. Why are we doing this project? How will we recognize we've accomplished our mission? The capabilities delivered by the project starts with the fulfillment of Critical Success Factors. Which in turn implements a Performance Goal in support of a Strategic Objective that measurably benefits the business or supports a mission.
Responding to Change is impossible unless the system is easy to change, easy to maintain, easy to fix, and easy to enhance.
The ability to easily change a product or a process starts and ends with the architecture. This understanding began with Notes on the Synthesis of Form, Christopher Alexander, 1964. It's the architecture that enables the change, assuring that coupling among the components is minimized, cohesion between the static and dynamic processes is maximized, separation of concerns is traceable to all architecture decisions. If you're developing these as you go - allowing them to emerge - you're going to be disappointed when you discover your product is coupled in ways you didn't know, has weak cohesion among it's parts, and has cross cutting concerns which result in a tangled mess when you start to make changes.
The notion that the best architectures emerge are suggested by those not working on complex systems interdependent components, but on systems with lower levels of compelxity between the components. Imagine an enterprise ERP system, a software intensive manufacturing system, the 32 flight and weapons computers on the F-35, the multiple levels of interaction of Future Combat System (I worked the rebaslining of the IMP/IMS for Class I), and process control system found in a nuclear power station.
Now ask, would you like the architecture of these software systems to emerge as the development takes place?
Here's where to start for architecture in the enterprise IT domain. There are architectures for realtime embedded systems as well. For defense systems DoDAF is the architecture framework. So when you here responding to change ask - what's the mechanism that allows you to do that, when the system you're working on is complex, high risk, critically important - say banking, navigation and control, oil & gas supply chain, electric power generation and delivery, health care, drug development, retail, transportation? You get the idea
The notion of Question Everything ignores the fundamentals of every professional process improvement paradigm.
Working on projects is not about the needs of the individual. It's about the needs of the whole. Personal desires must be subordinate to the needs of the mission success. It's not about you. It's about the customer and the governance framework in which the customer operates her business or fulfills her mission.
Questions are great, you can learn at lot from questions. But questions asked without doing your homework are a waste of your time and those you are asking the question to. Go do your homework. Learn about ITIL, COBIT, INCOSE Systems Engineering, SEI, and other professional frameworks first. Then you'll have a basis for your questions. Then start with the root cause test for your questions. When someone says those haven't worked in their experience, don't just ask the 5 whys, seek the root cause.
The Why's approach may be able to reveal the symptoms. But to get at the root cause a deeper assessment is needed. One based on a process framework. A place to the Whys to land. Why didn't the work team follow the established test procedures? Why didn't the customer establish a set of needed capabilities before we started developing stories for the software development effort? These whys then reveal the root cause. The whys need to have actionable outcomes, not just the question. 1st graders can ask why.
Process improvement needs to ask why, but it can only deliver value when there is an actionable answer. No actionable answer in units of measure meaningful to the decision makers? The question everything paradigm is Muda (waste).
Working Product is product that meets the Technical Performance Measures (TPM), the Measure of Performance (MoP), and Measure of Effectiveness (MoE) as defined by the Concept of Operations (ConOps), Statement of Objectives (SOO), and Statement of Work (SOW).
Without stating these attributes of the working product there is no way to tell if it is the right working product. Right for the needed capabilities. Right for the strategy. Right for the technical, operational, and performance requirements. Simply saying working product in the absence of these measures is ignoring the large context of effectiveness and strategic value. When we hear many software features have little value, we can only determine that if the planned strategic value is defined and tested along the way. This is NOT Big Design Up Front. It is the core of strategy making.
But the notion of having working software be put to immediate use needs a domain and context for it to be useful. Otherwise it's just another platitude of the agile vocabulary. Working on orbit for a Navigation and Guidance computer may not happen for 9 months. That's the time it takes to get to Mars. So working needs an operational definition. Working in the full fidelity emulator of the space craft. Working in the complete Verification and Validation (100% thread coverage) of the emergency shutdown system. (I was one of gthe orginal architects of this system). Working in the full transaction processes system test bed.
Crunch-time is a symptom of harmful and counter-productive attitudes.
It's got nothing to do with attitudes, and everything to do with competent and mature business management and processes. Newspapers have crunch time every day, sometime twice or three times a day. Banks have crunch time every month. Surgeons have crunch time once they make their fist cut. 3 miles out onto a hot LZ in I-Corp, 1969, is crunch time delivering critical supplies to Fire Base Rip Cord. Flying to New York City has crunch time every time the 777 pushes back from the gate at LAX. It's not attitude, it's competency to manage in the presence of uncertainty and deliver as promised because you've been trained, have experience and a support system. But in the end it's the process. Process rules.
Knowing the capacity for work starts with knowing the demand for work. Throughput can only be determined if you know both demand and capacity. Then and only then, can you add margin for the irreducible uncertainties. And reserve for the reducible uncertainties.
We are only innovators of our process if we are capable of providing the innovative solution within the governance framework of our business.
If it ain't broke don't fix it. If it's broken first find the root cause and fix that cause. Rarely in modern business is the a broken process that didn't work right at one time. A critical success factor for all process improvement is to determine the root cause of the failure. Then and only then examine if there is a process problem. If so, fix the process. If not, fix the application of the process. Stop wasting time looking for solutions to the wrong problem.
The object of all projects is to deliver value to those paying you to do the work.
Writing software for money is not the same as producing art for money. If you're producing art for money, you're probably not a very good artist. If you're treating your job of producing value for money as art, you're probably not getting a lot of repeat customers.
Customers bought the capability to do something, they only care if you're self-actualized if they are a relative. Customers are happy when you've fulfilled their need to possess a capability for the expected cost on the expected day. There must lots of opportunities for participants on a project to receive personal satisfaction, grow together as a team, increase their skills, and even be innovative - but the customer rarely is willing to pay for that directly. It better be wrapped in the overhead rate. Good artists copy, great artist steal - Pablo Picasso. Good firms hire people already prepared to succeed. Read Making the Impossible Possible: Leading Extraordinary Performance: The Story of Rocky Flats for specific actgionable advice on doing all the things needed for success, including all the people processes. The abstract is here.
The more we understand that improvement is hard work. There is no free ride.
Nobody Ever Gets Credit for Fixing Problems that Never Happened: Creating and Sustaining Process Improvement is a start. They suggest less than 10% of the firms adopting Toyota's TQM actually apply it properly. This loops back to the question everything nonsense, when the questioning is uninformed by the missing root cause analysis of the dysfunction. The source of Dysfunction in the workplace must be determined before any suggestions for improvement can be made. Stating something is the smell of dysfunction is like stating what's that rotten smell as we drive by the recycling center/ Look out the window and see the source. Find the source before doing anything.
At the end of the day successfully managing projects is hard work. But there is plenty of advice. This is one of my favorites.