At a recent conference the discussion of the integration of Agile with Earned Value Management on programs subject to FAR 34.201 and DFARS 252.234-7001 was the topic. Here's my presentation.
Principles, Practices, and Processes to Increase Probability of Success
At a recent conference the discussion of the integration of Agile with Earned Value Management on programs subject to FAR 34.201 and DFARS 252.234-7001 was the topic. Here's my presentation.
In the business of building software intensive systems; estimating, performance forecasting and management, closed loop control in the presence of uncertainty for all variables is the foundation needed for increasing the probability of success.
This means math is involved in planning, estimating, measuring, analysis, and corrective actions to Keep the Program Green.
When we have past performance data, here's one approach...
Many times I hear about Cost of Delay, Deliver Value, Measure story points, or Measure Stories. And a myriad of other assessments of project performance, all of which - OK, most of which are examples of Open Loop Control.
Back in 2014, we had a paper in a publication of the College of Performance Management, starting on Page 17. As well, a colleague Nick Pisano (CDR US Navy Retired) has a post on the same topic at his blog.
The notion of a baseline let alone a Performance Measurement Baseline is at the heart of Closed Loop Control of all processes, from your heating and air conditioning system in your house, to the flight controls on the 737-700 winging its way back home to Denver, to the project you're working - using what ever project management method or software development method of your choosing.
The notion that we can manage anything, the temperature of the room, the nice soft ride in the 737, or the probability of showing up on or before the need date, at or below the needed cost, with the needed capabilities - and NOT have a baseline to steer to is simply wrong.
Below is the framework for Closed Loop control. This paradigm says simply:
ESTIMATING IS THE BASIS OF DECISION MAKING - it can't be any clearer than that.
We gave a recent College of Performance Management webinar on using techncial progress to inform Earned Value. Here's the annotated charts.
In the estimating discussion there is a popular notion that we can't possibly estimate something we haven't done before. So we have to explore - using the customers money by the way - to discover what we don't know.
The when we hear about we've never done this before and estimating is a waste of time, think about the title of the post.
Everything's a Remix
Other than inventing new physics all software development has been done in some form or another before. The only true original thing in the universe is the Big Bang. Everything else is derived from something that came before.
Now we may not know about this thing in the past, but that's a different story. It as done before in some form, but we didn't realize it. There are endless examples of copying ideas from the past is thinking they are innovative, new and break through. The iPad and all laptops came from Allan Kay's 1972 paper, "A personal computer for childern of all ages." Even how the touch screen on the iPhone works was done before Apple announced it as the biggest breakthrough in the history of computing.
In our formal defense acquisition paradigm there are many programs that are research. This looks like this flow. Making estiimates about the effort and duration is difficult, so blocks of money are provided to find out. But these are not product production or systems development processes. The Systems Design and Development (SDD) is between MS-B and MS-C. We don't confuse exploring from developing. Want to explore work on a DARPA program. Want to develop, work in post-MS-B and know somethiong about what came before.
The Pre-milestone A works is to identify what capabilities will be needed in the final product. The DARPA programs I work are even further to the left of Milestone A.
On the other end of the spectrum from this formal process, a collection of sticky notes on the wall could have similar flow of maturity. But the principles are still the same.
So How To Estimate in the Presence of We've Never Done This Before
Here's a critical concept - we can't introduce anything new until we're fluent in the language of our domain, and we do that through emulation.† This means for us to move forward we have to have done something like this in the past. So if we haven't done something like this in the past, don't know anyone who has, or can't find an example of it being done, we will have little success being innovative. As well, we will hopelessly fail in trying to estimate the cost, schedule, and probability of delivering capabilities. In other words we'll fail and blame it on the estimating process and assume that we'll be successful if we stop estimating.
So stop thinking about we can't know what we don't know and start thinking someone has done this before and we just need to find that someone, somewhere, something. Nobody starts out being original, we need copying to get started. Once copied, transformation is the next step. With the copy we can estimate size and effort. We can now transform it into something that is better, and since we now know about the thing we copied, we have a reference class. Yes that famous Reference Class Forecasting used by all mature estimating shops. With the copy and it's transformed item, we can them combine ideas into something new. The Alto from Xerox and then the Xerox Star for executives, was the basis of the Lisa and Mac.
You can estimate almost anything, and every software system if you do some home work and suspend the belief it can't be done. WHY? because it's not your money, and those providing you the money have an interest in several things about their money - what will it cost, when will you be done, and using the revenue side of the balanced sheet, when will they break even on the exchange of money for value? This is the principle of every for profit business on the planet. The not-for-profits have to pay the electric bill as well, as do the non-profits. So everyone, everywhere needs to know the cost of that value they asked us top produce BEFORE we've spent all their money and ran out of time to reach the target market for that pesky break even equation.
Anyone tells you otherwise is not in the business of business, but just on the expense side and that means not on the decision making side either, just labor doing what they're told to do - which is a noble profession, but unlikely to influence how decisions are made.
The notion of decision rights is the basis of governance. When you hear about doing or not doing something in the absence of who needs this information, ask who needs this information and is it your decision right to decide to fulfill or not fulfill the request for that information? As my colleague, retired NASA Cost Director, says follow the money, that's where you find the decider.
† Everything is a remix, Part 3, Kirby Furgeson.
There was a question posed on LinkedIn
How do you get Project Manager understand the importance of Earned Value Management?
It's been shown over and again when we make EV a compliance process and hold managers accountable for compliance it is a necessary condition for success, but far from sufficient.
Until EVM starts providing actionable information to the program management staff in the form of "leading indicators" it will always be one of those compliance processes. This information must reveal where in the program, technical and programmatic changes can be made, the testable outcomes of those changes on program performance, and the impacts on cost and schedule forecasts and the EAC against the BAC. Starting at the program level, but going down to the Control Account and Work Package.
In other words...
Nice data there in your Formats 1-5 and really nice IMS, what should I do next?
EV numbers themselves are lagging indicators, non-statistically adjusted, non risk adjusted, not connected to the effectiveness and performance measures of the project - unless enlightened users do so - and not connected to the needed capabilities of the customer as delivered by the program. There is no DID for the IMP, so Systems Engineering rarely flows down MOEs and MOPs - except where they do, because of the knowledge of the power of this approach.
Next is a real problem. 748-C has a loop hole big enough to fly a 787 through. In section 3.8 "Earned value is a direct measurement of the quantity of work accomplished. The quality and technical content of work performed is controlled by other processes." Measuring quantity is a construction centric view of producing value. Not a value centric view. Our defense, space, biopharma, software intensive programs that apply EVM are not pouring concrete or welding pipe. (I used to work in that domain as a SW developers on piping design systems). If we see EV as measuring quantity we ignore all the concerns Paul Solomon has brought forward, starting with the missing Technical Performance Measures for the product.
Sure we have exit criteria for the Work Packages. But these need to trace vertically through the IMP to the Capability to accomplish the mission or provide a solution to the business need. In other do something with the resulting product or service. It has to be the right thing of course. But EV - as stated in 748 - only measures to quantity of parts needed to delivery a capability, not the ability of the system to fulfill the needed capabilities. This by the way is essentially a Systems Engineering issue. With the missing IMP, most of the motivation for connecting the dots between EV and mission success is simply not there. It's back to the immutable principle
We don't know what done looks like in units of measure meaningful to the decision makers
"Earning value" means assessing the efficacy of the BCWS. This means assessing the Technical Performance Measures of the work be delivered. Many implementations assign TPMs and QBD's this work. But this is not called out in 748-C nor any formal EV guidance. It is called out in the DAG and the old SEMP DID. No SEMP DID is in place.
So as EV practitioners we need to make the business case of EVM, not just the compliance case. When we add "leading indicators" that are statistically sound (Eric Druker has written about this as have others, including me), forecasting of EAC based on ARIMA processes rather than our linear, non-statistical, non risk adjusted CPI/SPI measures. As well those measure "roll up" the variances of all the past time series data, just like Darrell Huff tells us to do in "How To Lie With Statistics".
All the data is available in the EV engine and in the CR at PARCA for DOD jobs. What's next is to start using EVM in the same way supply chain managers, quality control managers, systems engineers, and every design and manufacturing engineer does - compare statistically adjusted performance against the probabilistic plan to see leading indicators emerge of where we are going to go into the ditch in the future, show the PM, that corrective action "before the fact" rather than after.
This will be a topic discussed at EVM World and ISCEA conferences coming soon.
Earned Value measures the production of tangible outcomes from work efforts on a project using Physical Percent Complete. This P%C is calculated used Quantifiable Backup Data shows what technical or operation goals must be met on a planned date in order to claim some Percent Complete. Measures of this physical percent are NOT how much money you spent or how much time you took to do the work.
They are only measured in unit of tangible technical progress to plan. This tangible technical progress is defined BEFORE the work starts. This progress goal is derived from the time phased Measures of Effectiveness and Measures of Performance goals of the project.
These MOE, MOP, KPP, and TPMs are defined below and their measures - in units meaningful to the decision makers - established before work starts, put on baseline and used to compare progress to plan (time phased) to produce that Physical Percent Complete in the Earned Value (BCWP) formula in the first chart.
Unless we know what capabilities we need at the end of the project, and know those somewhere near the beginning of the project, have a Plan to produce those capabilities at the needed time for an anticipated budget, we're going to be disappointed in the result.
Identifying the needed capabilities is the first and most important practice for the success of any project. To achieve the project's objectives or a particular end state, we need to define these capabilities through Measures of Effectiveness. We can do this through scenarios from the customer's point of view. The effectiveness measures describe how well the results from the project enable a business process or fulfill a strategic mission of the business - in this case enforcing the will of the emperor. These measures must be in units meaningful to the decision makers. In the case Darth Vadar.
When we let those needed capabilities emerge we have likely disconnected the products of the project from the mission or vision of the business. Requirements can emerge to fulfill the capabilities. But the capabilities need to be steady and focused on mission outcome, otherwise we are just on a spend plan. We'll have no way to know what done looks like until the money runs out.
Here's an example of an increasing maturity of the capabilities for a health insurance provider network ERP system.
This approach is not usually found in traditional or agile projects. These approaches start with requirements, stories, features - all related to the technical aspects of the project. A Master Plan is developed that shows the needed capabilities (above), the order of delivery of these capabilities, and the dependencies between these capabilities.
The increasing maturity of the project's outcomes is then stated in this map. Technical requirements implement the capabilities, so they need a plan - schedule actually - as well. Small incremental capability delivery is the best approach. This is obvious and the basis of agile, but not always planned that way.
This plan contains all the steps (delivered capabilities) needed to increase the maturity of the of the project from the start. This does not mean we know how these capabilities are going to be delivered, but it does show what DONE looks like at the end of the project. In this example Phase 1 - Go Live.
This approach does not define any specific development process - agile or traditional. It is the basis of success for any development process.
Capabilities Are the Vehicle For Delivering Value
The Capabilities and their Measures of Effective need to remain stable over the life of the project. Without this reached DONE with any sense of budget and schedule will not only be difficult, you will be rubber banding the baseline used to measure your performance. You'll have turn a project into an operational activity. Projects have defined outcomes, defined periods of performance and associated budget. Operations can be fixed budget, open ended periods of performance, and emerging outcomes. But reaching a defined description of DONE becomes much more difficult.
The notion of successfully managing a project starts - as always - with the 5 Immutable Principles of project success.
So What's the Point?
If these principles are not in place on your project, it'll be a disappointment. So in the end any successful project management process must answer the following:
By the way, the common objections by some in the agile community to the notion of on-budget, on-schedule as a measure of project performance is seriously misinformed. Those paying for the project not only want to know these, they must know this. It's not the developers money. It's the customers money. They must know the final cost and the date when the project is done. They must also - of course - know that what is being produced delivers the needed capabilities. But having those capabilities late is not good. Spending much more than budgeted is not good. All three attributes are needed.
These numbers by the way are probabilistic. Every number on a project is drawn from a statistical process to produce a probabilistic outcome. The notion that NOT forecasting the future performance of cost, schedule, and technical outcomes can't be done is simply nonsense. Of course it can be done. You just have to remember the things you learned in your high school statistics class. Plus a few other things about probability distributions and confidence levels.
There have been several posts and discussions on LinkedIn about applying Earned Value Management to projects. This coming Thursday, I'll be talking about applying Agile software development processes to Earned Value Management project, but in order to do that we need Earned Value to be applied first. So let's look at how to install EVM on your project.
Let's start with some basics and get rid of the myths of EVM
So now let's look at setting up an Earned Value Management System at the highest level - the principles level.1. What are we building?
3. Master Plan and Schedule
4. Resouce Plan
5. Cost collection
6. Baseline Management
So Now We've Got an EVMS, What Do We Do?
When we have our Performance Measurement Baseline (PMB), the Integrated Master Schedule (IMS) with its seqeunce of Work Packages and work activities, with resources assigned to do the work and the budget to fund that work, we can start to measure progress to plan.
At periodic times - usually weekly, possibly bi-monthly, but for sure every month, we can assess the physical percent complete on the work. This means measuring tangible progress to a pre-planned set of attributes. With this measurement comes the physical percent complete. With the physical percent complete we can calculate the Earned Value.
With this measure we can determine where we are along the way to complete, how much we need to improve out performance, and what corrective actions we need to take to get back on schedule, or stay on schedule.
There's a post on a Deltek implementation partner site about applying Earned Value. It has some good advice, but the premise of the starting point needs to be addressed.
Before going on to suggest things to do on a project using EVM, it's best to have one of those don't do stupid things on purpose discussions. If the customer or potential customer is doing stupid things on purpose, it's important to stop those first, before attempting to make any other improvements. Once that is done, the core paradigm of Earned Value Management is very simple and obvious ...
Turning the Subjective into the Objective
This phrase is not mine, it is a colleague's who said it while we were walking down the hallway of a client site, who is introducing EVM to a $1.9B weapons program that has never had EVM before, but is now being mandated to do so by Congress!!!
So Here's the Introduction to What Not To Do and the Simple Fix
Let's start with a picture of where to start. The are 32 Guidelines in ANSI-748-C. 11 are needed for he success of any project, in any domain. If you aren't doing these 11 in some way, the probability of success for your project is low. This is the case of all project management and product development processes, These are like the 5 Immutable Principles - they have to be there in some way. Here's the 32 from ANSI-748-C and the 11 are colored and listed below.
So Let's See How the 11 Can Address Each Of These Issues?
Here's their list of reasons why Earned Value Management is not readily adopted by contractors.
So What Does All This Mean?
It means applying the 11 Guidelines shown above to your program is the same as being a credible program manager. If you're not doing these 11, you're probably not doing your job as a Program Manager on an EVM program. Think about that. You've been assigned to look after $20,000,000.00 of the governments money. Would you actually use some the phrases above when asked why you're doing a poor job of managing the program? Not for long I suspect.
One Final Comment
Earned Value Management is one of several enabling technologies to turn the subjective into the objective. But it is only a necessary condition for success. It is far from a sufficient condition for success. You need an Integrated Performance Measurement Baseline with these elements
Ray Stratton's news letter just arrived. There he speaks to the "44 Day Rule," for the duration of work activities. Some places use that for the duration of a Work Package. Other places use that as the duration of a work activity contained in the Work Package.
Here's another idea.
You need to answer the question - How long are you willing to wait before you find out you are late?
The answer to this question, then drives two things:
With the answers to these two questions, the duration of a work activity should now be self evident. As well as the point in time needed to take corrective action. This corretive must have enough time to have the lateness be fixed. So some arbitrary durations cannot be used. You need to know how the work is being performed.
But a simple (and simple minded) rule assumes the work is being performed linearly. So yuo can answer the question by saying, what is the capacity for work that must be applied to fix the activity when it gets behond schedule?
By the way, when this corrective action is applied, it will cost more money, unless you change the scope or improve the effectiveness of the work to cover the extra work needed to correct the problem.
This is why short duration activties are best. The Agile folks know this, maybe not for the same reason. But short duration tasks - the 44 day rule - force the assessment of work so corrective actions can take place.
How many legs does a dog have if you call the tail a leg? Four. Calling a tail a leg doesn't make it a leg
- Abraham Lincoln
The notion that we can rename things and then assume they are the same doesn't work, just like Lincoln said. The notion of measuring progress with the passage of time and calling it Earned Value, doesn't mean we're doing Earned Value.
Doing Agile without doing Scrum, XP, Crystal, or DSDM is not doing agile.
The LinkedIn forum The Project Manager Network - #1 Group for Project Managers which I sometimes wonder if that is actually the case, has a conversation going on around project failure. A Master's student asked about project failure modes. That led to the use of Earned Value. One participants made some pretty boneheaded statement around EV:
5% (2 hrs)for applying the border and title
20% (8 hrs) for defining the area - boundaries and overall dimensions
30% (12 hrs) for showing all the equipment
5% (2 hrs) for the drawing being checked
10% (4 hrs) for it being Issued for Approval
15% (6 hrs) for it being Issued for Bid
15% (6 hrs) for it being Issued for Construction.
100% = 40 hrs.
So, silly me, I responded that determining physical percent complete was a straight forward process once you have Quantifiable Backup Data or measures of Physical Percent Complete, or even better, Technical Performance Measures. Let's look at each concept
Then the final come back from the OP.
If you're doing a drawing and it takes 40 hours to do the drawing and no two drawings are alike, how do you determine what percentage complete you are? When estimates of the work to be completed are put together, they will estimate X drawings at 40 hrs/drawing. They will have different standards by discipline. Will a person ever report 41% complete on a drawing? Most likely not. It will be shown as 40%. Collectively all the drawings may be 41%, but not individual. No matter how you look at it, it is still an estimate.
First measuring progress to plan by the number of hours consumed is called Level of Effort. Second LOE is a very poor measure of anything other than the passage of time and consumption of resources.
In the end the poster was insistent that ANSI-748-B is just a guide, and doing EV was a local issue. No doubt that is true in his domain. But as a practitioner of EVM using ANSI-748-B for programs subject to DCMA Validation, as well as engagements with the DOD office that own the Earned Value Management processes for the DOD, I'd be laughed out of the room with an approach like that.
For a summary Earned Value in a Nutshell using a similar drawing example.
So Here's The Way TO Do Earned Value
That's all there is - essentially - for doing Earned Value Management. Of course if you read ANSI-748-B, the NDIA Earned Value Management Intent Guide, and the DCMA Intent Guide, there is a lot more.
You Can't Make Stuff Up
OK, you can. But if you do, it makes having a conversation about EV much harder.
When I encounter a "subject matter expert," I've now learned to assess if this person is a "knowledge holder," or a "learning individual." This is what informed that "learning." This came to my from a friend and neighbor who is a graduate of the University of Chicago in economics and spoke at this conference.
In our business - project and program management - there are lots of "knowledge" providers, assessors of knowledge, frameworks for measuring our knowledge. What is needed now is how to learn to make better decisions about our projects. This is not going to happen without a decision making framework for the core principles, practices, and processes for project management. The current Bodies of Knowledge, self-proclaimed providers of better BOKs and assessment tools, authors (I'm one of them), and experts in the field, all need to reassess what we are doing to increase the Learning abilities, beyond the just the possesion of knowledge and the dispensing of this knowledge to others.
How can we actually learn to improve our probability of succes. The Decision Scientist for project success - as suggested here - needs to have art, science, and the scale to provide actionable information for the decision makers for the project.
We have most of the data we need, what we don't do is make decisions on this data, or even know how to make decisions with this data. We report it, but don't use it to make decisions. Numbers matter, but leadership needs to use the numbers.
Forecasting involves making projections about future performance on the basis of historical and current data.
The chart below is a synthetic (same as fake) Cost Performance Index at the program level 2½ years. The area on the right is the forecast of possible values of the CPI given the historical behaviour, with tuning parameters and smoothing functions.
The next step is to use real program performance data over some realistic period of time - say 4 year - and use H-W (or a similar forecasting tool), to confirm that a less that complete assessment, say take 2 years of data, forecast a 6 month value, and see of the actual 2 year, 6 month value come out close the to the forecast.
This is all done in R, with very simple commands, once the time series for the CPI is formatted and partitioned properly.
Shim Marom has a nice series of posts about Earned Value. It this post he provides the calculation of Earned Value, or Budgeted Cost for Work Performed.
BCWP = BCWS x Percent Complete
Shim, then mentions that many times it is difficult to determine Percent Complete.
Here's where to start
This approach is independent of applying Earned Value. It should be used for all project work.
Below are the other measures of performance.
So when asked what is the percetn complete, take out the KPP's and TPM's and look at your delivered product to see if it is compliant with the planned values? Do ot weigh the proper amount for this point in the project, Does it go fast enough? Does it have the planned reliability? Can it process the transactions properly at the right rate. Does it provide the planned storage capacity.
Does it do what you planned it to do on the planned day you said it would, for the planned budget?
That's how you determine your physical percent complete. Here's a way to lay out those high level measures that will show where you need to assign these measures
Came across (through Google agent) a nice short description of what is Earned Value. It's from a British construction firm. The phrasing is a bit off for our domain, so I'd like to clean it up. Let's start with a simple and s imple minded project.
Earned Value has three components for our project:
Notice the at the time of assessment statement. This is critical. If we plan to spend $1,000 as the total budget for the work over a 4 week period, this is the Planned Value, the Budgeted Cost for Work Scheduled or BCWS. This is the budget to produce our outcome. When we assess the performance of our efforts, we do that at a point in time. This point has a cummualtive budget and we need to assess our cummulative performance, and our cummulative actual costs all at the same time.
At the half way point (week 2), our plan says we should have spent 50% of our budgeted cost (BCWS = $500 at this point in time). This is naive of course, but makes our calculations simple. This means that after 2 weeks of work, we planned to have spent $500.
We assume - again naively - that over the course of the 4 week, the ACTUAL value of the product - the drawings - we are producing is growing linearly. So at the 2 week point, our product should be worth 50% of the Planned Cost or $500. This is the Earned Value. We essentially earn our budget in this simple minded example.
So now the 2 weeks has come, it's Friday of the second week of our 4 week deliverable. We planned to spend $500 and to earn $500 of value. At our status meeting, here's where we are:
During those two weeks, we spent $50 per drawing on the first 7 drawings, just like we planned. But on the 8th drawing we spent $70 because it was harder. Which is also the reason we didn't get to finish the 9th and 10th drawings. So for the first 7 drawings that were produced we spent (7 x $50) and for the 8th drawing we spent $70 for a total of $420 for 8 drawings.
Now we can compute some indicies
So let's do some claculations using Earned Value formula.
That's all interesting, but the real value in Earned Value is the ability to forecast the future. The first forecast calcualtion is our Estimate At Completion. This our estaimate of how much this project is going to cost when we are do, if we don't change anything
But that's not the real problem, let's see how late we are going to be.
Here's the quick summary:
So what can we do?
This is what Earned Value can tell us. No other method can do this.
This is the Value of Earned Value