Herding Cats

DoD EVM Implementation

Paul Solomon has posted a link to the DoD Earned Value Management: Performance, Oversight, and Management Report to Congress. 

Anyone interested in the least just "favorite" Paul's site and subscribe to his posts. Paul's book Performance Based Earned Value, is heavily yellow highlighted and "tabbed" in my backpack everywhere I go. You can't really say you're an EV junkie unless you have his book.

GAO Identifies Need for EVM Improvement

Bill Mathis on LinkedIn Earned Value Management and Paul Solomon both posted news of the GAO released a report on the need to improve the application of Earned Value to manage major acquisitions. Defense was the only agency with "all black," except for training.

From the introduction:

Given the size and significance of the government’s investment in IT, it is important that projects be managed effectively to ensure that public resources are wisely invested. Effectively managing projects entails, among other things, pulling together essential cost, schedule, and technical information in a meaningful, coherent fashion so that managers have an accurate view of the program’s development status. Without meaningful and coherent cost and schedule information, program managers can have a distorted view of a program’s status and risks.

Note that any solution to the "problem" of project management needs to "pull together cost, schedule, and technical (performance) information"

Try that with your email based, IM, twitter, Face Book, social networking PM 2.0 project management system on anything other than a toy project and see how fast you can get into the ditch.

Agile Earned Value

Paul Solomon has an article "Agile Earned Value and the Technical Baseline," in this months SoftwareTech, a publication of The Data & Analysis Center for Software. DACS is a US Department of Defense Information Analysis Center (IAC). This site and the journal is a source of software development news and processes application to a variety of domains and contexts. DoD is one of the world's largest consumers of software - COTS and purpose built.

DCAS is the holder of the Gold Practices content. These documents are a good starting point for anyone looking for guidance on processes for developing and managing mission critical software systems.

This article, like all of Paul's articles provides insight into the subtle and sometimes complex world of Earned Value. In this case in the presence of Earned Value. 

A core problem in commercial projects is "feature deferral." This means "we'll not be able to give you this feature in the current iteration (if you're using Agile), but will get to it in the next iteration." The question becomes:

How is this deferral reflected in the reporting of Physical Percent Complete?

Paul has a suggestion. Even in the absence of the discipline of Earned Value, this is always an issue. So if you're in the enterprise software development business, subscribe to SoftwareTech and read how software intensive systems are built in the presence of government contracting. Then see if some of the methods to increase the probability of success can be applied to your non-EVM project controls method.

On our software intensive programs (flight avionics), Work Package performance and Rolling Waves are a nice fit for Scrum. Maintaining the PMB (Performance Measurement Baseline) can be done with Planning Packages for out year rolling waves - IF the level of detail needed to define the content of the Planning Packages is carefully considered before placing them on baseline.

This is a constant "tuning" effort and the role of our staff as Program Planning and Controls.

As well there is an article of "Dispelling Myths about Earned Value Management," that must be read by everyone claiming to have a simpler or alternative approach.

Simplifying Earned Value?

There are many discussions floating around about how and why to simplify the application of Earned Value. Some are credible some are not.

There are several places to look for the discussion of Earned Value. LinkedIn has a forum for EV.

One of the best is Paul Solomon's book and site. The current post describes an assessment of several government agencies.

Earned Value as a principle is applicable to any project. The 32 criteria of ANSI-748B need not be applied in their entirety outside a federal government program. But there are a minimal set of criteria needed for the success of EV. Here's a previous suggestion on those minimum criteria.

There are subtleties in reducing the EV processes. So when you "simplify" you must also maintain the discipline needed to produce Value from Earned Value:

• Maintaining the integrity of the baseline is critical - "rubber banding" the baseline is common practice in troubled programs or projects. When you overrun a work package, you've over run the work package. You're "off baseline." Report it and move one. Get a "get to green plan" for the remaining work.
• Maintaining the funding integrity is critical for government projects. This should also be critical for commercial project. This is not always the case. Without integrity of the funding sources future forecasting based on past decision is difficult.
• Performance management at the Work Package level for BCWS is as low as you want to go on a non-trivial project. Lower - at the task level - means change requests to the baseline occur all the time. The notion proffered by a few that managing at the task level for 5,000 tasks gives the program manager "control" is complete nonsense. It creates a nightmare for change control and traceability. Once change starts, the integrity of the baseline is lost and you're ability to forecast future performance with it.

Change to task sequencing should be a work package issue, not an EV baseline issue. Many fail to realize EV is a "time phased" measurement. If you don't start the work on the time you said you were, then you "off baseline." Move that up to the next level. Keep the natural variances inside the Work Package. Don't let them leak.

Buy Paul's book, read it, mark it up. Read all the materials on Paul's site. Use EV on software projects. Increase your probability of success.

Using Earned Value in a Changing Scope Environment

The basic principle of Earned Value starts with the establishment of a Performance Measurement Baseline (PMB) and the measurement of progress against it. The Budget At Completion is the total target for the budgeted cost for the project when it completes.

Everywhere we turn we hear about projects that are over budget, behind schedule, and are failing to deliver the promises made when the project started. This is the mantra of most IT projects. Business requirements change. Technology requirements change. Everything changes.

But for the owners of the funding source, change is not something they like to hear about when it comes to budget increases and schedule delays. The agilest will tell us as well, that the system be produced must be ready to be put into production - or at least be complete at some level of maturity - at all times. This way the customer - the one with the check book - can get something for their money at all times.

And here's the rub with most IT projects. Most projects in general. The customer has a budget. The customer has specific needs that must be there to fulfill the business case. Those needs need to appear on or before a specific time to start earned their keep for the business.

Being told the requirements will emerge along with the code that we're developing is of course little comfort to the holder of the check book, when what was planned to be delivered turns out to be less than what was delivered - even if the budget and schedule numbers are met. Most business people I've met kinda want to know when it will be done, will I be within my budget, and can I get what I need - at a minimum - to fulfill the promises I made to the board of directors.

We showed up on time and on budget, but the features you wanted - or even the features you needed for the business to work - aren't there.

So Let's Look at Some Environmental Impacts That Drive Scope Changes

• The original baseline plan left some things out. 

This is a common occurrence. Happens all the time, even in very formal programs. This should actually be expected and therefore a process is needed for handling it when it does.

The first step is to establish a time when "new" items can be added. This can be an open widow, on the rolling wave boundaries, through a formal process of change management. Something that is predictable.

These changes need to be connected to the CURRENT scope of the project. They need to have a reason. The best approach is to connect the change to the business case, the Concept of Operations, or the Mission Statement. But if they expand the scope of the project, there must be another process. Otherwise the scope grows with bound and the value of EV becomes worthless.

• We've got a new way of doing the work

This is similar to the item above. No change in scope has occurred but we've discovered a better way of doing things. This of course will impact the BCWS and how we measure physical percent complete. Like the additions to the existing scope - usually more details and slight changes in the external behavior of a feature.

• The original estimates for budget and duration turn out to need "fixing"

We always learn as the project progresses from left to right. We must use this knowledge to improve the schedule baseline. This is a motivation for rolling waves. The Planning Packages in the future rolling waves have less fidelity at the task or work package level, than the current rolling wave. This is why they are called Planning Packages.

• There is real scope changes to the project from the customer

Additional scope - not just improvements in the current scope - must go through a formal change management process. This process is not, nor should ever be, an impediment to improving the project. It is a red herring from agilest to suggest that change control is heavy weight. But if we're changing the scope of the project without understanding the impact of those changes, the business value of those changes, the risk impacts of these changes, then we're no longer managing the project. We're just spending the customers money.

• The work we planned is taking longer than we planned and it is costing more than we planned

These activities can not be accommodated in Earned Value. If we're late, we're late. If we're over budget, we're over budget. The Performance Measurement Baseline must show that. Rebaselining or "rubber banding" the baseline is the beginning of the end for  using Earned Value effectively.

For Earned Value to produce value for the project, baseline integrity is a must.The only reason to deploy EV is to identify variances in cost and schedule from the baseline. Fiddling with the baseline negates all the beneficial outcomes of EV.

So there is a simple and hard rule, "no messing with the baseline in the absence of formal change control." No changes to the baseline can be made to fix poor performance - period.

So Now What's the Real Problem?

• Maintaining the Performance Measurement Baseline in the presence of pressures for scope changes
• A change control system that is not burdensome, but at the same time maintains the integrity of the PMB

And in the end...

• How to actually use Earned Value in the presence of changing scope.

Beyond the Novice and the Advanced Beginner

I've been chatting back and forth on email, blogs, and forums around several things of interest to me this week. Some conversations were fruitful some were not. These topics include: Earned Value, Project Management, and the competencies needed to successfully manage project.

On the way home from Montgomery Alabama and the Air Force IT Conference at Maxwell Air Force Base, I come to understand that humans have a proclivity for fuzzy thinking and logical inconsistency.

This should come as no surprise when reading blogs and forum messages on almost any topic. The topic of earned value, competency measurement of project managers, success rates of government contracts, the effectiveness of agile software development, the interpretation of CMMI–DEV V1.2 are a few discussions that easily succumb to fuzzy thinking.

Fuzzy thinking and logical inconsistency is not the same as differences of opinion, or even differences in fact. The level of fuzziness and logical inconsistency usually follows the level of maturity of the speaker. According to Dreyfus gaining expertise in any subject has 5 levels

• Novice
• Advanced Beginner
• Competent
• Proficient
• Expert

Let's look at Earned Value Management as an example.

• Novice – a set of context free rules are acquired. These rules ignore the context sensitive features of Earned Value, such as the stability of the Performance Measurement Baseline, the management of Management Reserve, the flow of funds from the Total Allocated Budget down to Control Accounts and Work Packages.
• Advanced Beginner – through practical experience, the novice begins to recognize situations where the rote rules cannot be described in objective, context free terms. Size implicit size of a Work Package and the associated budget (BCWS) compared to the implicit or explicit risk determines the method of physical performance measurement.
• Competent – superimposing an overall strategy on the development of the Performance Measurement Baseline (PMB) on top of the Advanced Beginner and Novice's understanding of the principles. Does this PMB describe the inherent risk of the program and the mitigation activities needed to handle that risk? What impacts on physical percent complete arise from this risk?
• Proficient – in the previous levels of maturity decisions are made using deliberative and conscious choices. The proficient user of Earned Value makes decisions, as well, on the "feel" of the data. "These numbers don't feel right for our current phase of the program." "This SPI should be decreasing after we have passed Preliminary Design Review, and since they're not, there is some other issue we have to discover." The instinctive reactions to information come of similar experiences in similar situations. There is a spontaneous response comes from an "understanding" of the plan or strategy.
• Expert – no longer sees Earned Value as a data gathering and reporting process. The PP&C expert uses Earned Value as a paradigm to "manage" the program. The problem solving and decision making processes become integrated with the information produced by Earned Value. These "actionable information" processes are embedded in the daily rhythm of the program.

Now To The Problem

When there is a suggestion to "simplify" something or replace it with an alternative, or even to redefine formulas and words, say the application of Earned Value, the proficient or expert will respond with "OK, here's the minimal set of activities within the 32 Criteria of ANSI/EIA–748B that can be used and still call it Earned Value." The Novice might say, "I've got a better idea, let's copy ACWP to BCWP and we'll be doing Earned Value." (This is a nonsense approach of course, but one I have actually seen in practice on a $130M ERP deployment.)

The Advance Beginner might say "Let's move our unused Work Package funds back to Management Reserve (MR) to be used latter," or "let's extend duration of this Work Package and re–spread the BCWS." This is called "rubber banding the baseline."

When there is a discussion on a forum or a blog about applying EV and response come back, "why are you making this so complex?" the first response might be "well to get actual value from Earned Value you need to apply it at the Competent level or above.

Simplification "tricks" are just that "tricks." To move from the Novice to Advanced Beginner requires careful step–by–step deployment. Seeking advice and feedback from more mature sources. These don't have to be people, but working examples of how EV is applied at the expert or at least the proficient level. The best source of expert and proficient application of Earned Value is to look at the Earned Value Management System Descriptions that can be found on the web. These are DCMA (Defense Contract Management Agency) assessed processes for programs subject to Earned Value Management (EVM).

Not every use of EV is guided by DCMA, but every use of EV should – I'd say must – be guided by ANSI/EIA–748B and the NDIA Earned Value Management Intent Guide (EVMIG). In the absence of this framework, you might call what you're doing Earned Value – using BCWS, ACWP, and computing or "making up" BCWP – but it "ain't so."

Of the 32 Criteria from 748B there 10 or 11 "must do's" for anyone to say "we're doing EV." They are:

Organization

1. Define the authorized work elements for the program. A work breakdown structure (WBS), tailored for effective internal management control, is commonly used in this process.

2. Identify the program organizational structure including the major subcontractors responsible for accomplishing the authorized work, and define the organizational elements in which work will be planned and controlled.

Planning and Scheduling

6. Schedule the authorized work in a manner which describes the sequence of work and identifies significant task interdependencies required to meet the requirements of the program.

7. Identify physical products, milestones, technical performance goals, or other indicators that will be used to measure progress.

8. Establish and maintain a time-phased budget baseline, at the control account level, against which program performance can be measured. Budget for far-term efforts may be held in higher level accounts until an appropriate time for allocation at the control account level. Initial budgets established for performance measurement will be based on either internal management goals or the external customer negotiated target cost including estimates for authorized but undefinitized work. On government contracts, if an over target baseline is used for performance measurement reporting purposes, prior notification must be provided to the customer.

9. Establish budgets for authorized work with identification of significant cost elements (labor, material, etc.) as needed for internal management and for control of subcontractors.

Accounting Consideration

16. Record direct costs in a manner consistent with the budgets in a formal system controlled by the general books of account.

Analysis and Management Reports

22. At least on a monthly basis, generate the following information at the control account and other levels as necessary for management control using actual cost data from, or reconcilable with, the accounting system: (1) Comparison of the amount of planned budget and the amount of budget earned for work accomplished. This comparison provides the schedule variance. (2) Comparison of the amount of the budget earned the actual (applied where appropriate) direct costs for the same.

27. Develop revised estimates of cost at completion based on performance to date, commitment values for material, and estimates of future conditions. Compare this information with the performance measurement baseline to identify variances at completion important to company management and any applicable customer reporting requirements including statements of funding requirements.

Revisions and Data Maintenance

28. Incorporate authorized changes in a timely manner, recording the effects of such changes in budgets and schedules. In the directed effort prior to negotiation of a change, base such revisions on the amount estimated and budgeted to the program organizations.

Is EV for Experts Only?

No, but it's not for Novices either. Some level of knowledge and experience is needed for success. The advanced beginner will soon run into trouble once the reporting process starts to be "cooked" in any way. No amount is "simple" process can seem to prevent this even in sophisticated environments. Constant vigilance is needed to assure the numbers have meaning.

There are dozens of ways to make the number look better than they are. Some level of competency is needed to get beyond "reporting yesterday's weather," and start using EV to forecast the future. This requires experience and skill and most all the INTENT of Earned Value as described in the Earned Value Management Intent Guide and some higher level overview from Neil Albert on Earned Value Management Application, Guidance, and Education.

If you're doing all of these, you're doing Earned Value Management. If you're doing some but not the others, you're NOT doing Earned Value. You may be doing something else, but it's not Earned Value. The level of maturity needed to perform these 10 activities is beyond the Novice. Possibly the Advanced Beginner can take a stab at it. But competency is needed to use Earned Value Management in a project management context. Beyond just reporting. Managing the project.

So in the end the usefulness of Earned Value starts with the Competent user and gets better as you move to Expert. But even programs looked after by experts have troubles. All programs have troubles. It's simply the nature of complex, high risk programs. EV must become of tool to provide actionable information to the program manager (as one of our tag lines suggest), otherwise it's just a bunch of meaningless numbers.

Impediments to Successful Earned Value Management

There is so much blather about Earned Value, I've run out of energy trying to provide corrections and switched to have other people do my work for me.

Here's a nice paper that surveys many of the reasons EV fails to fulfill it's promise.

Earned Value Analysis - Why It Doesn't Work

The answers have nothing to do with Earned Value and everything to do with the competencies of the project, business, and technical management of the project or program.

The organization sponsoring the conference this paper is from has a nice library is important topics in cost, schedule,  risk, and earned value management. Worth a read and a shortcut for reference.

Fidelity of Earned Value Planning

There is a concept that detailed planning is part of Earned Value Management. Some have described projects with 100's of thousands of tasks - 200,000 in one example. This of course is a fundamentally wrong approach to developing the Performance Measurement Baseline for an Earned Value Management based project.
    Here's why.

• In the MIL-STD-881A and the related Earned Value guidance, the EV measures are taken at Level 3 of the WBS - NOT the detailed tasks. The WBS elements before are still not the tasks, since the WBS contains the Product Breakdown Structure (PBS) and the supporting services needed to produce that product. Words like "design" are never at Level 3 in the WBS.
• These detailed tasks are part of a Work Package. This Work Package is where the EV is collected.These tasks usually fit inside some Work Packages time boundaries. Many Earned Value shops limit the duration of a Work Package to one accounting period, 60 calendar days, 45 work days or something like that. The reason of course is the answer the question "how long are we willing to wait before we find out we're late?"
• The Tasks in the Work Package are used to establish the physical percent complete for the Work Package. Usually the measure is 0% or 100% for each task. Either the task is done or it is not. The 0/100 approach allows you to "roll up" each task in MSFT Project and compute a total percent complete for the Work Package. Or you can do a weighted calculation in MSFT Project or Excel before moving this information to the Cost Accounting system (SAP, Cost View, Safran MPM)
• These Work Packages are sequenced in an order that moves the project forward. The collection of Work Packages in the space and defense business is called the Integrated Master Schedule. This sequence is usually "nose to tail," "finish to start" for the Work Package. The reason here is any downstream work that starts with partially completed predecessor work is called "rework." Try as hard as you can to not have leads and lags between Work Packages. Some US DoD procurements guideline prohibit leads and lags or limit them to 5 days. 5 days is the same as 0 days. Break the Work Packages up to prevent this.

So What Does This Mean in Practice?
    First, the rolling waves are the basis of every credible planning process, whether it is Earned Value based or not. The active rolling wave (the one we're working now) is usually 6 to 9 months for the programs we're on. This is around 20% of the total period of performance. The Planning Packages for the future Rolling Waves have less detail than the current rolling.
    But these Planning Packages need a credibly estimated budgeted cost and duration, so they may be vague on the details, but they describe the total budget of the program

Total Authorized Budget = Current Rolling Wave + Sum(Planning Packages)

As an aside here's the structure of the budget flow, used in US DoD contracting for Earned Value Management:

All of this means:

• Detail plan only the stuff you know enough about. Making detailed plans for work a year from now at the same granularity you are in the currently rolling wave is fantasy.
• Don't waste effort on these Planning Packages until they come closer to starting. have credible estimates for future work, but have this work be simple durations and work effort style planning. Any competent engineer shoudl be able to say with a reasonable percent how much effort it will take to do something. If they honestly don't know, go find someone who does. If there is no one who know, then you're on a science project and need to do some research before anyone will give you budget to proceed.
• Don't waste your time planning the details at the Earned Value. Let the Work Package owner compute the physical percent complete for the Work Package. Don't expose this level of detail to the Earned Value Management processes. For the Planning Packages, use a "peanut butter" spread of the BCWS remaining.
• Always measure progress as physical percent complete at the Work Package level for Earned Value. Anything else for EV is a waste and probably wrong. Physical tangible evidence of progress is required for any credible program management process.
• Stay focused on the Work Package level as the indication of progress. Let the Work Package manager figure out how to get the detailed tasks executed to get to the end of the Work Package.

This by the way is the one-for-one match for agile development. Work Packages = Iteration.

Project Controls Cost Money ! (V2)

This is true, project controls costs money.
Now ask yourself some simple questions:

• Are you, as the project manager, accountable in any way to steward the customers money, to show up on or near the planned time at or near the planned cost?
• Does your customer need to know when the project will be done and how much it will cost when it is done?
• Does anyone outside your small circle of developers on your team have a concern that the money is being spent in ways that support the business strategy - why are you doing this project?
• Is there someone in your organization that hands out money? If so, might they ask you "what did you do with my money?"

If the answer to any of these is NO, then Project Controls costs money that cannot be recovered by the success of the project. Stop doing project controls, you're wasting money.

But if the answer to these questions is YES, then project controls still cost money, but they MAY increase the Probability of Project Success (PoPS). I say MAY, because we work several programs where PP&C is a massive endeavor and the program is still "challenged."

Having good PP&C is a necessary but not sufficient condition for success. I exactly the same way having a credible software development method is necessary but not sufficient.

Program Success Factors

    At the PMI EVM World 2009 conference, Pat Barker of MCR presented an alternative talk when an opening appeared due to a cancellation. First MCR is one of those firms that is a real life "thought leader." Many what to call themselves "thought leaders." Very few are.
    Pat presented a concept that needs to be repeated. Repeated again and again. Shouted from the roof tops. He'll forgive me I know for restating his thesis here. His talk was titled "Anchoring EVM Analysis in GAO Guide." For anyone serious about Program Management and the cost and schedule aspects of program must own a copy of Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs. There are chapters on Earned Value as well.
    This may sound pedantic, but if you're not familiar with the concepts of managing cost and schedule as an inseparable set of variables, you're probably not an actual project manager. Those who manage project managers - program managers - probably have come to understand this.
    Any way here's Pat's concepts. There are 5 elements of a programmatic control system:

• Cost
• Risk
• EVMS
• Schedule
• Technical

MCR calls this CREST. This is a trademarked and copyrighted for concept.
    Inside the CREST concept there are questions that need to be answered. Like any good method, these questions reveal the maturity of the program controls processes:

1. Is the management system producing reliable information to support decision making?
2. How much progress has been made?
3. Is the schedule reasonable and can I meet my required date?
4. Are cost and schedule trends getting better or worse?
5. Will performance expectations be met?
6. Is Management paying attention ot the performance trends and forecasts?
7. Is the estimate to complete (EAC) reasonable?
8. Can the program effectively identify and handle risks?

    These questions are simialr to the Project Breathalyzer questions and the questions I've mentioned in the past. They're the questions program managers and project managers ask. If you're not asking these question every week or every month at least, then you're not "managing" the project, the project is managing you.

The Reason for Program Performance Measures

    There's always voices on the web and blog sphere saying that the effort for measurement is not paid back.
    Here's two simple questions that need answers for any credible project management paradigm:

1. What is the customer getting for the money being spent?
2. If it was your money, would you spend it in the way you're spending the customers money?

That's it. All other project management processes can be derived from these questions. The Knowledge Areas of PMBOK, the DoD Earned Value Management processes described in ANSI-748B, the DoD IMP/IMS Preparation Guide all provide detailed gory guidance in an attempt to answer these questions.
    Maybe these can be test questions for any proposed PM method or suggestion that old methods need to be replaced by new methods.

Can Earned Value Save Government Projects?

I'm attending, along with many 100's of others, the PMI EVM World 2009 conference in Naples Florida. (The proceedings should be available on the College of Performance Management's web site.) During a conversation with one of the attendees, he expressed his frustration with how government (I'll assume US Federal government) projects are poorly run at times.

The speaker works in international construction which certainly has a less than stellar track record of project success. Ignoring the built in inefficiencies of the largest organization on the planet for the moment, I asked "what is the alternative?" The answer I seemed to hear was "well I don't have one; my role is to point out the problem." His clearer answer was "earned value," but that of course is not the solution or the alternative. EV simply reports poor past performance in a timely manner and in units of measure of money, not time.

Pointing out the problem is the the easy part. We all recognize the problem; it's the answer that is elusive. One speaker at the conference stated government projects are different in many ways, some obvious some subtle. This struck a chord with me. They're different but in a way I had not thought of. They have do profit incentive (obvious actually) in the way a commercial IT project does (and ERP deployment), or a commercial constriction project (a high rise or shopping mall).

More important they're "emergent" in exactly the ways agile proponents suggest (but with less success). They are emergent on the defense side because the enemy is changing, the requirements are changing, the technology is changing. They are emerging on the civil side because of the political influences on the project. I've worked one city and one state project. If you think your commercial project has "politics," city projects have real politics!

It might be conjectured that in the absence of a profit motive the behavior of the system is inefficient. That is actually a tautology as well as a restatement of the obvious.

Leaving us with the unanswered question - "what's the alternative?"

What projects should the government - any government undertake? Can these projects be managed in ways different than commercial projects given they are spending the public's money? Is there any evidence that any alternative process than the ones being used today are more effective. By evidence I mean something beyond personal opinion?

I earn my living "working" government projects at NASA and DoD. Our answer is to make all attempts to improve the performance of those projects in what small ways we can.

Earned Value "can" and I emphasize "maybe can" provide actionable information to the Program Manager. But EV only is not a viable solution to increasing the Probability of Program Success (PoPS). http://www.afmc.af.mil/news/story.asp?id=123020393 is a short article on the broader tool set of approaches to increasing the PoPS. Earned value is one of those. But as my paper at EVM World suggests, CPI/SPI are driving in the rearview mirror using past months data with data measures in Dollars. The units of measure for the evidence produced by the actions and documents in this post are part of the attempts to increase the PoPS. Look on Google for the PoPS (Probability of Program Success Air Force Army Navy) concept to see if it adds light to the conversation. I'll post the EVM World paper next week.

The real issue at hand is how to turn all our opinions into some type of actionable outcome, with evidence that those outcomes are beneficial to the stakeholders in some unit of measure meaningful to them? Politics aside, For DoD programs the stakeholder is the war fighter. As a former combatant in Vietnam, I might suggest the units of measure are defined by them, not the non–combatants attending conferences in Florida.

DoD Acquisition Reform Focused on Earned Value

Paul Solomon's web site is a wealth of "breaking news" on topics on interest to Program Planning and Controls and Earned Value. At the site is testimony by the Deputy Desecrate of Defense, Lynn on Acquisition Reform. While there is some conflicting ideas in the testimony, the concept of performance measurement using EV is starting to spread to domains outside DoD.
    The ARRA (American Recovery and Reinvestment Act) requires reporting processes starts with the White Memo. Earned Value for capital projects is obvious. For service contracts its probablematic, since there is usually no clear measure of physical percent complete. For money flowing to "projects" some form of "done," progress toward done, and the units of measure meanigful to the stakeholder makes Earned Value a natural method for measuring progress to plan.

Earned Value Means Measuring Progress Against Plan Cost Not Actual Cost

The Project Management Institute  has several Blogs. Michael Hatfield writes about Earned Value - a topic near and dear to my heart and paycheck. I was excited to see the title of the Blog. When I read the content I was confused.  Because the Estimate At Complete (EAC) formula did not contain the cumulative earned value (BCWP) or the cumulative planned cost (BCWS) or the Budget At Completion (BAC), the way the DoD does Earned Value.
    Let's first start with the source materials. Anyone interested in or actually using Earned Value needs to have the DAU Gold Card in the hands at all times. Access to the DAU site gives a Certificate Error. Just proceed with the link and you'll get to the right page. All DoD sites do this for those outside the net. While you're at it, go to the Defense Acquisition University site and start reading the articles on Earned Value, Programmatic Risk, and subscribe to the Defense AT&L Magazine.
    At a minimum in their notebook. We have a scaled down version of the Gold Card on our lanyards, where our picture security badge, the financial calendar, and other "feathers" hang.
    The Gold Card is a summary of the Earned Value principles. Principles that seem to have been missed in the PMI Blog.
    Michael speaks about a quick way to compute the Estimate at Complete (EAC). Something every project or program manager should know on a weekly basis. But first there is some needed background:

• Earned Value is a powerful tool in the hands of the right people. But good numbers must be in place. With these numbers, experienced program controls people as well. These good numbers come from the Performance Measurement Baseline (PMB). Establishing the PMB is a straight forward process. Here's a talk I gave at the PMI College of Scheduling with the same title. There is more to the PMB than described here, but the principles are simple.
• The three variables of EV are ACWP, BCWS, and BCWP. Only two are independent. BCWP is calculated. Forget the PMI notation of AC, PV, EV. The DoD world has the first notation baked into every form and every ERP system on the planet. Learn the notation.
• Finally, program and project managers are most interested in the Schedule Performance Index (SPI). Cost analyst are interested in the Cost Performance Index. So measuring the performance of a project through the Actual Cost for Work Performed (ACWP) is not allowed under the DCMA (Defense Contract Management Agency) guidelines. Cost Variance is reported, but the Control Account Manager (CAM) lives by the schedule performance.

Simple Formulas Are Just That Simple - Maybe Too Simple
    In the PMI Blog post, there is a formula for EAC (Estimate at Complete)

EAC = ACWP(cum) / Percent Complete

ACWP(cum) is the cumulative ACWP (Actual Cost of Work Performed). This is the cumulative cost of the work to date. In other words "how much money have I spent so far?" The Percent Complete is a measure of the physical progress. BUT this measure is not against the money spent to date, but against the "planned" money that SHOULD have been spent to date. This value is the Budgeted Cost for Work Scheduled (BCWS). Measuring against ACWP provides little or no insight into the project performance, other that the expenditure of money. And as we all should know spending money has little to do with progress other than spending money.
    The gory chart below shows that using ACWP can be a forecast of the future, ONLY if you have spent exactly what you have planned to spend. This is almost never the case unless the project is a Level of Effort activity.
 
Guidance for Applying Earned Value

1. Determine what money should have been spent at a period of time (the period of performance). This is BCWS.
2. Determine the physical percent complete at that period of time. This should - this MUST - be evidenced by some physical deliverable, that is 100% complete. No almost done, no "we'll be done on Monday," no rework, no bugs. It's either done or not done. Any other result is unfunded work past the due date. You're know late AND over budget.
3. The "earned value" is calculated by the Planned Cost (BCWS) x The Physical Percent Complete. This the BCWP - The Earned Value.

With these values the Estimate at Complete can be calculated using:

EAC = ACWP(cum) + [(Budget at Complete (BAC) - BCWP(cum)] / Cost Performance Index (CPI)
CPI = BCWP / ACWP

The notion that:

EAC = ACWP(cum) / Percent Complete

only works if the spend rate is linear and therefore ACWP = BCWP. And that's the definition of Level of Effort.
Actually LOE is defined as

BCWS=ACWP

Some Serious Misunderstandings About Earned Value (update)

The Department of Defense Earned Value Management Implementation Guide, October 2006 says:

1.1 Concepts of Earned Value Management. Earned Value Management (EVM) is a program management tools that integrates the technical, cost, and schedule parameters of a contract. During the planning phase, an integrated baseline is developed by time phasing budget resources for defined work. As work is performed and measured against the baseline, the corresponding budget value is "earned." From this earned value metric, cost and schedule variances can be determined and analyzed. From these basic variance measurements, the program manager (PM) can identify significant drivers, forecast future cost and schedule performance, and and construct corrective action plans to get the program back on track. EVM therefore encompasses both performance measurement (i.e. what is the program status) and performance management (i.e. what can we do about it). EVM is program management that provides significant benefits to both the Government and the contractor.

1.2 EVM and Management Needs. A fundamental requirements for managing any major acquisition system is insight into the contracts' performance specifically the program management and control. Proper EVM implementation ensures that the PM is provided contractor performance data that:

• relates time-phased budgets to specific contract tasks and/or statements of work (SOW)
• objectively measures work progress
• properly relates cost, schedule, and technical accomplishment
• allows for informed decision making and corrective action
• is valid, timely, and able to be audited
• allows for statistical estimation of future costs
• supplies managers at all levels with status information at the appropriate level, and
• is derived from the same EVM system used by the contractor to manage the contract

That's it, other than the other 96 pages of detail in the document.
    But notice there is no "business value" phrase in this statement. The production of Business Value is a separate issue. An important issue, but completely separate topic. Earned Value Management (EVM) is NOT about business value creation and management. No matter how much the agile thought leaders would like it to be. It is about knowing the performance efficiency of the planned budget (BCWS) in terms of Percent Complete within the planned period of performance to produce the "Earned Value," BCWP.

When it is stated

The ... claim is that EVM enables you to objectively measure the value being produced by project teams. Their inherent assumption is that these project teams are actually earning value for your organization, yet in practice we know that this couldn't possibly be true.

    This is simply wrong. The value of Earned Value is not business value, it is the "earned" dollars (BCWP)(EV) from the "planned" dollars (BCWS)(PV).

    All the proposed enhancements with burn down, burn up, business value are outside the scope of earned value as it is practiced by the earned value professionals.

Simple Training Example

Here's a Gentle Introduction to Earned Value Management, that may help clarify the concept of "earning value," while NOT speaking about the business side - that is the cost of the cokkies. Beacuse of course the cookies are not sold, they are Christmas Cookies.

Earned Value and IT Incompatible?

I was poking around for an EV document I had misplaced and came across a 2008 article by Scott Abler in Doctor Dobb's Journal. Scott is a thought leader in many areas of software development. I used his "process" books for many years when I was managing development organizations. I was troubled though by the misunderstanding of a core concept in Earned Value.

The premise that the "value" in Earned Value, is monetary value to the firm is not the case. The "V" in EV, is the measure of physical percent complete in units of measure of money. The money of the budgeted cost of work scheduled (BCWS). That is how much of the BCWS was converted into tangible items – physical percent complete. Whether the functional specification has are "balanced sheet" value to anyone is a local issue. If may be invaluable to NASA for the design of a manned vehicle. It may be worthless to the CIO who is on the line for making a service work.

The Value in Earned Value is not business value

For the government, it's an indication of how much "physical percent complete" has been accomplished to the "planned percent complete." Our firm manages software development projects in aerospace and defense and large IT (enterprise ERP) were EV provides measures of progress in ways not available in the absence of EV. These developments – even in A&D – make use of many of the principles of agile development, along with CMMI process areas.

In a later paragraph you mention EV effective if you have an effective plan. The opposite is actually true – if your plan is not effective you don't need project performance measures, because your project won't last long. Changes to the Performance Measurement Baseline (PMB, the source of BCWS) occur all the time. The need for a PMB, to show where we have come from and where we are going is the key to success of EV. Without some form of "baselining" the team is simply spending some else's money without accountability – this is a common failure mode for IT projects. In the EV world, this is called Level of Effort (LOE) – or "train watching" – actually "watching the train wreck." In LOE, the consumption of funds = progress.

BRUF, BDUF, big anything up front is not a prerequisite for EV. This is a popular myth of those unfamiliar with applying EV. Just the opposite in fact. Rolling Wave Planning Packages are part of DOD 5000.02. Only the open active work packages are used to perform measurements for management decisions.

If you invert your last paragraph of Scotts article and remove a few NOT's you've got how EV is done in A&D software development and Enterprise ERP.

• Define work packages that produce tangibly measureable outcomes. Be they complete and ready to ship, or measurably valuable intermediate deliverables need to produce the actual deliverables. It's the focus on deliverables and the planned value – earned value- for these. Predefining the Earned Value target, and measuring physical progress – physical percent complete – in a standard unit of measure, is what allows EV to be a stable estimator of progress to plan.
• Define what "physical percent complete" means for each work package. We typically (of IT) use an apportioned milestone method within the work package. Define this upfront, measure to the agreed outcomes (deliverables) and record this progress for forecasting future progress. NEVER CONFUSE EFFORT WITH RESULTS. Only measure tangible deliverables. Typically the tasks within a work package have 0/100 measures. Either you're done or you're not done. No partial credit.
• In the worlds we work, Scrum and EV, along with Integrated Master Plan / Integrated Master Schedule are all cousins. IMP/IMS requires you measure progress by the increasing maturity of the product or service being built. Assessment points (Events), are "assessed" with Significant Accomplishment (what is the evidence that you are at a level of maturity you say you are), and Accomplishment Criteria (what is the exit criteria for the Work Package)?
• The notion that partial maturity and 100% completion are strategies for performance is lost on most people outside of defense. For example:
  • I need a database to be stood up for a customer order tracking system
  • What are the incremental measures of maturity for this database? This could be the "walking skeleton" as someone called incremental parts.
  • But what are the actual units of measure for this incremental piece? Critical tables defined, critical PL/SQL tested, etc.
  • Define all this before starting work so you can do one critically important thing that the majority of IT projects we rescue don't – WHAT DOES DONE LOOK LIKE? And how would we recognize we're moving toward DONE, along the way?

Answering the question – What Does Done Look Like – means defining done in some unit of measure meaningful to both the customer and the project team. Physical Percent Complete of a tangible deliverable is a good place to start. Testable Requirements is the formal approach to this outside the agile world. No matter the approach the legal definition is "evidentiary materials." Show me something that you said you were going to show me, and show that it is "done." Not almost done, not "I'm gonna fix the bugs over the weekend," not "well I worked on my machine." No Done, means you won't have to touch this again unless the requirements change. We can ship it. to the customer or the SCC system.

Now how much did we plan to spend for this "done-ness?" If it is completely done, then you get 100% credit for being 100% physically percent complete. BCWP = BCWS X Physical Percent Complete. The actual cost to do the work is a cost management issue, not a schedule management issue.

If you spend the planned budget but were not 100% done, then several things result.

• You're over budget
• You're behind schedule
• You've had to remove features
• You've got rework to do
• You'll have to rob future Work Packages of their budget and schedule to get this Work Package "done"

These are all correct responses. Making them visible is the role of Earned Value.

 

 

Earned Value for Time and Materials

Earned Value for discrete contracts is obvious. It's use for Time and Materials and IDIQ (Indefinite Duration and Indefinite Quantity) contracts is discourages in the guidance found in the 7 March 2005 Memorandum from the OUSD(P):

5. EVM is discouraged on firm-fixed price, level of effort, and time and materials efforts, including contracts, subcontracts, intra-government work agreements, and other agreements, regardless of dollar value. If knowledge by both parties requires access to cost/schedule data, the first action is to re-examine the contract type (e.g., fixed price incentive). However, in extraordinary cases where cost/schedule visibility is required and cannot be obtained using the Truth in Negotiations Act, the program manager shall obtain a waiver for individual contracts from the Milestone Decision Authority. In these cases the program manager will conduct a business case that includes rationale for why a cost or fixed price incentive contract was not an appropriate contracting vehicle.

This said, it may be an easy out to say - No EV.

An 28 April 2008 memorandum states

The application of EVM to work efforts that are not discrete in nature (non-schedule based), such as level of effort, time and materials, and services, should be considered on a case-by-case basis.  In cases where the nature of the work does not lend itself to the meaningful use of EVM, it may be appropriate to waive the EVM requirement.  When EVM is waived, the program manager will implement an alternate method of management control.

What method used for "management control" to replace EV needs to answer the questions:

• What does "Done" look like? - when the money runs out? Or the time runs out? Both? When the contract is canceled for lack of performance?
• How can we know we are actually providing "value" (earning this value) for the customer?
• What "value" should be earned during the performance of a T&M contract?
• If we are spending money and passing time, what is the customer getting for this?

These are answers that Earned Value provides. But getting these answers in the absence of Earned Value requires the use of some other performance measurement process.

Latest Earned Value Papers

The current Earned Value concepts are described in a set of new papers:

• Project Duration Forecasting: Comparing Earned Value Management Methods with Earned Schedule, Walter Lipke
• The Two Mist Useful Earned Value Metrics: The CPI and TCPI, Quentin Fleming and Joel Koppleman
• In the Fall Edition of the Measurable News, the magazine of the PMI College of Performance Management.
  • "Does Project Performance Stability Exist?: A Re-examination of CPI and Evaluation of SPI(t) Stability?,"Kym Henderson
  • "Integrating Systems Engineering with Earned Value," Paul Solomon
  • "The Trouble with Earned Schedule," Wayne Abba

These papers and many others in this topic provide visibility into the complexities of Earned Value. These complexities should in no way prohibit the application of Earned Value. In fact they should encourage the application of Earned Value for all the right reasons.
    What they show is managing projects is a profession, with professional tools, with the expectation of professional results.

Another Confused Look at Earned Value Management

David Schmaltz provides a look at the application of Earned Value to projects. Several common mistakes are made in this post, possibly from lack of experience on programs subject to 748B or possibly just from not seeking out information from those who earn their living complying with DoD, NASA, and DOE DCMA regulations.
    For those seeking the background to the application of EV, there are nearly unlimited sources. But the starting point is quite simple:

• EIA/ANSI-748 describes the 32 areas for a compliant Earned Value Management System (EVMS)
• EVMSIG describes the intent of EVMS and its application to programs subject to EVM
• EVMS Surveillance Guide is the directive for evaluating an EVMS

One of the best overview for Earned Value is provided at a Department of Energy site by Booz Allen.

• Introduction
• Work Breakdown Structure
• Scheduling
• Budgeting
• Concepts and Methods
• Metrics, Performance Measures and Forecasting
• IBR and Change Control
• Reporting

Any one providing criticism of EV would be well served by first going through these course training materials to confirm they have a solid understanding of the principles. Once this is done, finding someone or someplace where EV is being practiced under the guidance of 748 would be a next step. This avoids the common mistake of assuming that one knows the practice of EV before starting down the path of criticism. By the way, there are many things to be critical of. But they are not the items you might think.

Earned Value is Necessary But Not Sufficient

    The use of Earned Value in construction, government, and defense programs is common practice. Some SOX IT organizations have discovered the benefits of EV. But using the EV variables - Cost Variance, Schedule Variance, Cost Performance Index, and Schedule Performance Index alone is necessary but not sufficient.
    The consumption of money, the passage of time, and the measures of Percent Complete must be combined with the assessment of the technical performance of the produced product or service. The units of measure in the Earned Value Standard ANSI/EAI-748B are Quantitative. They are also measured in Dollars. This is a problem in any work that is producing something new. Something that is not like laying bricks, painting airplanes, or pouring concrete. In the "development" world there are several other issues as well:

• The work is usually not repetitive, development programs are not like production or manufacturing.
• The baseline requirements may be changing over the period of performance of the project. Changing for all the right reasons. «If they are changing for the wrong reasons, that's another whole problem»

So, we need a Qualitative measure of performance for the project. Some notion of technical compliance or Technical Performance Measurement must be added to make Earned Value sufficient for project "management" work, not just project performance reporting. This is the difference between Driving in the Rear View Mirror and Driving out the front Windshield.
    Here's why:

When the end of the period of performance arrives, be it a week, a few weeks, but hopefully not more than a month or two, and the planned budget has been spent, and the planned duration has elapsed, was the produced produce or service "on specification?"

If not, then several things result:

• There is rework
• There is more money needed to complete the product or service
• The features need to be de-featured
• The customer needs to be convinced to take the product or service "as is"

In all cases more money and more time is needed (expect that last one) to get the product or service to the planned level of performance.
    So the Earned Value performance indicators are in fact not showing the actual state of the product or service. It is over budget (more money for rework, repair, or completion) and over schedule (more time needed to rework or repair). Or and this is a subtle outcome that comes home to bite you every time, the undelivered features get pushed into a future release, WITHOUT the associated budget or resources - since these have been spent on the current period of performance (iteration, release, workpackage, what ever).
    This is called mortgaging the future for the failures of today. It's called "we'll make it up in system test." NOT.

In the End Cost, Schedule and Technical Performance are Inseperable
    There are numerous posting, articles, "magic bean" purveyors claiming that the Iron Triangle is dead. Not True, Never was True, Can Not be True.
    If you spend the money, and spend the time, and the result is not 100% what was ordered (or at least within the tolerance of error), then it's going to take more money and or more time, probably both. And most probably this effort and money is unbudgeted and unplanned.
    So what to do?:

• Define what done looks like in units of cost, time, and technical performance
• Ask "how long a, I willing to wait before I find out we're late?"
• Never speak about effort, only speak about measurable results. Never speak about output, only speak about outcomes.

These outcomes and results, must have predefined units of measure for compliance with the planned result. These units must be meaningful to the customer as well.

One Last Comment
    There are some that equate agile burn down charts with Earned Value. This is not correct - unless the burn down charts include cost, schedule, and technical performance. Other than that, the two charts have two coordinates and lines that change shape with the passage of time, so that's the only way they are related.
    This is a popular myth in the agile community for some reason. And has been propagated by several "Thought Leaders," in error. It'd be wonderful if they were related, but they're not. So more work still remains to educate the both communities about the subtle behaviors of Earned Value in the presence of agile discussions.