Scott Ambler writes again about the use of Earned Value on IT projects. Scott makes several fundamental errors in this discussion.
The ANSI-748B version of "earned value" is NOT the same as business value. It never has been, it never will be, it never can be. Scott repeats this misunderstanding in other forums as well as Dr. Dobbs.
The second misunderstanding is that blaming EV for not having a credible Performance Measurement Baseline, for uncontrolled changing requirements, for all the other sources of project performance degradation is simply - how should I put this - Nonsense.
It'd by like blaming Scrum for the poor results of a software development projects, because the developers did not follow the guidance of Scrum.
From the post
The fundamental problem with EVM is that it has very little to do with earning value and everything to do with managing against what often proves to be naive/fictitious schedules and estimates committed to far too early in the project.
Yes, EV has noting to do with business value. This is not an EV problem. To use this as a basis for discussion is not just a red herring, it's just bad knowledge. Naive and fictitous schedules are just that naive and fictitous. Why on God's green earth would you allow a project to proceed with a naive and fictitous schedule. You wouldn't let a Scrum project proceed with a bogus set of stories, use cases, and planning process. Can't understand why Scott makes these kind of statements? It doesn't seem very productive to the real problem.
Although EVM is interesting project management theory, and I have no doubt that it may work in some non-IT domains, it proves to be a really bad project management strategy in practice for IT development projects.
Theory? Please. I seriously doubt Scott has even seen a project where EV is used to develop software that was guided by ANSI-748B, FAR 34, FAR 52, DCMA, DoD 5000.02, let alone actually worked a program under 748B guidance.
We do this for a living. We work $500M to $1B (yes, that a billion with a B) software development programs that use EV weekly and at the same time use "agile-style," practices - monthly drops fo working spaceflight software, bounded requirements changes (rolling waves), emergent requirements, rolling waves), and drastic changes in mission (we're not going to the moon, we're only going to station, not problem changing all the guidance, navigation, and control (GN&C) and the command and data handling (C&DH) software, firmware and hardware right?"
Although EVM can in fact be applied to agile projects, in my opinion EVM is a questionable practice, regardless of paradigm. Organizations that are trying to govern their IT project teams should monitor them in such a way that accurate and timely information is presented. This is clearly not the case with EVM. We have significantly better options available to us. Let's adopt them. More on this in future blog postings.
This is simply wrong headed.
- Teams must govern their IT projects in such a way to provide accurate and timely information, but this information must convey actionable information to the decision makers. EV is one way to do that when you're spending other peoples money. To do this, EV must have a credible baseline, it must have measures of physical percent complete, it must produce the Estimate to Complete (ETC) and the Estimate at Completion (EAC) in time to make these decisions.
- The sentence This is not the Case with EVM - is misinformed at best, since there are many 100's of examples in DoD, NASA, DOE, DHS, NIH, FAA, major aerospace, defense, and major construction and enterprise business systems (ERP) firms of IT and software intensive programs doing just what Scott says can't be done. Maybe Scott needs to do a survey of sites that actually make this work. I can introduce him to Northrup, Lockheed, Boeing, and other clients doing this daily.
So Here's the Point (One More Time)
- Earned Value is a project performance measurement system that forecasts future performance based on past performance by measuring the "actual" (BCWP) performance against the "planned" (BCWS) performance in units "money."
- Earned Value defines the Planned Value (BCWS) as the "budgeted" cost for the work to be performed. It DOES NOT (I can't say this loudly ) define the "business value" of that work. The "business value" is a business domain issue. And BTW the way agile doesn't define "business value" in any units meaningful to the actual business directly either. The "business" value is measured in "money."
- If you want to see how to connect the dots between project measures and measures, Notes On Balanced Scorecard (Page 48, 49, and 50) as a place to start.
- Earned Value measures accomplishments to plan be assessing the Percent Complete of the planned work. The best way to measure this is with Physical Percent Complete. Which BTW is exactly what Scrum does, but no in the units of measure meaningful to the decision maker - "money."
- This means that the planned value (budget) must be measured at the time it is planned to be complete and them calculate the "earned value" (BCWP) for that work.
This is EXACTLY what agile does with story points during the planning session and the measure of completed story points at the end of the iteration.
What Scott seems to fail to understand it that the deployment of EV is problematic in the defense business as well. The fact that he sees executives mis-using is not a surprise. The same thing happens on major weapons systems. Just look at Joint Strike's little "problem."
It's not EV's problem people don't follow the rules. Don't do stupid things purpose!
There are overwhelming examples of following the rules producing benefits.
So Some Final Comments
It takes a few minutes, but it's pretty easy to get senior management to admit to horror stories where there were some nasty surprises at the end of projects where EVM was reporting good news throughout most of the effort.
There are lots of horror stories - even in our A&D space - but that's not EV's fault, it's simply bad management. Scott seems to have fallen into a mode that blames the tools for the poor carpentry.
We've seen too many IT projects which reported that they were 90% complete, only to have them come hat in hand asking for sufficient time and funding to complete the second 90% of the effort.
If this is the case those projects are not compliant with EV and the DCMA guidelines defined in ANSI-748B.
- You cannot not physically have this happen in an EV system. Measures of Physical Percent Complete are defined in the Earned Value Management System Description. (here's a good example for the DOE).
- Reporting 90% complete and not being physically 90% is a violate for the DCMA surveillance and the FAR/DFAR clauses. You're a felon, you'll get a "corrective action report" sent to Congress, and they'll cancel your program and toss you into the parking lot - come on get real here!
The Final Advice
- Ignore the arguments against EV from those not having deployed EV successfully in the IT intensive world. I've heard this in the past from the husband of an agile "thought leader." "We tried EV and it didn't work." Really, did you know what you were doing?
You wouldn't take agile advice from someone who has not succesfully deployed agile in a domain and context similar to yours. Don't do the same for anything else.
- Apply EV to projects that are mission critical, enterprise class, expensive, and using other peoples money ONLY if you are actually capable of deploying EV in that enviornment.
- Don't let executives - who are uninformed about EV - deploy EV. Don't let anyone uninformed about EV deploy EV.
- Don't let anyone uninformed ab EV tell you EV won't work, without the hands on direct experience of actually making EV work.
The absence of evidence does not mean the evidence of absence.
- Pick the right project. One that can actually deal with the issues of an agile development process in the presence of a "governance" process like EV or anyother governance process - ITIL, CobiT, even Scott's employeer's Rational tools and processes.
- Read the Successfully Integrating Agile with Earned Value Management and Earned Value + Agile = Success for advice on how to actually do wat Scott says can't or shouldn't be done.
