using Principles, Practices, and Processes to Increase Probability of Success
I'm working two programs where Agile at Scale is the development paradigm. When we start an engagement using other peoples money, in this case the money of a sovereign, we make sure everyone is on the same page. When Agile at Scale is applied, it is usually applied on programs that have tripped the FAR 34.2/DFARS 234.2 levels for Earned Value Management. This means $20M programs are self assessed and $100M and above are validated by the DCMA (Defense Contract Management Agency).
While these programs are applying Agile, usually Scrum, they are also subject to EIA-748-C compliance and a list of other DID's (Data Item Description) and other procurement, development, testing, and operational guidelines . These means there are multiple constraints on how the progress to plan is reported to the customer - the sovereign.
These programs are not 5 guys at the same table as their customer exploring what will be needed for mission success when they're done. These programs are not everyone's cup of tea, but agile is a powerful tool in the right hands of Software Intensive System of Systems for Mission Critical programs. Programs that MUST, deliver the needed Capabilities, at the Needed Time, for the Planned Cost, within the planned Margins for cost, schedule, and technical performance.
One place to start to improve the probability that we're all on the same page is this reading list. This is not an exhaustive list, and it is ever growing. But it's a start. It's hoped this list is the basis of a shared understanding that while Agile is a near universal principle, there are practices that must be tailored to specific domains. And one's experience in one domain may or may not be applicable to other domains.
Like it says in the Scrum Guide.
Scrum (n): A framework within which people can address complex adaptive problems, while productively and creatively delivering products of the highest possible value.
And since Scrum is an agile software development framework, Scrum is a framework not a methodology. Scrum of Scrums, Agile At Scale, especially Agile at Scale inside EIA-748-C programs has much different needs than 5 people sitting at the same table with their customer with an emerging set of requirements where the needed capabilities are vague until they appear.
One of the classes every aspiring grad student has to take is research methods. This class teaches the PhD hopefuls (I didn't make the cut and got a consolation prize of a MS), all about doing research and preparing to be a real scientist. A topic in this class is literature search. This makes sure that your cleaver idea of a research topic, in case your advisor hasn't gotten around at actually talking to you, has already been taken, researched, and solved. This is one problem in the physics world - you need an original idea. Replicating old ideas doesn't get you very far.
Here's a start of a literature search on merging Agile at Scale with Earned Value Management. I haven't gotten to the European and Far East journals yet. Instead is a list, I'll just type this once and repurpose the resources here. This PDF is the Resources section of a briefing being used with our clients who are integrating Agile into EVM programs. Go to the LinkedIn Slideshare site - the LI logo in lower right, to open the PDF and follow the links.
When we mean to build, we first survey the plot, then draw the model, and we see the figure of the house. Then we must rate the cost of erection, which if do find outweighs ability. What do we then, but draw a new model in fewer offices, or at least detest to build at all?
- Bardolph, Henry IV, Part II, Act I
The goal of managing other people's money when build a product or providing a service is to plan and coordinate the needed work activities to deliver a satisfactory outcome, or complete enterprise endeavor within the constraints of schedule, budget, resources, infrastructure, and available technology.
The intellectual content of the discipline of engineering, business and technical management, risk management, and program controls, are oriented on the components and are value neutral.  Not matter the outcome the processes are the same or similar. The value produced by these efforts is independent of the means to produce them. Once delivered he consumer of this value cares little how they arrived. That consumer didn't buy the process to produce that value, they bought the value. When these are confused the notion of focusing on value is perverted to focused on those spending the money rather than on those providing the money.
The underlying principles of these disciplines are focused inside the boundaries of that system. The resulting value is focused outside the system of it production.
Project success depends on the integration of the activities below. The primary role of the processes below guides the value producing activities to …
Design the Programmatic Process to support the Technical Project Engineering activities to Increase the Probability of Project Success.
 Systems Engineering Body of Knowledge, V0l.5,
At a recent conference the discussion of the integration of Agile with Earned Value Management on programs subject to FAR 34.201 and DFARS 252.234-7001 was the topic. Here's my presentation.
In the business of building software intensive systems; estimating, performance forecasting and management, closed loop control in the presence of uncertainty for all variables is the foundation needed for increasing the probability of success.
This means math is involved in planning, estimating, measuring, analysis, and corrective actions to Keep the Program Green.
When we have past performance data, here's one approach...
Many times I hear about Cost of Delay, Deliver Value, Measure story points, or Measure Stories. And a myriad of other assessments of project performance, all of which - OK, most of which are examples of Open Loop Control.
Back in 2014, we had a paper in a publication of the College of Performance Management, starting on Page 17. As well, a colleague Nick Pisano (CDR US Navy Retired) has a post on the same topic at his blog.
The notion of a baseline let alone a Performance Measurement Baseline is at the heart of Closed Loop Control of all processes, from your heating and air conditioning system in your house, to the flight controls on the 737-700 winging its way back home to Denver, to the project you're working - using what ever project management method or software development method of your choosing.
The notion that we can manage anything, the temperature of the room, the nice soft ride in the 737, or the probability of showing up on or before the need date, at or below the needed cost, with the needed capabilities - and NOT have a baseline to steer to is simply wrong.
Below is the framework for Closed Loop control. This paradigm says simply:
ESTIMATING IS THE BASIS OF DECISION MAKING - it can't be any clearer than that.
We gave a recent College of Performance Management webinar on using techncial progress to inform Earned Value. Here's the annotated charts.
In the estimating discussion there is a popular notion that we can't possibly estimate something we haven't done before. So we have to explore - using the customers money by the way - to discover what we don't know.
The when we hear about we've never done this before and estimating is a waste of time, think about the title of the post.
Everything's a Remix
Other than inventing new physics all software development has been done in some form or another before. The only true original thing in the universe is the Big Bang. Everything else is derived from something that came before.
Now we may not know about this thing in the past, but that's a different story. It as done before in some form, but we didn't realize it. There are endless examples of copying ideas from the past is thinking they are innovative, new and break through. The iPad and all laptops came from Allan Kay's 1972 paper, "A personal computer for childern of all ages." Even how the touch screen on the iPhone works was done before Apple announced it as the biggest breakthrough in the history of computing.
In our formal defense acquisition paradigm there are many programs that are research. This looks like this flow. Making estiimates about the effort and duration is difficult, so blocks of money are provided to find out. But these are not product production or systems development processes. The Systems Design and Development (SDD) is between MS-B and MS-C. We don't confuse exploring from developing. Want to explore work on a DARPA program. Want to develop, work in post-MS-B and know somethiong about what came before.
The Pre-milestone A works is to identify what capabilities will be needed in the final product. The DARPA programs I work are even further to the left of Milestone A.
On the other end of the spectrum from this formal process, a collection of sticky notes on the wall could have similar flow of maturity. But the principles are still the same.
So How To Estimate in the Presence of We've Never Done This Before
Here's a critical concept - we can't introduce anything new until we're fluent in the language of our domain, and we do that through emulation.† This means for us to move forward we have to have done something like this in the past. So if we haven't done something like this in the past, don't know anyone who has, or can't find an example of it being done, we will have little success being innovative. As well, we will hopelessly fail in trying to estimate the cost, schedule, and probability of delivering capabilities. In other words we'll fail and blame it on the estimating process and assume that we'll be successful if we stop estimating.
So stop thinking about we can't know what we don't know and start thinking someone has done this before and we just need to find that someone, somewhere, something. Nobody starts out being original, we need copying to get started. Once copied, transformation is the next step. With the copy we can estimate size and effort. We can now transform it into something that is better, and since we now know about the thing we copied, we have a reference class. Yes that famous Reference Class Forecasting used by all mature estimating shops. With the copy and it's transformed item, we can them combine ideas into something new. The Alto from Xerox and then the Xerox Star for executives, was the basis of the Lisa and Mac.
You can estimate almost anything, and every software system if you do some home work and suspend the belief it can't be done. WHY? because it's not your money, and those providing you the money have an interest in several things about their money - what will it cost, when will you be done, and using the revenue side of the balanced sheet, when will they break even on the exchange of money for value? This is the principle of every for profit business on the planet. The not-for-profits have to pay the electric bill as well, as do the non-profits. So everyone, everywhere needs to know the cost of that value they asked us top produce BEFORE we've spent all their money and ran out of time to reach the target market for that pesky break even equation.
Anyone tells you otherwise is not in the business of business, but just on the expense side and that means not on the decision making side either, just labor doing what they're told to do - which is a noble profession, but unlikely to influence how decisions are made.
The notion of decision rights is the basis of governance. When you hear about doing or not doing something in the absence of who needs this information, ask who needs this information and is it your decision right to decide to fulfill or not fulfill the request for that information? As my colleague, retired NASA Cost Director, says follow the money, that's where you find the decider.
† Everything is a remix, Part 3, Kirby Furgeson.
There was a question posed on LinkedIn
How do you get Project Manager understand the importance of Earned Value Management?
It's been shown over and again when we make EV a compliance process and hold managers accountable for compliance it is a necessary condition for success, but far from sufficient.
Until EVM starts providing actionable information to the program management staff in the form of "leading indicators" it will always be one of those compliance processes. This information must reveal where in the program, technical and programmatic changes can be made, the testable outcomes of those changes on program performance, and the impacts on cost and schedule forecasts and the EAC against the BAC. Starting at the program level, but going down to the Control Account and Work Package.
In other words...
Nice data there in your Formats 1-5 and really nice IMS, what should I do next?
EV numbers themselves are lagging indicators, non-statistically adjusted, non risk adjusted, not connected to the effectiveness and performance measures of the project - unless enlightened users do so - and not connected to the needed capabilities of the customer as delivered by the program. There is no DID for the IMP, so Systems Engineering rarely flows down MOEs and MOPs - except where they do, because of the knowledge of the power of this approach.
Next is a real problem. 748-C has a loop hole big enough to fly a 787 through. In section 3.8 "Earned value is a direct measurement of the quantity of work accomplished. The quality and technical content of work performed is controlled by other processes." Measuring quantity is a construction centric view of producing value. Not a value centric view. Our defense, space, biopharma, software intensive programs that apply EVM are not pouring concrete or welding pipe. (I used to work in that domain as a SW developers on piping design systems). If we see EV as measuring quantity we ignore all the concerns Paul Solomon has brought forward, starting with the missing Technical Performance Measures for the product.
Sure we have exit criteria for the Work Packages. But these need to trace vertically through the IMP to the Capability to accomplish the mission or provide a solution to the business need. In other do something with the resulting product or service. It has to be the right thing of course. But EV - as stated in 748 - only measures to quantity of parts needed to delivery a capability, not the ability of the system to fulfill the needed capabilities. This by the way is essentially a Systems Engineering issue. With the missing IMP, most of the motivation for connecting the dots between EV and mission success is simply not there. It's back to the immutable principle
We don't know what done looks like in units of measure meaningful to the decision makers
"Earning value" means assessing the efficacy of the BCWS. This means assessing the Technical Performance Measures of the work be delivered. Many implementations assign TPMs and QBD's this work. But this is not called out in 748-C nor any formal EV guidance. It is called out in the DAG and the old SEMP DID. No SEMP DID is in place.
So as EV practitioners we need to make the business case of EVM, not just the compliance case. When we add "leading indicators" that are statistically sound (Eric Druker has written about this as have others, including me), forecasting of EAC based on ARIMA processes rather than our linear, non-statistical, non risk adjusted CPI/SPI measures. As well those measure "roll up" the variances of all the past time series data, just like Darrell Huff tells us to do in "How To Lie With Statistics".
All the data is available in the EV engine and in the CR at PARCA for DOD jobs. What's next is to start using EVM in the same way supply chain managers, quality control managers, systems engineers, and every design and manufacturing engineer does - compare statistically adjusted performance against the probabilistic plan to see leading indicators emerge of where we are going to go into the ditch in the future, show the PM, that corrective action "before the fact" rather than after.
This will be a topic discussed at EVM World and ISCEA conferences coming soon.
Earned Value measures the production of tangible outcomes from work efforts on a project using Physical Percent Complete. This P%C is calculated used Quantifiable Backup Data shows what technical or operation goals must be met on a planned date in order to claim some Percent Complete. Measures of this physical percent are NOT how much money you spent or how much time you took to do the work.
They are only measured in unit of tangible technical progress to plan. This tangible technical progress is defined BEFORE the work starts. This progress goal is derived from the time phased Measures of Effectiveness and Measures of Performance goals of the project.
These MOE, MOP, KPP, and TPMs are defined below and their measures - in units meaningful to the decision makers - established before work starts, put on baseline and used to compare progress to plan (time phased) to produce that Physical Percent Complete in the Earned Value (BCWP) formula in the first chart.
Unless we know what capabilities we need at the end of the project, and know those somewhere near the beginning of the project, have a Plan to produce those capabilities at the needed time for an anticipated budget, we're going to be disappointed in the result.
Identifying the needed capabilities is the first and most important practice for the success of any project. To achieve the project's objectives or a particular end state, we need to define these capabilities through Measures of Effectiveness. We can do this through scenarios from the customer's point of view. The effectiveness measures describe how well the results from the project enable a business process or fulfill a strategic mission of the business - in this case enforcing the will of the emperor. These measures must be in units meaningful to the decision makers. In the case Darth Vadar.
When we let those needed capabilities emerge we have likely disconnected the products of the project from the mission or vision of the business. Requirements can emerge to fulfill the capabilities. But the capabilities need to be steady and focused on mission outcome, otherwise we are just on a spend plan. We'll have no way to know what done looks like until the money runs out.
Here's an example of an increasing maturity of the capabilities for a health insurance provider network ERP system.
This approach is not usually found in traditional or agile projects. These approaches start with requirements, stories, features - all related to the technical aspects of the project. A Master Plan is developed that shows the needed capabilities (above), the order of delivery of these capabilities, and the dependencies between these capabilities.
The increasing maturity of the project's outcomes is then stated in this map. Technical requirements implement the capabilities, so they need a plan - schedule actually - as well. Small incremental capability delivery is the best approach. This is obvious and the basis of agile, but not always planned that way.
This plan contains all the steps (delivered capabilities) needed to increase the maturity of the of the project from the start. This does not mean we know how these capabilities are going to be delivered, but it does show what DONE looks like at the end of the project. In this example Phase 1 - Go Live.
This approach does not define any specific development process - agile or traditional. It is the basis of success for any development process.
Capabilities Are the Vehicle For Delivering Value
The Capabilities and their Measures of Effective need to remain stable over the life of the project. Without this reached DONE with any sense of budget and schedule will not only be difficult, you will be rubber banding the baseline used to measure your performance. You'll have turn a project into an operational activity. Projects have defined outcomes, defined periods of performance and associated budget. Operations can be fixed budget, open ended periods of performance, and emerging outcomes. But reaching a defined description of DONE becomes much more difficult.
The notion of successfully managing a project starts - as always - with the 5 Immutable Principles of project success.
So What's the Point?
If these principles are not in place on your project, it'll be a disappointment. So in the end any successful project management process must answer the following:
By the way, the common objections by some in the agile community to the notion of on-budget, on-schedule as a measure of project performance is seriously misinformed. Those paying for the project not only want to know these, they must know this. It's not the developers money. It's the customers money. They must know the final cost and the date when the project is done. They must also - of course - know that what is being produced delivers the needed capabilities. But having those capabilities late is not good. Spending much more than budgeted is not good. All three attributes are needed.
These numbers by the way are probabilistic. Every number on a project is drawn from a statistical process to produce a probabilistic outcome. The notion that NOT forecasting the future performance of cost, schedule, and technical outcomes can't be done is simply nonsense. Of course it can be done. You just have to remember the things you learned in your high school statistics class. Plus a few other things about probability distributions and confidence levels.
There have been several posts and discussions on LinkedIn about applying Earned Value Management to projects. This coming Thursday, I'll be talking about applying Agile software development processes to Earned Value Management project, but in order to do that we need Earned Value to be applied first. So let's look at how to install EVM on your project.
Let's start with some basics and get rid of the myths of EVM
So now let's look at setting up an Earned Value Management System at the highest level - the principles level.1. What are we building?
3. Master Plan and Schedule
4. Resouce Plan
5. Cost collection
6. Baseline Management
So Now We've Got an EVMS, What Do We Do?
When we have our Performance Measurement Baseline (PMB), the Integrated Master Schedule (IMS) with its seqeunce of Work Packages and work activities, with resources assigned to do the work and the budget to fund that work, we can start to measure progress to plan.
At periodic times - usually weekly, possibly bi-monthly, but for sure every month, we can assess the physical percent complete on the work. This means measuring tangible progress to a pre-planned set of attributes. With this measurement comes the physical percent complete. With the physical percent complete we can calculate the Earned Value.
With this measure we can determine where we are along the way to complete, how much we need to improve out performance, and what corrective actions we need to take to get back on schedule, or stay on schedule.
There's a post on a Deltek implementation partner site about applying Earned Value. It has some good advice, but the premise of the starting point needs to be addressed.
Before going on to suggest things to do on a project using EVM, it's best to have one of those don't do stupid things on purpose discussions. If the customer or potential customer is doing stupid things on purpose, it's important to stop those first, before attempting to make any other improvements. Once that is done, the core paradigm of Earned Value Management is very simple and obvious ...
Turning the Subjective into the Objective
This phrase is not mine, it is a colleague's who said it while we were walking down the hallway of a client site, who is introducing EVM to a $1.9B weapons program that has never had EVM before, but is now being mandated to do so by Congress!!!
So Here's the Introduction to What Not To Do and the Simple Fix
Let's start with a picture of where to start. The are 32 Guidelines in ANSI-748-C. 11 are needed for he success of any project, in any domain. If you aren't doing these 11 in some way, the probability of success for your project is low. This is the case of all project management and product development processes, These are like the 5 Immutable Principles - they have to be there in some way. Here's the 32 from ANSI-748-C and the 11 are colored and listed below.
So Let's See How the 11 Can Address Each Of These Issues?
Here's their list of reasons why Earned Value Management is not readily adopted by contractors.
So What Does All This Mean?
It means applying the 11 Guidelines shown above to your program is the same as being a credible program manager. If you're not doing these 11, you're probably not doing your job as a Program Manager on an EVM program. Think about that. You've been assigned to look after $20,000,000.00 of the governments money. Would you actually use some the phrases above when asked why you're doing a poor job of managing the program? Not for long I suspect.
One Final Comment
Earned Value Management is one of several enabling technologies to turn the subjective into the objective. But it is only a necessary condition for success. It is far from a sufficient condition for success. You need an Integrated Performance Measurement Baseline with these elements
Ray Stratton's news letter just arrived. There he speaks to the "44 Day Rule," for the duration of work activities. Some places use that for the duration of a Work Package. Other places use that as the duration of a work activity contained in the Work Package.
Here's another idea.
You need to answer the question - How long are you willing to wait before you find out you are late?
The answer to this question, then drives two things:
With the answers to these two questions, the duration of a work activity should now be self evident. As well as the point in time needed to take corrective action. This corretive must have enough time to have the lateness be fixed. So some arbitrary durations cannot be used. You need to know how the work is being performed.
But a simple (and simple minded) rule assumes the work is being performed linearly. So yuo can answer the question by saying, what is the capacity for work that must be applied to fix the activity when it gets behond schedule?
By the way, when this corrective action is applied, it will cost more money, unless you change the scope or improve the effectiveness of the work to cover the extra work needed to correct the problem.
This is why short duration activties are best. The Agile folks know this, maybe not for the same reason. But short duration tasks - the 44 day rule - force the assessment of work so corrective actions can take place.
How many legs does a dog have if you call the tail a leg? Four. Calling a tail a leg doesn't make it a leg
- Abraham Lincoln
The notion that we can rename things and then assume they are the same doesn't work, just like Lincoln said. The notion of measuring progress with the passage of time and calling it Earned Value, doesn't mean we're doing Earned Value.
Doing Agile without doing Scrum, XP, Crystal, or DSDM is not doing agile.
The LinkedIn forum The Project Manager Network - #1 Group for Project Managers which I sometimes wonder if that is actually the case, has a conversation going on around project failure. A Master's student asked about project failure modes. That led to the use of Earned Value. One participants made some pretty boneheaded statement around EV:
5% (2 hrs)for applying the border and title
20% (8 hrs) for defining the area - boundaries and overall dimensions
30% (12 hrs) for showing all the equipment
5% (2 hrs) for the drawing being checked
10% (4 hrs) for it being Issued for Approval
15% (6 hrs) for it being Issued for Bid
15% (6 hrs) for it being Issued for Construction.
100% = 40 hrs.
So, silly me, I responded that determining physical percent complete was a straight forward process once you have Quantifiable Backup Data or measures of Physical Percent Complete, or even better, Technical Performance Measures. Let's look at each concept
Then the final come back from the OP.
If you're doing a drawing and it takes 40 hours to do the drawing and no two drawings are alike, how do you determine what percentage complete you are? When estimates of the work to be completed are put together, they will estimate X drawings at 40 hrs/drawing. They will have different standards by discipline. Will a person ever report 41% complete on a drawing? Most likely not. It will be shown as 40%. Collectively all the drawings may be 41%, but not individual. No matter how you look at it, it is still an estimate.
First measuring progress to plan by the number of hours consumed is called Level of Effort. Second LOE is a very poor measure of anything other than the passage of time and consumption of resources.
In the end the poster was insistent that ANSI-748-B is just a guide, and doing EV was a local issue. No doubt that is true in his domain. But as a practitioner of EVM using ANSI-748-B for programs subject to DCMA Validation, as well as engagements with the DOD office that own the Earned Value Management processes for the DOD, I'd be laughed out of the room with an approach like that.
For a summary Earned Value in a Nutshell using a similar drawing example.
So Here's The Way TO Do Earned Value
That's all there is - essentially - for doing Earned Value Management. Of course if you read ANSI-748-B, the NDIA Earned Value Management Intent Guide, and the DCMA Intent Guide, there is a lot more.
You Can't Make Stuff Up
OK, you can. But if you do, it makes having a conversation about EV much harder.
When I encounter a "subject matter expert," I've now learned to assess if this person is a "knowledge holder," or a "learning individual." This is what informed that "learning." This came to my from a friend and neighbor who is a graduate of the University of Chicago in economics and spoke at this conference.
In our business - project and program management - there are lots of "knowledge" providers, assessors of knowledge, frameworks for measuring our knowledge. What is needed now is how to learn to make better decisions about our projects. This is not going to happen without a decision making framework for the core principles, practices, and processes for project management. The current Bodies of Knowledge, self-proclaimed providers of better BOKs and assessment tools, authors (I'm one of them), and experts in the field, all need to reassess what we are doing to increase the Learning abilities, beyond the just the possesion of knowledge and the dispensing of this knowledge to others.
How can we actually learn to improve our probability of succes. The Decision Scientist for project success - as suggested here - needs to have art, science, and the scale to provide actionable information for the decision makers for the project.
We have most of the data we need, what we don't do is make decisions on this data, or even know how to make decisions with this data. We report it, but don't use it to make decisions. Numbers matter, but leadership needs to use the numbers.