Customers buy capabilities to accomplish a business goal or successfully complete a mission. Deliverables are the foundation of the ability to provide this capability. Here's how to manage a project focused on Deliverables.
Principles, Practices, and Processes to Increase Probability of Success
Customers buy capabilities to accomplish a business goal or successfully complete a mission. Deliverables are the foundation of the ability to provide this capability. Here's how to manage a project focused on Deliverables.
When we hear about project planning and scheduling, we think about tasks being planned, organized in the proper order, durations assigned to the work, resources committed to perform the work.
This is all well and good, but without a risk adjusted view of the planned work, it's going to be disappointing at best. There are some key root causes of most project failure. Let's start here.
Each of these has been shown, through research on failed programs, to contribute to cost and schedule impacts. Unrealistic expectations of the project's deliverables, Technical issues, Naive Cost and Schedule estimating, and less than acceptable risk mitigation planning.
Project Mangement in Six Steps
Here's how to address the cost and schedule estimating
Develop a schedule. What ever your feelings are for Gantt's, sticky note, or any handwaving processes you've learned to use, you need a sequence of the work, the dependencies, the planned durations. Without something like that , you have no idea what work is needed to complete the project. Here's a straightforward Master Schedule for some flight avionics on a small vehicle. All software, has to complete as planned, otherwise the users can't do their job as planned. And since they're the ones paying for our work, they have an expectation of us showing up, near oru budget, with the needed capabilities. Not the Minimum, the NEEDED
Using a Monte Carlo tool (RiskyProject), here is run for the probabilities of cost, duration and completion dates. All project work is probabilistic, any notion that a deterministic plan can be successful is going to result in disappointment.
We usually call our planning sessions done when we can get the risk adjusted Integrated Master Schedule to show the completion date of on or before the need date to an 80% confidence level.
With a resource loaded schedule - or some external time phased cost model - we can now show the probability of completing on or before the need date, and at of below the planned budget. The chart below informs everyone what the chances are for success for the cost and schedule aspects of the project. Technically it has to work. The customer gets to say that. The Fit for Use and Fit for Purpose measures if we're in an IT Governance paradigm. The Measures of Effectiveness and Measures of Performance if we're in other paradigms. Those measures can be modeled as well, but I'm just focusing on cost and schedule here.
With this information we can produce the needed margin and management reserve to protect the delivery date and budget. Showing up late and over budget for a product that works is usually not good business.
Do You Need all This?
What's the Value At Risk?
Don't know the value at risk? Don't have a clear vision of what done looks like in units of measure meaningful to the decision makers? You've got bigger problems. This approach won't help?
Jorion (2007) wrote that “Western Europe conquered the world because of a technological revolution that started from the attempts to measure the world.” In the same way, attempts to measure risk - (and its related project performance impacts) - more definitively, realistically, and accurately will surely lead to better project management. - thanks to: "Here, There Be Dragons: Considering the Right Tail in Risk Management," Christan B. Smart, Missile Defense Agency, Redstone Arsenal, Alabama, Journal of Cost Analysis and Parametrics, 5:65–86, 2012.
Making estimates of cost, schedule, and technical performance outcomes and their impacts to programmatic and technical risk in the absence of long tails is venturing into waters where Dragons live.
In the insurance business - and other financial domains - conditional tail expectations is applied to mitigate risk.
When questioning to perform or not perform some method of managing a project, select from a variety of features, or make any decision involving cost, schedule, or performance, ask first what's the value at risk? The answer is the basis of your decision. And making decisions without estimating these impacts creates even more food for the Dragon.
And for all projects in between, ask what am I willing to lose if I'm seriously wrong about the cost and schedule? What should I invest to decrease the probability to an acceptable level that I'm wrong about my estimates? That's one of the basis of Value at Risk.
When you hear we can make decisions about future outcomes in the absence of estimates - think how tasty you'll be to that Dragon when he eats you alive.
Plans are critical, a schedule to implement that plan comes next. With the Plan, the sequence of the work is needed to assure the value to the customer is delivered in the proper order. That order is an order because the business (or the mission) usually can't accept all the features and functions at once. So the Plan is the top level sequence, and the schedule is the next level down.
Many times I hear about Cost of Delay, Deliver Value, Measure story points, or Measure Stories. And a myriad of other assessments of project performance, all of which - OK, most of which are examples of Open Loop Control.
Back in 2014, we had a paper in a publication of the College of Performance Management, starting on Page 17. As well, a colleague Nick Pisano (CDR US Navy Retired) has a post on the same topic at his blog.
The notion of a baseline let alone a Performance Measurement Baseline is at the heart of Closed Loop Control of all processes, from your heating and air conditioning system in your house, to the flight controls on the 737-700 winging its way back home to Denver, to the project you're working - using what ever project management method or software development method of your choosing.
The notion that we can manage anything, the temperature of the room, the nice soft ride in the 737, or the probability of showing up on or before the need date, at or below the needed cost, with the needed capabilities - and NOT have a baseline to steer to is simply wrong.
Below is the framework for Closed Loop control. This paradigm says simply:
ESTIMATING IS THE BASIS OF DECISION MAKING - it can't be any clearer than that.
It was overheard on Twitter
"Don't fall in love with your plan, it is almost certainly wrong "
A plan is a strategy for success. Strategies are hypothesise . Hypothesise need tests to verify - just like you learned in High School science class. That test is the Measures of Effectiveness and Measures of Performance of the outcomes from your project as well as the Technical Performance Measures that are used to take corrective actions needed to reach the goal of delivering value in exchgange for time and money.
The Plan describes where we are going, the various paths we can take to reach our destination, and the progress or performance assessment points along the way to assure we are on the right path.
These assessment points measures the “maturity” of the product or service against the planned maturity. This is the only real measure of progress – not the passage of time or consumption of money.
Wrong in the planning sense can only be wrong if you are managing your project Open Loop with no assessment of Effectiveness, Performance, Risk reduction, cost absorption, time performance or all the ...ilities associated with spending other peoples money.
So it is true that the Plan is wrong if you're not managing in the presence of uncertainty, feedback, and taking corrective action.
Orion launched today and recovery after two orbits. The test of the launch system, Pad Abort system, and Heat Shield were the main purposes of the flight.
I worked the proposal - after coming off the winning proposal for Hubble Robotic Service Mission. The Crew Exploration Vehicle was the original name of the flight vehicle. The Integrated Master Plan and Integrated Master Schedule described the increasing maturity of the deliverables for the space craft and it's flight support systems. After the contract win, I moved to the flight avionics firm and defined the IMP/IMS and project performance management processes for that major subcontractor. When you get to minute 21:17, Tracking Data Relay Satellite is mentioned. I worked that project as a new graduate student many decades ago.
Starting back on TDRSS, agile - meaning emerging requirements, test driven development, direct customer feedback on short iterations - and the development process were deployed with rolling waves, 44 day rule Work Packages, and emergent technical requirements derived from Mission Capabilities.
Here's the long version of the launch to orbit.
After two orbits, Orion came home. The double boom is the sonic boom. Tests of the heat shield will confirm if it functioned properly.
Recently a statement was made about agile and complexity and it was conjectured if the project is too complex for a physical board - a place to put the stickies for the stories - then we've missed opportunities to simplify. Possibly not realizing that complexity, as well as complex system, are the norm in many domains and complexity management processes using tools - rather than manual means - is also the norm.
If your Agile planning needs are too complex for a physical board, you've probably missed opportunities to simplify / improve.
When I suggested that agile and agile tools are used to deal with complex problem in these environments, without the need to reduce that complexity, there was a conversation of sorts that suggested...
I'd be surprised to hear Orion was using a COTS Agile project management tool in a significant way
Some Necessary Complexity
On Hubble mission, there is a Service Mission Assurance Process that reveals some of the complexity of the System of Systems found in space flight. The Interface Control processes for example for the payload on STS 125.
External knowledge of what tools were used, what processes were applied, how the flight avionics software for Orion was converted from the 777 suite to the spacecraft suite, tested, altered to user needs, simulated, emulated, verified and validated on rolling waves, on 44 day iteration cycles could have only been obtained if you were actually in the building in the vendors shop.
But there are other surprises in the business of space flight. A few good places to start include:
Beyond the outsiders comments of surprise inside space and defense firms, agile tools from Rally, VersionOne, and JIRA are used in a wide variety of domains from embedded systems to intelligence systems, where the requirements don't come from the users, they come from the enemy. Here's an example of agile in the INTEL business.
Maybe those surprised by the many different applications of the principles of agile - developed long before the Agile Manifesto - missed those processes in Building O6, Sepulveda Blvd, Redondo Beach, circa 1978.
In The End
There are numerous approaches to applying agile development in a wide variety of domains. I work in a domain where Systems Engineering and Earned Value Management is the starting point and Agile is used to develop code guided by EAI-748-C and DID 81861.
In these environments, development of software is incremental and iterative, with emerging requirements, with stable capabilities. These programs are complex and tools are the basis of success for managing all the moving parts of the program. Rarely is everyone in the same room, since these are System of Systems programs. As well Integration and Test are done by external sources - V&V for flight safety. So many of the processes found in small commercial projects are not applicable to programs in our domain.
To suggest there is but one way to reduce complexity by putting all the stories on cards on the wall is a bit naive in the absence of establishing the external needs of the project first, then deciding what processes to apply.
Some background on applying agile in the DOD can be found at:
Domain first, Context second, Only then Process
Working in Phoenix this week on a Program Management Office deployment for a government agency providing public health services. We're installing processes and training staff on the notion of Measures of Success (a term I had not heard before), that have close resemblance to the Integrated Master Plan (IMP).
Here's a paper given at the Energy Facilities Contractors Organization (EFCOG) on a similar topic.
There are several critical points here:
Estimates are difficult. When requirements are vague — and it seems that they always are — then the best conceivable estimates would also be very vague. Accurate estimation becomes essentially impossible. Even with clear requirements — and it seems that they never are — it is still almost impossible to know how long something will take, because we’ve never done it before.
This is going to turn out to be true. The project is going to spend money on discovering the requirements that delivery the capabilities. This is the case sometimes, but that cost needs to be accounted for in the business case.
The last sentence above is actually not the case, expect where the development team has no experience in the business or technical domain. And in that case, why did you hire people to spend your money when they've never done this kind of work before?
Systems Engineering has two components
When we work in the Enterprise IT domain or any Software Intensive Systems ...
...systems engineering is focused on the system as a whole; it emphasizes its total operation. It looks at the system from the outside, that is, at its interactions with other systems and the environment, as well as from the inside. It is concerned not only with the engineering design of the system but also with external factors, which can significantly constrain the design. These include the identification of customer needs, the system operational environment, interfacing systems, logistics support requirements, the capabilities of operating personnel, and such other factors as must be correctly reflected in system requirements documents and accommodated in the system design. [Systems Engineering Principles and Practices, Alexander Kossiakoff, John Wiley & Sons]
So what does this mean in practice?
It means when we start without knowing what DONE looks like, no method, no technique, clever process is going to help us discover what DONE looks like, until we spend a pile of money and expend a lot of time trying out various ideas in our search for DONE.
What this means is that emergent requirements, mean wandering around looking for what DONE looks like. We need to state DONE in units that connect with Capabilities to fulfill a mission or deliver success for a business case.
What this doesn't mean is that we need the requirements up front. In fact we may not actually what the requirements up front. If we don't know what DONE means, those requirements must change and that change costs much more money then writing down what DONE looks like in units of measure meaningful to the decision makers.
So Here's Some Simple Examples of What A Capability Sounds like
Here's a more detailed example
Identifying System Capabilities is the starting point for any successful program. Systems Capabilities are not direct requirements, but statements of what the system should provide in terms of “abilities.”
For example there are three capabilities needed for the Hubble Robotic Service Mission:
How is this to be done and what are the technical, operational, safety and mission assurance requirements? Don’t really know yet, but the Capabilities guide their development. The critical reason for starting with capabilities is to establish a home for all the requirements.
To answer the questions:
Capabilities statements can then be used to define the units of measure for program progress. Measuring progress with physical percent complete at each level is mandatory. But measuring how the Capabilities are being fulfilled is most meaningful to the customer. The “meaningful to the customer” unit of measures are critical to the success of any program. Without these measures the program may be cost, schedule, and technically successful but fail to fulfill the mission.
This is the difference between fit for purpose and Fit for Use.
The process flow below is the starting point for identifying the Needed Capabilities and determining their priorities. Starting with the Capabilities prevents the “Bottom Up” requirements gathering process from producing a “list” of requirements – all needed – that is missing a well formed topology. This Requirements Architecture is no different than the Technical or Programmatic architecture of the system.
Capabilities Based Planning (CBP) focuses on “outputs” rather than “inputs.”
These “outputs” are the mission capabilities that are fulfilled by the program. Without the capabilities, it is never clear the mission will be a success, because there is no clear and concise description of what success means. Success means providing the needed capabilities, on or near schedule and cost. The concept of CBP recognizes the interdependence of systems, strategy, organization, and support in delivering the capability, and the need to examine options and trade‒offs in terms of performance, cost and risk to identify optimum development investments. CBP relies on Use Cases and scenarios to provide the context to measure the level of capability.
Here's One Approach For Capturing the Needed Capabilities
In Order To Capture These Needed Capabilities We Need To...
What Does All This Mean?
When we hear of all the failures of IT projects, and other projects for that matter, the first question that must be answered is
What was the root cause of the failure?
Research has shown that unclear, vague, and many times conflicting requirements are the source of confusion about what DONE looks like. In the absence of a definitive description of DONE in units of effectiveness and performance, those requirements have no home to be assessed for their appropriateness.
The book on the left is the latest addition to this topic in the domain of agile software development. The book is based on an Adaptive Complex Project Framework. The notion, a naive notion, that complexity can be reduced and complex systems should be avoided, is just that notional. In practice complex systems can't be avoided in any business or technical domain where mission critical systems exist. That is non-trivial systems are complex.
The book emerged from a 2010 IBM report of 1,541 executives in 60 countries about the preparedness for complex systems work. Capitalizing on Complexity. From the report there are ten Critical Success Factors, in priority order:
In a sufficiently complex project we need measures of progress to plan beyond burning down our list of same sized stories, which by the way require non-trivial work to make same sized and keep same sized. And of course if this same size-ness does not have a sufficiently small variance all that effort is a waste.
But let's assume we're not working on a sufficiently small project where same sized work efforts can be developed, we need measures of progress related to the Effectiveness of the deliverables and the Performance of those deliverable in producing that effectiveness for the customer.
Here's a recent webinar on this topic.
I came across a nice blog post from DelancyPlace about the navigation powers of birds.
This bird was taken from Wale's to Venice Italy and released and found it's way home in 14 days. 930 miles, over mountains .
To be able to find their way home from an unfamiliar place, birds must carry a figurative map and compass in their brains.
The map tells them where they are, and the compass tells them which direction to fly, even when they are released with no frame of reference to their home loft.
Projects Are Not Birds
As project managers, what's our map and compass? How can we navigate from the start of the project to the end, even though we haven't been on this path before.
How can we find our way Home?
We have a map. It starts with a Capabilities Based Plan. The CBP states what Done looks like in units of measure meaningful to the decision makers. These units of measure are Measures of Effectiveness and Measures of Performance.
Measures of Effectiveness - are operational measures of success that are closely related to the achievements of the mission or operational objectives evaluated in the operational environment, under a specific set of conditions.
These measures speak to our home and the attributes of the home. The map that gets us ti home is the Integrated Master Plan. This shows the increasing maturity of the deliverables that implement the Measures of Performance and those Performance items that enable the project to produce the needed capabilities that are effectively accomplish the mission or fulfill the business need.
This looks like a map for the increasing value delivery for an insurance company. The map shows the path or actually paths to home. Home is the ability to generate value from the exchange of money to develop the software.
The Death March project starts when we don't know what DONE looks like. Many of the agile approaches attempt to avoid this by exchanged not knowing for budget and time bounds. In the enterprise IT domain, those providing the money, usually have a need for all the features on a specific date to meet the business goal.
ICD-10 go live, new product launch enabled by new enrollment, Go Live of new ERP company wide, with incremental transition across divisions or sites. Maintenance support systems for new fielded products on or before products put into service - are examples of all features, on budget, on schedule.
The elicitation of the underlying technical and operational requirements has to be incremental of course, because knowing all the requirements upfront is just not possible. Even in the nth install of ERP at the nth plant, there will be new and undiscovered requirements.
It's knowing the needed Capabiliities of the system that are the foundation of project success.
Here is a top level view of how to capture and use Capabilities Based Planning in enterprise IT.
In a previous post on How to Assure Your Project Will Fail, the notion that the current project management processes are obsolete and the phrase of dealing with complexity on projects is a popular one in the software domain. By the way that notion is untested, unreviewed, and is missing comparable examples of it working outside the specific references in the orginal paper.
But here's the simplest approach to deal with project complexity...
Don't Let The Project Get Comnplex
Nice platitude, It's that simple and it's that hard.
Before the nashing of teeth is heard, here's a working example of not letting the project get complex.
So how is this possible? First let's make some assumptions:
Here's the steps to dealing with project complexity that have been shown to work in a variety of domains:
Let's pause here for a process check. If there is no narrative about what DONE looks like in units of measure meaningful to the decision makers (MOE, MOP, TPM, KPP) then the project participants have no way to recognize DONE other than when they run out of money and time.
This is the Yourdon definition of a Death March project. Many who use the term complexity and complex projects are actually speaking about death march projects. We're back to the fundamental problem - we let the project become complex because we don't pay attention to the processes needed to manage the project to keep it from becoming complex. Read Yourdon and the Making the Impossible Possible: Leading Extraordinary Performance - The Rocky Flats Story books to see examples of how to keep out of this condition.
When hear the notion that chaos is the basis of projects in the software world - run away as fast as you can. That is the formula for failure. When failure examples are presented in support of the notion that chaos reigns and there is no actual, verifiable, tangible, correctable Root Causes listed - run away as fast as you can. Those proposing that idea have not done their home work.
But the question of dealing with complexity on projects is still open. The Black Swans, that get misused in the project domain, since the term refers to the economics and finance domain through Taleb, may still be there. There are there because the project management process have choosed to ignore them, can't afford to seek them out, or don't have enough understanding to realize they are actually there.
So if Black Swans are the source of worry on projects, you're not finished with your project management planning, controlling, and corrective actions duties as a manager. Using project complexity as the excuse for project difficulties is easy. Any one can do that.
Taking corrective actions to eliminate all but the Unknowable uncertainties? Now that's much harder.
The needed capabilities for business or mission success starts with the Concept of Operations. With the ConOps we can identify the needed capabilities. Below is an actual project, with increasing capabilities for an insurance firm. The order of these delivered capabilities are defined by the business strategy and the Balanced Scorecard. Both define needed capabilities, the value stream map for producing business value or mission fulfillment and the order in which these capabilities should arrive to sustain the needed business benefits from the project. The by the way the mechanism for prioritizing features, which are much lower in the hierarchy of project elements that are commonly referred to in agile community. Features fulfill requirements, requirements implement capabilities.
Customer didn't buy features or requirements, they buy the capability to do something to improve their business.
A useful method of managing the project for delivering these capabilities is the Integrated Master Plan and Intetgrated Master Schedule.
These 5 questions need credible answers in units of measure meanigful to the decision makers.
What Does All This Mean?
With these top level questions, many approaches are available, not matter what the domain or technology. But in the end if we don't have answers the probability of success will be reduced.
Twas the Night Before Baseline
(With apologies to Clement Clarke Moore)
Twas the eve for the holidays,
And all through the shop,
Our consultants were working,
Would they ever get to stop?
All the CAMs had corrected
their IMSs with care,
With hopes that the IBR interviews
Would not be nightmares.
When all of a sudden
There arose such email chatter,
IMS updates were piling up,
So which ones would matter?
And what did I see
To my eye’s disbelief?
But even more IMS updates
In BoxNet motif!!
But jolly fat fingers
flew over keyboards,
And corrections were entered
Errors vanquished evermore!
Now CAs! Now WADs!
Now WPs! Now RAMS!
On CARs! On Baselines!
On Schedules! On CAMs!
To the TMT briefings,
To the top TMT Generals,
Now CAM away, CAM away, CAM away all!
Courtesy of Jay Charleston, CAM on a DTRA Anti-Virus Program. Our team was literally working the baseline submittal in late December for an IBR in mid January. Only the strong survive the 72 hours straight of re-baselining $350M a bio-pharma drug development program.
There are endless discussions about what went wrong with the Affordable Care Act web site development and deployment. It'll be hard to tell at this early point in the project assessment. But what is clear is this was mist likley a failure of project management.
Below is the acquisition life cycle for Business systems in the DOD, not that HHS is a DOD-style shop, but the paradigm of iterative and incremental development is in place. The release cycles shown here are way too long for something like the ACA Site. But to topology of the process is sound.
Looking at this process there is an obvious starting point. The Business Capability Definition. What is the resulting system supposed to do in terms of capabilities. Not the technical and operational requirements, but what business capabilities will the system provide to the stakeholder when it is in full operation? This is called Initial Operating Capability (IOC).
In our domain we start defining the capabilities using the Defense Acquisition Guide. Here is where Measures of Effectiveness (MoE) are defined. The Measure of Effectiveness is assigned to a capability. If we want a capability, how effective does it have to be? This measure is not a technical performance or a requirement. It is an effectiveness measure.
A MoE for a UAV program we work would be The UAV shall be transportable within a 3,000 mile radius via a C-17/C-141/C-5 package. From the MoE there is a Measure of Performance (MoP). For example weight is a MoP that enables the MoE to be fulfilled. Lower down are Technical Performance Measures (TPM). For example the weight of a Electro Optical / Infrared sensor platform must be under 55 pounds for the UAV to operate properly. It can't be too light or it would disrupt the center of gravity and can't be too heavy because the UAV would burn too much fuel to accomplish it's mission.
So for the ACA site, we'd need to know if there were MoE's, MoP's, TPM's defined that enable the Capabilities to be delivered. Here's the Performance Reference Model for federal IT.
Since the ACA site is pretty much all software, I'm going to suggest that this approach of using Capabilities Based Planning, MoE's, MoP's, TPM's has nothing to do with how the software is built. Either traditional or agile methods can be used. Agile is likely faster, but agile can only work in a domain like this if you know what DONE looks like in terms of MoE's, MoP's, and TPM's. This is a fixed launch date, fixed set of requirements guided by all the insurance regulations, and hopefully some not to exceed budget.
It is a common myth that government acquisition is waterfall and big design up front. DoD 5000.02 prescribes an iterative process designed to assess the viability of technologies while simultaneously refining user requirements. (pg 16 of 5000.02).
One starting question of the ACA Site would be - did they apply the iterative acquisition process in some form, no matter the fidelity of the iterations?
Here are some other fundamental questions as well:
If the answer to any of these is no or we don't know, go find out, get project managers who can do this. Otherwise the probability of project success is reduced. In fact look at the Probability of Program Success literature for further guidance.
The final question is did they have an Integrated Master Plan and Integrated Master Schedule for all the work as described in the Integrated Master Plan and Integrated Master Schedule Preparation and Use Guide? This paradigm has been shown to significantly increase the probability of success not matter the domain, context, development method, technology, or business process. It states in clear, concise, and unequivocal terms what DONE looks like at every point in the project in units of measure meaningful to the decision makers.
The final - and killer question is - did the project team ruthlessly manage the changes to the capabilities? This is suspect is the root cause of the failure. Late changes to complex projects are the kiss of death.
As repeated often here...
Don't do stupid things on purpose
So Now What?
We have to wait to see what the Root Cause Analysis (RCA) shows for the failure of the project. But I'd conjecture the program management processes found in large DoD or NASA programs where not applied in any meaningful way. The site is not larger compared to most of the programs we work ($400M is small), but the processes used to manage those programs can be scaled down with ease. The Principles are the same. The Practices are scalable and the Processes scalable as well.