The Project Management Manifesto for America is the starting point for improving the project management of the American Recovery and Reinvestment Act.
Principles, Practices, and Processes to Increase Probability of Project Success
During a discussion – of sorts – with Mary Poppendieck about Chapter 2 of her upcoming book, I suggested several ideas needed clarification regarding the use of "Waterfall" in modern project management. Mary forwarded a paper – an opinion piece actually – from Denning, Gunderson and Hayes-Roth titled "The Profession of IT: Evolutionary System Development," Communications of the ACM, Vol 51, No 12, December 2008.
Mary's book by the way looks like good reading – as usual. It is completely my issue, when I get wrapped around the axle about the details of how information is being presented. The perfunctory simplification of a topic for editorial reasons, or simply because of one's point of view. The use or mis use of Earned Value and the understanding or misunderstanding of such a topic is typical of the exchange between agilest and traditionalist. Dave Nicolette has suggested a discussion of the irreducible principles of project management to turn the light on the differences between principles and practices and that agile and conventional practice have much in common.
In this article they suggest that the failure of large systems is caused by inability to rapidly adapt to the changing environments. They provide an example of a system built within the U.S. Department of Defense using an agile approach:
The W2COG is a self-selecting collaborative community of experts formed among federal agencies, DoD, and commercial firms to enable agility, aka "netcentricity," via adaptive, open, SOA by removing the traditional barriers to cross-stovepipe collaboration.
W2COG projects …"include netcentric studies, demonstrations, and prototypes of bundled off-the-shelf capabilities validated against operational use cases. These are essentially "reference designs" if you were in the hardware/firmware platform business.
One of the backgrounders states …
Joint Interoperability Test Command (JITC) employs the W2COG Institute (WI), a government and industry expert body established by OSD, to serve as a computer network-enabling "Capability Broker." Accordingly, the WI has designed a "Mission Thread Market" (MTM) process to incentivize sustained COTS software competition around government use case requirements in 90 day production cycles. In particular, Government seeks to incentivize industry to bind innovative SOA solutions to government-furnished high assurance services, e.g. for authentication and authorization.
The ACM paper goes on to state several things that need some examination:
So like many topics in our domain, simple answers usually come from simple understandings. When in fact the actual understanding is much more complex. So I decided to do a little weekend research.
The authors provide no references for these assumptions. So it must be assumed these are their personal assumptions. They then proceed to use these assumptions with the anecdotal examples of failed systems. Several examples of failed large systems include:
Examples of successful projects using evolutionary design include the World Wide Web and Linux.
Comparing Apples to German Sheppard's
While there are well document failures in the large complex systems development business, providing a counter example of the WWW and Linux seems curious. The web is certainly evolutionary as is Linux. But these were projects whose market was evolutionary and emergent as well. The FBI Virtual Case File systems and the NMCI system had specific operational needs, with measurable business and mission benefits. Not the same as a solution looking for the problem. As a very early user of DRAPA – Net at Wright Patterson Air Force Base, we had "interest" in communicating over a network that was not dial up. But I doubt anyone who was a user at our level – FORTRAN signal processing programs using DEC-10 running BBN's Tenex. There were visionaries for sure. It wasn't us though.
But the FBI VCF and to some part NMCI were purpose built systems, with the "capabilities" predefined – solve this problem in this manner.
This by the way is one of those diversionary approaches by some in the agile community. Use an example of poor or even downright bad management as the counter example for some agile techniques.
The FBI Virtual Case File
The VCF failures are well documented:
GAO reports states … "analysis of available performance data, however, showed that the Navy had met only 3 of 20 performance targets (15 percent) associated with the program's goals and nine related performance categories. By not implementing its performance plan, the Navy has invested, and risks continuing to invest heavily, in a program that is not subject to effective performance management and has yet to produce expected results."
... "analysis also showed that the contractor's satisfaction of NMCI service level agreements (contractually specified performance expectations) has been mixed."
… "analysis further showed that NMCI's three customer groups (end users, commanders, and network operators) vary in their satisfaction with the program."
Sounds like your typical commercial enterprise project doesn't it? One telling statement in the introduction of the GAO report states …
A plan that the Navy developed in 2000 to measure various aspects of the program, and thereby gauge program goal attainment, has not been implemented, and associated performance reports have not been issued. According to Navy officials, implementing this plan has not been as high a priority as, for example, deploying NMCI and measuring contractor performance. While program officials told us that NMCI has achieved much, they were unable to provide performance data to demonstrate these achievements relative to either the program's strategic goals or the nine performance categories that its 2000 performance measurement plan and other initiatives defined for these goals.
Gee no plan to measure the performance of the various aspects of the program? Sound familiar.
Why And How Can Evolutionary Approaches Correct Poor Management?
The authors don't actually say how. They do reference the W2COG project. Remember this is an integration process of openly available COTS products of a communications system. Now there is a poster child for failure in that area as well. The World Wide Military Command and Control Information System (WWMCCIS), http://archive.gao.gov/d47t13/116661.pdf.
There are some interesting causal issues here:
In the end the killer statement from this 1980 report is …
Without first identifying the critical data elements needed to carry out assigned missions, the usual result is a system design that does not completely or efficiently meet the needs of all its users. Identifying these data elements before system design and acquisition is essential to ensure proper system sizing and capacity, and hence responsiveness, as well as providing clear guidance to software developers. For these reasons, identification of detailed information requirements--the essential data elements and their associated qualities of timeliness, accuracy, and currency-is incorporated in both DOD and the Office of Management and Budget directives on developing automated systems.
So Why Would Evolutionary Systems Alone Help?
Several fundamental axioms of system development include:
The WWW did not need to operate under these constraints.
So How Can the Best of Both Worlds Be Combined?
This is an ongoing question. No doubt agile has much to offer for the development of software. But for large, complex, mission critical, enterprise systems, can agile alone provide all that is needed for success. The ACM article makes that case, in the absence of evidence though.
Using the NMCI example (an integrated internet solution for the Navy and Marines). How would an evolutionary system development work in the field? In the field of combat Marines with Navy support? Would the Marines get partial capabilities, then upgrade to more capabilities over time? How those upgrades would be managed. Would everyone have to have the same version of the system in order to communicate? Good questions? NMCI is probably not performing as planned. "Users find it does not meet requirements, is inflexible, and has long waits for trouble assistance" [Marine Corp Gazette, Dec 2008]. OK, and evolutionary deployment will fix this?
Maybe. Maybe not. The first problem seems to be no one wants to admit there is a problem. How exactly would evolutionary development make the problems visible in ways not currently being done.
And The Answer Is?
Establish an irreducible set of project management principles. If these principles cannot be found in the programmatic aspects of the project, the project will likely be headed for failure. What are these irreducible set of principles?
Rapid adaptation is nice. But within what principled boundaries. Do the Marines get to choose what machines they use, what networks they communicate over, what software gets loaded on those machines? Probably not.
Did the FBI VCF project come unglued because it was not adaptive to change? That's not what the report says. Only mention of the poor project management processes, manipulative vendors, simply bad leadership.
In The End
When the discussion continues to mix software development methods with project management, we will continue to get sideways with overlapping methods. I'm not hopeful this will be resolved. PMI is starting to use the term "agile." At one time I was adamant that process and product was the same thing in the project management space.
I no longer believe this is a good idea – project management processes (project controls) and product development (software development) are separate - both needed, but not interchangeable.
Bill Duncan is currently spewing out his breakfast coffee at the sound of this admission.
Project accounting is many times confused with project management.The Project Manager is most interested in Schedule. Project accounting is interested in Cost. The critical success factor here is the derive the project costs from the project schedule. Most often we speak about "cost and schedule," it should actually be "schedule and cost." This is subtle but important:
Here's the essential elements of Program Planning:
The Red herring of Waterfall has risen again as the favorite stalking horse for agilest when trying to find some way to sell their approach. There are several issues with this tactic, not the least of which is Waterfall as described by an agilest is actually prohibited in the government sector. The second issue comes from the favorite reference to the Royce's paper on Waterfall (father and son). Ignore the details of the paper and use it as an example of how Waterfall was pulmigated and still in use. But you need to read the fine print in this paper to discover iterations, customer involvement, "do it twice," feedback up and down the line, etc. etc. For 1970 this was far out stuff.
How It's Done on a Larger Scale Today
The actual top level procurement process for government (DoD) acquisition is an incremental and iterative approach described in the Integrated Defense Acquisition, Technology, and Logistics Life Cycle Management System. Notice all the loops in the "W" style systems engineering process down in the Technical Section. Of course this approach should NEVER EVER be considered outside projects larger than $20M. Of course $20M is chump change in the DoD world. But the point is NEVER NEVER speak of this process in any form when discussing procurement or development process in the commerical world.
But at the same time the Waterfall Red Herring is not allowed even in this world. So why on God's green earth do agile thought leaders speak of Waterfall as the process they are trying to replace. It's simple marketing really. Find the ugliest, least successful, worst track record approach to developing software and use that as your comparison. Never mind that all the "evil" outcomes of the described process are actually BAD Project Management Practices.
In the end ANY project management method - from "magic beans" to DoD IMP/IMS using ANSI-748 Earned Value needs to have an irreducible set of PRINCIPLES on which to base their PRACTICES. Principles and Practices are different. I already spoken to the irreducible principles that can be found in PMBOK
So the question can be asked again. What principles of project management would NOT be found in any credible project management practice? I might suggest Procurement would be one. But I'm stumped to think of any others.
Separating Principles from Practices
So when the latest critique of "waterfall" starts out the discussion rarely if ever is about Principles. It's almost always about practices - and bad practices at that. It's like complaining you can't loose weight while eating Big Macs morning, noon, and night. That !@#$'ing MacDonald's diet just doesn't work. Really? You Think?
Look at the Principles here and ask which ones you would not want in your favorite project management practice. Now the practices can range from back of a napkin, white board or some other favorite media - to a full blown SAP (heavens no, please) based program management process framework.
I might conjecture if you ask and answer the project management process questions in this manner all the "us versus them," the "they were really not very smart in the past," and other opening apporoaches to project management replacements go away. And what you're left with is an assessment of th type of Practices best suited for your domain and context.
Jason Yip has a nice post describing the status expected from an Agile project. From the point of view of an agile iteration these are:
From a Deliverables Based Planningsm approach these are:
The "red herring" of waterfall is rising again. The proponents of agile use waterfall as the whipping boy (or girl) for what's wrong with software development processes. What is completely lost here is:
All processes are serial in the end. Steps are taken one at a time, in some order. The old saw:
In the control systems world - and one of the anti-serial process authors claims to have been a control systems engineer - the sampling time in the control loop determines how responsive the loop is to change.
Of course it's not quite that simple, but that's the rudimentary principle. You're house thermostat has a few minute sample time, plus a delay for restart, plus a dwell time for shut down. Same for the air conditioner compressor. Your cruise control has a difference sample-compute-response behavior. Our crappy 1997 Suburban cruise control used to hunt when climbing I-70 west of Denver. It was not tuned to the engine, the load, or the grade, but taken from the Chevy Impala - like the A/C compressor. The flight controls for a 777 have difference response times than the controls for Joint Strike Fighter.
A "design, code, test" cycle in months has a different response time than one that samples the resulting output in a month, two weeks, or even a day.
All processes are serial in some form. Parallel activities can take place, but the value stream is usually serial. If it were parallel there are concurrency issues, integration stabilization point issues and a hoard of other artifacts caused by parallel developments.
Any control system engineer knows this. You can learn more starting here.
Here's the essential elements of the organizing processes for a project:
There are several posts out in the last week describing planning and scheduling methods. Some are useful, some are repeats of the obvious, a couple are simply wrong and one is nonsense. Some are based on PMI's PMBOK (which by the way is not really organized around a method, but is a collection of processes and knowledge areas that can or shoudl be found in a credible method).
I'll do this in a series of posts, describing the core elements of any planning and controls method. My motivation here is Paul Ritchie's "check list" post. The notion of a check list is obvious, but overlooked. The elements in my "check list" are guided by the Earned Value Management framework:
No matter what project management method is used, these 5 elements need to be present in some form. These are not the technical aspects of the project, but are the programmatic elements. Whether you're using full DoD IMP/IMS, some agile method of development, or so ad hoc made up "magic beans" these needs to be there.
Why? Good question. There is not simple answer of course, but here are some starting answers. The stakeholder needs know at least four things at all times:
The notion that there are simple project management processes is just that "notional." There are straight forward processes. There are important and less important processes. There are processes more complex than others. But "simple" usually means simple results, simple understanding, simple results.
I've talked about the Irreducible Program Controls elements in the past. Here's my current thoughts on the Irreducible Program and Project Management Principles:
These are principles, so remember ...
The urban myth going around the office is that an egg can be stood on its end two times a year, at the equinox. The equinox of course is when there is equal light and dark and as nothing to do with gravity and egg standing tricks.
While looking on the web for "myth buster" articles for my colleagues, I was reminded of some long ago experiences. While a graduate study I was privileged to be in the presence of two icons of the physics world. Our principle investigator Fred Reines and a frequent visitor Richard Feynman.
Here's some videos of Feynman, that provide insight into one of the 20th century's best thinkers. Listen to the thought processes and the development of understanding in the presence of mystery.
The Department of Defense Earned Value Management Implementation Guide, October 2006 says:
That's it, other than the other 96 pages of detail in the document.
But notice there is no "business value" phrase in this statement. The production of Business Value is a separate issue. An important issue, but completely separate topic. Earned Value Management (EVM) is NOT about business value creation and management. No matter how much the agile thought leaders would like it to be. It is about knowing the performance efficiency of the planned budget (BCWS) in terms of Percent Complete within the planned period of performance to produce the "Earned Value," BCWP.
When it is stated
The ... claim is that EVM enables you to objectively measure the value being produced by project teams. Their inherent assumption is that these project teams are actually earning value for your organization, yet in practice we know that this couldn't possibly be true.
This is simply wrong. The value of Earned Value is not business value, it is the "earned" dollars (BCWP)(EV) from the "planned" dollars (BCWS)(PV).
All the proposed enhancements with burn down, burn up, business value are outside the scope of earned value as it is practiced by the earned value professionals.
Simple Training Example
Here's a Gentle Introduction to Earned Value Management, that may help clarify the concept of "earning value," while NOT speaking about the business side - that is the cost of the cokkies. Beacuse of course the cookies are not sold, they are Christmas Cookies.
I was poking around for an EV document I had misplaced and came across a 2008 article by Scott Abler in Doctor Dobb's Journal. Scott is a thought leader in many areas of software development. I used his "process" books for many years when I was managing development organizations. I was troubled though by the misunderstanding of a core concept in Earned Value.
The premise that the "value" in Earned Value, is monetary value to the firm is not the case. The "V" in EV, is the measure of physical percent complete in units of measure of money. The money of the budgeted cost of work scheduled (BCWS). That is how much of the BCWS was converted into tangible items – physical percent complete. Whether the functional specification has are "balanced sheet" value to anyone is a local issue. If may be invaluable to NASA for the design of a manned vehicle. It may be worthless to the CIO who is on the line for making a service work.
The Value in Earned Value is not business value
For the government, it's an indication of how much "physical percent complete" has been accomplished to the "planned percent complete." Our firm manages software development projects in aerospace and defense and large IT (enterprise ERP) were EV provides measures of progress in ways not available in the absence of EV. These developments – even in A&D – make use of many of the principles of agile development, along with CMMI process areas.
In a later paragraph you mention EV effective if you have an effective plan. The opposite is actually true – if your plan is not effective you don't need project performance measures, because your project won't last long. Changes to the Performance Measurement Baseline (PMB, the source of BCWS) occur all the time. The need for a PMB, to show where we have come from and where we are going is the key to success of EV. Without some form of "baselining" the team is simply spending some else's money without accountability – this is a common failure mode for IT projects. In the EV world, this is called Level of Effort (LOE) – or "train watching" – actually "watching the train wreck." In LOE, the consumption of funds = progress.
BRUF, BDUF, big anything up front is not a prerequisite for EV. This is a popular myth of those unfamiliar with applying EV. Just the opposite in fact. Rolling Wave Planning Packages are part of DOD 5000.02. Only the open active work packages are used to perform measurements for management decisions.
If you invert your last paragraph of Scotts article and remove a few NOT's you've got how EV is done in A&D software development and Enterprise ERP.
Answering the question – What Does Done Look Like – means defining done in some unit of measure meaningful to both the customer and the project team. Physical Percent Complete of a tangible deliverable is a good place to start. Testable Requirements is the formal approach to this outside the agile world. No matter the approach the legal definition is "evidentiary materials." Show me something that you said you were going to show me, and show that it is "done." Not almost done, not "I'm gonna fix the bugs over the weekend," not "well I worked on my machine." No Done, means you won't have to touch this again unless the requirements change. We can ship it. to the customer or the SCC system.
Now how much did we plan to spend for this "done-ness?" If it is completely done, then you get 100% credit for being 100% physically percent complete. BCWP = BCWS X Physical Percent Complete. The actual cost to do the work is a cost management issue, not a schedule management issue.
If you spend the planned budget but were not 100% done, then several things result.
These are all correct responses. Making them visible is the role of Earned Value.