During a discussion – of sorts – with Mary Poppendieck about Chapter 2 of her upcoming book, I suggested several ideas needed clarification regarding the use of "Waterfall" in modern project management. Mary forwarded a paper – an opinion piece actually – from Denning, Gunderson and Hayes-Roth titled "The Profession of IT: Evolutionary System Development," Communications of the ACM, Vol 51, No 12, December 2008.
Mary's book by the way looks like good reading – as usual. It is completely my issue, when I get wrapped around the axle about the details of how information is being presented. The perfunctory simplification of a topic for editorial reasons, or simply because of one's point of view. The use or mis use of Earned Value and the understanding or misunderstanding of such a topic is typical of the exchange between agilest and traditionalist. Dave Nicolette has suggested a discussion of the irreducible principles of project management to turn the light on the differences between principles and practices and that agile and conventional practice have much in common.
In this article they suggest that the failure of large systems is caused by inability to rapidly adapt to the changing environments. They provide an example of a system built within the U.S. Department of Defense using an agile approach:
The W2COG is a self-selecting collaborative community of experts formed among federal agencies, DoD, and commercial firms to enable agility, aka "netcentricity," via adaptive, open, SOA by removing the traditional barriers to cross-stovepipe collaboration.
W2COG projects …"include netcentric studies, demonstrations, and prototypes of bundled off-the-shelf capabilities validated against operational use cases. These are essentially "reference designs" if you were in the hardware/firmware platform business.
One of the backgrounders states …
Joint Interoperability Test Command (JITC) employs the W2COG Institute (WI), a government and industry expert body established by OSD, to serve as a computer network-enabling "Capability Broker." Accordingly, the WI has designed a "Mission Thread Market" (MTM) process to incentivize sustained COTS software competition around government use case requirements in 90 day production cycles. In particular, Government seeks to incentivize industry to bind innovative SOA solutions to government-furnished high assurance services, e.g. for authentication and authorization.
The ACM paper goes on to state several things that need some examination:
- Dependable large system can only be attained through rigorous application of software engineering design process (requirements, specifications, prototypes, testing, and acceptance).
- The key design objective is an architecture that meets specifications derived from knowledgeable and collectable requirements.
- Individual of sufficient talent and experience can achieve an intellectual grasp of the system.
- The implementation can be completed before the environment changes very much.
So like many topics in our domain, simple answers usually come from simple understandings. When in fact the actual understanding is much more complex. So I decided to do a little weekend research.
The authors provide no references for these assumptions. So it must be assumed these are their personal assumptions. They then proceed to use these assumptions with the anecdotal examples of failed systems. Several examples of failed large systems include:
- FBI Virtual Case File – a project that violated every single principle of project management on the planet.
- NMCI – a project that "failed to meet its two strategy goals," according to the GAO Report titled:"TECHNOLOGY: DOD Needs to Ensure That Navy Marine Corps Intranet Program Is Meeting Goals and Satisfying Customers," http://www.gao.gov/new.items/d0751.pdf
Examples of successful projects using evolutionary design include the World Wide Web and Linux.
Comparing Apples to German Sheppard's
While there are well document failures in the large complex systems development business, providing a counter example of the WWW and Linux seems curious. The web is certainly evolutionary as is Linux. But these were projects whose market was evolutionary and emergent as well. The FBI Virtual Case File systems and the NMCI system had specific operational needs, with measurable business and mission benefits. Not the same as a solution looking for the problem. As a very early user of DRAPA – Net at Wright Patterson Air Force Base, we had "interest" in communicating over a network that was not dial up. But I doubt anyone who was a user at our level – FORTRAN signal processing programs using DEC-10 running BBN's Tenex. There were visionaries for sure. It wasn't us though.
But the FBI VCF and to some part NMCI were purpose built systems, with the "capabilities" predefined – solve this problem in this manner.
This by the way is one of those diversionary approaches by some in the agile community. Use an example of poor or even downright bad management as the counter example for some agile techniques.
The FBI Virtual Case File
The VCF failures are well documented:
- Lack of a strong blue print from the outset led to poor architecture decisions – what does "done" look like?
- Repeated changes in specification
- Repeated turnover of management
- Micromanagement of the developers
- Use personnel with little or no domain experience for software development
- Scope creep – continuous adding requirements while at the same time experiencing schedule delays and cost overruns
- Code bloat due changing specifications
- Adding more people as the project fell behind. An explicit violation of Brooks' Law.
- Planning to use a hard cut over, removing any chance to "try before buy"
NMCI
GAO reports states … "analysis of available performance data, however, showed that the Navy had met only 3 of 20 performance targets (15 percent) associated with the program's goals and nine related performance categories. By not implementing its performance plan, the Navy has invested, and risks continuing to invest heavily, in a program that is not subject to effective performance management and has yet to produce expected results."
... "analysis also showed that the contractor's satisfaction of NMCI service level agreements (contractually specified performance expectations) has been mixed."
… "analysis further showed that NMCI's three customer groups (end users, commanders, and network operators) vary in their satisfaction with the program."
Sounds like your typical commercial enterprise project doesn't it? One telling statement in the introduction of the GAO report states …
A plan that the Navy developed in 2000 to measure various aspects of the program, and thereby gauge program goal attainment, has not been implemented, and associated performance reports have not been issued. According to Navy officials, implementing this plan has not been as high a priority as, for example, deploying NMCI and measuring contractor performance. While program officials told us that NMCI has achieved much, they were unable to provide performance data to demonstrate these achievements relative to either the program's strategic goals or the nine performance categories that its 2000 performance measurement plan and other initiatives defined for these goals.
Gee no plan to measure the performance of the various aspects of the program? Sound familiar.
Why And How Can Evolutionary Approaches Correct Poor Management?
The authors don't actually say how. They do reference the W2COG project. Remember this is an integration process of openly available COTS products of a communications system. Now there is a poster child for failure in that area as well. The World Wide Military Command and Control Information System (WWMCCIS), http://archive.gao.gov/d47t13/116661.pdf.
There are some interesting causal issues here:
- the operational concepts were stated too broadly
- the plan prematurely focuses on system architecture
- centralized program management of the current system (WIS) is necessary but unattainable
- modernization efforts were unlikely to resolve the problems with the existing system (WIS)
- an approved concept of operations is needed before proceeding
- detailed information requirements is inadequate
- reliance on technological advances is unwise
In the end the killer statement from this 1980 report is …
Without first identifying the critical data elements needed to carry out assigned missions, the usual result is a system design that does not completely or efficiently meet the needs of all its users. Identifying these data elements before system design and acquisition is essential to ensure proper system sizing and capacity, and hence responsiveness, as well as providing clear guidance to software developers. For these reasons, identification of detailed information requirements--the essential data elements and their associated qualities of timeliness, accuracy, and currency-is incorporated in both DOD and the Office of Management and Budget directives on developing automated systems.
So Why Would Evolutionary Systems Alone Help?
Several fundamental axioms of system development include:
- We need to know what "done" looks like in some way. Without knowing this, time and effort may be applied to work that does not proceed toward "done."
- We need to know at some level of confidence what this will cost in the end.
- Same for duration – some level of confidence how long it will take.
The WWW did not need to operate under these constraints.
So How Can the Best of Both Worlds Be Combined?
This is an ongoing question. No doubt agile has much to offer for the development of software. But for large, complex, mission critical, enterprise systems, can agile alone provide all that is needed for success. The ACM article makes that case, in the absence of evidence though.
Using the NMCI example (an integrated internet solution for the Navy and Marines). How would an evolutionary system development work in the field? In the field of combat Marines with Navy support? Would the Marines get partial capabilities, then upgrade to more capabilities over time? How those upgrades would be managed. Would everyone have to have the same version of the system in order to communicate? Good questions? NMCI is probably not performing as planned. "Users find it does not meet requirements, is inflexible, and has long waits for trouble assistance" [Marine Corp Gazette, Dec 2008]. OK, and evolutionary deployment will fix this?
Maybe. Maybe not. The first problem seems to be no one wants to admit there is a problem. How exactly would evolutionary development make the problems visible in ways not currently being done.
And The Answer Is?
Establish an irreducible set of project management principles. If these principles cannot be found in the programmatic aspects of the project, the project will likely be headed for failure. What are these irreducible set of principles?
Rapid adaptation is nice. But within what principled boundaries. Do the Marines get to choose what machines they use, what networks they communicate over, what software gets loaded on those machines? Probably not.
Did the FBI VCF project come unglued because it was not adaptive to change? That's not what the report says. Only mention of the poor project management processes, manipulative vendors, simply bad leadership.
In The End
When the discussion continues to mix software development methods with project management, we will continue to get sideways with overlapping methods. I'm not hopeful this will be resolved. PMI is starting to use the term "agile." At one time I was adamant that process and product was the same thing in the project management space.
I no longer believe this is a good idea – project management processes (project controls) and product development (software development) are separate - both needed, but not interchangeable.
Bill Duncan is currently spewing out his breakfast coffee at the sound of this admission.
