Mishkin Berteig pointed out a nice article in the IEEE Spectrum, "Who Killed the Virtual Case Field?" The question he poses - could agile have saved this project? brings up another question.
When and where is agile (software development) applicable?
As a starting point, the current edition of CrossTalk: The Journal of Defense Software Engineering, has an article "Army Simulation Program Balances Agile and Traditional Methods with Success," Doug J. Parsons. This piece describes how the author answered that question
The IEEE Article
The description of how the Virtual Case File project got into trouble is a classic project management failure. It's worth reading the IEEE article but he's my synopsis.
- The Project Manager had never led a project before. Special Agent Larry Depew was selected to lead this project because of his experience building a personal database (FoxPro) to support his investigations. Not that this experience had no value, but his project management experience in the IT domain was zero.
- The requirements elicitation process was overwhelmed by changes, instability, and conflicting requirements.
- The two prime contractors - SAIC and DynCorp provided the software and hardware for the contract. One (SAIC) used an iterative delivery method (spiral, which is common in the defense community), while DynCorp approached the work with little or no detailed planning.
- The project manager's style was to "shoot from the hip," get something in the hands of the field agents as fast as possible. The question was what?
- There was no baseline architecture, baseline master schedule, no detailed description of the FBI's IT processes, no method of measuring progress to plan other than the passage of time and the consumption of funding.
- A JAD (Joint Application Development) method was used to capture ideas from the users. These ideas formed the basis of the requirements.
- A "Flash Cutover" approach was planned in which the legacy system would be stopped and the VCF system started over a weekend. There was no going back once the cutover was made.
- The were no formal project schedules, milestones that SAIC and DynCorp were contractually obligated to meet.
As these activities proceeded a professional program manager was brought on the project - Sherry Higgins, a 29-year veteran of AT&T and Lucent led the PMO reporting to the FBI Director. Higgins was unable to get detailed schedules from DynCorp, while SAIC posted their detailed plans in their "war room."
No connection between cost and schedule could be made when changes in requirements or acceleration in schedule was requested.
The current FBI CIO arrived with the dubious task of overseeing the death of the VCF project. He summarized the problems as:
- Flawed requirements
- No requirements management discipline
- Overly specific nature of the requirements, with no overview of why or how a specific requirement fit into the architecture of the project
- SAIC was determined to write much of the functionality from scratch
- The investment management process was seriously flawed. This is called governance and there appeared to be none on this project
- Significant management turbulence created disconnects between the various teams
- A "trial and error," "we'll know it when we see it,"approach to requirements verification
- 400 defeciencies in the final product lead the FBI to a arbitration court. "We have requirements that are not in the final product, yet we have capabilities in the final product that we don't have requirements for."
Two recommendations came from a Senate Committee report:
- The flash cut over can not be allowed to take place
- The FBI should create an enterprise architecture to guide the development of its IT systems
A commissioned report from Aerospace Corporation (a prestigious engineering and consulting firm) stated...
..."the [VCF] architecture was developed without adequate assessment of alternatives and conformance to various architectural standards and in a way that precluded the incorporation of significant commerical off-the-shelf software."
..."high level documents, including the concept of operations, systems architecture, and system requirements were neither complete nor consistent, and did not map to user needs."
..."The requirements and design documentation was incomplete, imprecise, requirements and design tracinf have gaps, and the software cannot be maintained without difficulty. And it is therefore not fit for use."
So What Can Agile Speak to Here?
Mishkin has suggested that agile may have something to say. An agile software development process can most certaintly...
- Produce bug free software modules using 100% unit tests
- Provide feedback to the developers about the integration of modules through Test Driven Development
But can an agile software development process...
- Manage changing, conflicting, overlapping, inconsistent, confusing, missing, and sometime downright wrong requirments?
- Coordinate the various prime and subcontarctors in their execution of a performance based procurement contract?
- Provide the architectural baseline on which software modules can be developed, tested, integrated and deployed to the user community?
Before there is an answer to these last questions, I'm reminded of the words of a nationally known risk management consultant, when a manager from corporate decided to pick a tool for use on one his programs that was seriously flawed in its opertaion, inappropriate for the problem and was not compatible with our other tools.
Can you introduce me to three (3) people who have used this tool (or process) on three (3) seperate programs that have completed (not neessary that it was a successful completion) and that will talk about how it worked for them and how it might work for me?
So when it is suggested that a process can solve a problem, ask the person (or at least ask yourself) that question phrased to fit the situation.