Today's Wall Street Journal 28 Oct 2013, has a front page article on the difficulties with the Affordable Health Care (ACA) web site Health Site Stymied By Lack of Direction.
There are lots of opinions going around on the source of the problems with the ACA. Some are informed by similar experience, some informed by simple observations of the facts in the press, some not so well informed.
In general the problems started, as they do with any large project, with the absence of a single source of integrative management accountability. This is a long term for no single program manager accountable for delivering the capabilities needed to enroll in the health exchanges.
I don't want to dissect each suggestion of the source of the problems, but I would like to clear up many misconceptions of how projects like this get into trouble, by starting with the framework of federal acquisition. The Federal Information Technology Acquisition Reform Act (FITARA) is the framework for how non-DOD agencies buy IT solutions. I'm a DOD person, but the framework for IT acquisition is shared across many agencies, including DoD. DoD does it this way. Let's start with some background on how to sort out fact from fiction.
Just for full disclosure, there are DoD programs that are monumental failures, so I'm not holding up the DoD as the poster child for how to do large IT projects. But there are shining examples of how to do it right, using evolutionary, incremental, iterative, and agile like processes.
The WSJ article has a nice summary as well and I'll make comparisons to our DoD, DOE, and NASA programs which are also software intensive. The ACA had or lacked.
- No single leader for the ACA project - we always have a single Program Manager and some times a Program Executive Office (PEO) accountable for the success of the program.
- Disjointed bureaucracy - while there are complex organizations on the programs we work, there is a single definitive Work Breakdown Structure (WBS) and Organizational Breakdown Structure (OBS) that is on contract showing who is accountable for what deliverables, what those deliverables are, and for larger programs (ACAT 1's) the Measures of Effectiveness, Measures of Performance, Key Performance Parameters, and Technical Performance Measures. Without these - in some form - the project stakeholders have no way to know what DONE looks like, and who is accountable for delivering on done.
- Divergent agency cultures - while there are several distinct cultures in the DoD, the reporting chain of command lands on the Secretary of Defense and those at the top level of the chain have uniforms on and share - for the most part - the same culture. There are plenty of examples of disaster programs in DoD, (please read the Rand reports for the Root Cause), they usually share a singular outcome - support the warfighteri and they usualy share a singular root cause - political misdirection.
- Policy making, interagency coordination, web development, Medicare and Medicaid all working in silos - while there is overlap and duplication between the services (Navy has aircraft, Air Force has "infantry," Army has missiles), these for the most part are clear to everyone. And the interactions at the Joint Operations levels worked out in advance.
So let's see how to address some common myths around the ACA site and large complex IT projects in general:
- The larger the effort, the more we need to estimate. And, the more your estimate will be wrong. The more we estimate, the more we have schedule games. The smaller your effort, the easier it is to estimate.
- Smaller efforts are certaintly easier to estimate. This is a tautology. But larger programs can have credible estimaets as well. There are several professional organizatons for cost and schedule estimating. International Cost Estimating and Analysis Association, Association for Cost Estimating International, as well as vendors that provide software cost estimating tools and processes, Galorath, Price, as well as general guideance for software estimating. This is a process that is done every month on the programs we work. Using Reference Class Forecasting and Monte Carlo Simulation for both cost and schedule and model case assessments of technical performance.
- Let's start at the beginning - you need to estimate any project when you are spending other peoples money. To not provide an Estimate At Completion or an Estimate to Complete is simply bad project management at best and negligent of your responsibility to those with the money at worse.
- The GAO Cost Estimating and Assessment Guide, provides explicit process for estimating cost, schedule, and its related risk. As a user of this guide on both the policy side and the program executon side, it has a set of Best Practices for developing estimates. Along with the GAO Schedule Assessment Guide: Best Practices for Project Schedules, the Integrated Master and Integrated Master Schedule Preparation and Use Guide, and the Air Force Integrated Master Schedule (IMS) Assessment Process, there is pleanty of guidance on how to make credible estimates.
- (It's been) ... suggested you use agile approaches in this estimation series. You can break things down, and iterate. You get more information, and estimate smaller chunks. You are more likely to have accurate estimates.
- Turns out itertaive and incremental development of products and services are just what happens in federal acquistion. GAO Software Development: Effective Practices and Federal Challanges in Applying Agile Methods.
- The NDAA (National Defense Authorization Act) 2011 Section 804 states essentially the same.
- Here’s one problem ... with estimation. Software is not construction. We can’t build software the same way we construct or manufacture anything. Software is all about learning as a team. We can timebox our learning. We can choose to stop doing something. We can put acceptance or release criteria around it and say, “We have done enough for now.”
- This turns out to also be the prescribed method for developing products and services.
- Rolling waves, Integrated Master Plan Significant Accomplishments and Accomplishment Criteria, Earned Value Management all integrated with agile-style techniques (embedded in the federal acquisition governance framewwork). Integrating agile processes with Earned Value has been in place for several years now.
- Tangible evidence of physical percent complete is the basis of Earned Value Management. Measures of Effectiveness and Performance, Key Performance Parameters, and Technical Performance Measures are all critical to increasing the Probability of Program Success.
So What's the Root Cause of the ACA Site's Failure to Meet Expectations?
At the moment there is no way to know. So speculating that it was one thing or another - especially that estimating was the cause and not estimating would somehow have helped is actually beyond speculation. It's simply an unsubstantiated suggestion. GAO will certaintly wade in soon with their assessment. In June GAO had done some initial assessment in June of this year. So we'll see what comes next.
In The End
In the absence of a domain and context, it's not possible to assess the applicability of any software development method, process, or tool. So when someone, anyone, suggest as better way, suggests they know what the problem is, or suggests that we stop doing one thing and start doing another - please have them answer these simple questions before continuing to discuss the options:
- Do you have access to the root causes of the observed problems?
- Have you worked in the domain before, worked there now, or know someone who has or does?
- Do you have some basis of understanding the externalities of the situation. I learned this word from a cycling colleague who is an Economics graduate of the University of Chicago and it has served me well. This means in harsh terms - do you have an understanding of the problem outside your personal anecdotal experiences in your domain and are those in any way applicable to the problem at hand.
- Is the problem at hand a wicked problem that you are trying to solve with a simple solution? Wicked problems are those problems where the problem is not really known, let alone the possible solutions.
- So take care when you hear words like estimates can't be trusted or, process is inhumane, or you can't forecast the future, to confirm these are based on evidence in specific domains and contexts that are applicable to your domain and context. There is a huge - actual orders of magnitude (x10) - difference in software development domains. Projects ranging from 10's of 1,000's of dollars to 100's of millions if not billions of dollars of software development. Along that spectrum there are many options that can lay the ground work for increasing the probability of success. Not all of which can be applied in neighboring domains.
Learn to Estimate
Assuming you actually need to estimate, and actually want to learn how to estimate there are many ways to do that. If you've decided you don't want to estimate, don't need to estimate - for all the right reasons - or simply refuse to estimate, then there is much else to say about estimating. Now you may not need to estimate, because your work is short, of a value that doesn't have much at risk - unlike the ACA site, or can sustain the production of work in the absence of knowing what DONE looks like.
But if making estimates of cost, schedule, and technical performance are part of the project process while spending other peoples money, Google will find you most everything you need to learn that those statements about this can't be done are actually too narrowly focused to be credible outside small, self contained, low risk project.
