Herding Cats

The World of Probability and Statistics

This is from an article about the application of Bayesian Statistics to a civil suit in the UK over the source of a building fire.

The idea that you can assign probabilities to events that have already occurred, but where we are ignorant of the result, forms the basis for the Bayesian view of probability. Put very broadly, the 'classical' view of probability is in terms of genuine unpredictability about future events, popularly known as 'chance' or 'aleatory uncertainty'.

The Bayesian interpretation allows probability also to be used to express our uncertainty due to our ignorance, known as 'epistemic uncertainty', and popularly expressed as betting odds. Of course there are all gradations, from pure chance (think radioactive decay) to processes assumed to be pure chance (lottery draws), to future events whose odds depend on a mixture of genuine unpredictability and ignorance of the facts (whether Oscar Pistorius will be convicted of murder), to pure epistemic uncertainty (whether Oscar Pistorius knowingly shot his girlfriend).

When we build probabilistic models of project performance - cost, schedule, and technical - we assume we understand the underlying statistical processes that drive these probabilistic generating functions. These are the aleatory uncertainties in duration, cost, and performance. We define the Probability Density Function in the Monte Carlo Simulator. Then we apply that to the network of work activities (the Integrated Master Schedule), to produce confidence outcomes for completing on or before a planned date and at or below a planned cost. This is all fine and dandy. But we really don't know the underlying drivers that create coupling, correlation, and cross correlations between the work activities, cost, and technical performance. These can be model by discovering the drivers in the network.

For the Epistemic uncertainties we need another modeling tool. The current tools don't actually use Bayesian statistics, rather they use Monte Carlo Simulation and treat the Probability of an Event as an aleatory process integrated with the other PDF's, ranges, and their shapes (Kurtosis and Skew).

We're missing the tools needed to construct a credible epistemic model of how the program works. Using the Integrated Master Schedule (IMS) as the topology for work, the probabilistic behaviours of the work elements at each node - cost, schedule, and technical performance compliance of the products - and the coupling and cohesion of the nodes. With this information - assuming it is credible, which is a HUGE assumption - we could model the behaviour of the program and ask what if questions. 

Seven Principles of Project Success

Complex Problems Require Better Solutions

There is a class of responders on the forums that continue to seek simple and simple minded solutions to complex problems. Claiming the problems can be solved with simple solutions. Of course those solutions cannot be demonstrated with any credibility outside of a limited domain. Let's start with Mencken

For every complex problem there is an answer that is clear, simple, and wrong.  - H. L. Mencken

In the project domain most of the problems start with poor estimates of cost and schedule. Hollmann starts it off  http://lnkd.in/vjf4X4. Bent Flyvbjerg takes a more radical view, calling estimators liars , but I chalk that up to a language barrier.

The core problem starts with the politics of estimating. Rarely do the buyers of the project know what it should cost. So when they hear a number that is larger than their expectations, they reject the number. They signal to the sellers what they expect to pay for something. This expectation can come from poor knowledge or simply a naive understanding of the problem, or worse a public promise that simply can't be kept. 

But a second problem exists. That across domains, there is the wrong headed belief that one domain has the solution to another domains problems. We need to break the loop we're stuck thinking that problems are only found in other domains and making the ill-informed suggestions that "if you just did it my way," there would be no problem. There is a fundamental issues with managing complexity and managing in the presence of uncertainty. We want it to be simple, it is not, but since it is hard, people don't want to actually work hard on the solution. Instead seeking simple and may times "simple minded" solutions - buy this tool, use my excel spread sheet, dumb down the processes.

So What Is The Solution

First is to recognize that ALL domains have similar problems. Cost and Schedule estimates have been discussed for many decades. Lots of literature leads us back to the singular source - the political will to search for the credible estimate, even if that means not starting the project. Or better yet, to start the project knowing the estimates are flawed.

One provocateur voice in our community suggests that PMs that take on project with poor estimates should be held criminally liable for the results. This of course is laughable, but it demonstrates the complete understanding of the problem and the solution.

This approach reminds me of Pauli's quote when presented with a nonsensical submission from a student.

They were worse than wrong because they could not be proven wrong. Famously, he once said o: Das ist nicht nur nicht richtig, es ist nicht einmal falsch! "Not only is it not right, it's not even wrong!"

Thus is where we are now. Not much in the form of research and questioning, lots of pontificating about how the other guy is all messed up, and if you'd only do it my way, you'd have your solution. Reminds me of the current US political problems. Do it my way. Of course that doesn't work, because we have complex - wicked - problems, and there is no single way. Mencken was right, and he was right long before any of the current problems.

We may be doomed to repeat history with first recognizing the problem is us.

Agile Means ...

I was at a conference for government contractors and the software systems we use to manage those programs. In the federal government agile is moving forward. Greater Accountability and Faster Delivery Through Modular Contracting.

During one of the session there was a quote that says it all about Agile Development

AGILE MEANS YOU'RE BEING PAID FOR DISCOVERING THE SCOPE OF THE PROJECT

This says it all. Agile is a powerful approach to software development in the commercial world. Agile in the government domain, encounters the Federal Acquisition Regulations (FAR) and the Defense Federal Acquisition Regulation Supplement (DFARS).

In these environments, Earned Value Management, the Six Business System Rules, and other procurement guidance define how work is performed. This is all being worked out, but the quote above did it for me!!

Systems Engineering Technical Review (SETR) Timing

In the past NAVAIR had a CD with all the SETR Program Event Checklists. Each had an Excel spread sheet that you filled out to determine if you were ready for the Event and a supporting document that describes all the details of the Event, both entry and exit criteria. The picture above has the link to the current spread sheets.

This approach to the Integrated Master Plan spelled out explicitly what "done" looked and there was no way to skip around the check list and call the work "done," without passing all the items with a GREEN. I've requested from NAVAIR the latest, but looks like it has been expanded to be DOD level now.

This paradigm is at the heart of describing what "done" looks like in units of measure for both the buyer and the seller. Measures of Effectiveness for the buyer and Measures of Performance for the seller.

Federal Goes Agile

Jeff Sutherland has a post about agile in the Department of Defense. Good stuff, but there are some issue with "going agile" in the Federal IT space. The first is for contracts over $20M require Earned Value Management, per FAR/DFARS flow downs and the OMB Part 7 guides for "FedCiv."

The second is the contract vehicles need to be adapted to how agile does things with requirements, changes, deliverables, progress payments, fee calculations. Here's the "tip of the iceberg" for integrating Scrum with Federal Acquisition. While Scrum is a wonderful concept and applicable to large majorities of projects, but it's not quite as simple as it might seem.

Here is a recent briefing on the top of agile in the federal procurement world is the focus of many organizations in and out of the government. Management Concepts is one source of training for agile in the Fed space, there are others. But in the end this is an acquisition issue.

Without changes in the FAR/DFARS clause flow downs that "put on contract" the behaviors mandated by the regulations of how software is developed, acquired, and put into service. While the agile community has many power tools to improve the problem in the federal government, they need come at this from a acquisition point of view for those benefits to be realized.

Here's some prior discussion

• Agile in the Federal Government
• Agile in the Federal Environment Webinar
• Big Program Guidance - A Caution to Agile

 

Agile Processes and the US Department of Defense

David Rico provides a very nice overview of how Agile development is done on DoD systems. Click on picture, a name and email required by AFEI for a record of who watched.

Veterans Day, 2011

Today is Veterans Day. As a veteran I have a special experience working with and for veterans. It is more than a shared experience, no matter the service, rank, or experience. It is a shared understanding of service, sacrifice, duty, and honor.

Let us remember those who served today. Not because it is a national holiday, but because their service allows us to celebrate this holiday.

  
  

I returned 14 August 1970. Many did not, let us remember them most of all.

Happy Birthday USMC - 10 November 1775

Two Battalions of Marines were formed 10 November 1775 in Philadelphia as Naval Infantry. The rest is history.

To my Marine friends - Happy Birthday

Agile in the Federal Environment Webinar

Management Concepts is hosting a webinar John Driessnack and I will be speaking about Agile in the Federal environment. This is the first ini a series of discussion about integrating Scrum and Scrum-like software development processes into project and programs subject to the Federal Acquisition Regulations and other guidance like Office of Management and Budget (OMB) Part 7.

If you work in the Federal Government or for a contractor that works for the Federal Government and are wondering when the Fed's are going to figure out "agile," please join us to hear more.

Agile in the Federal Environment: A Formula for Success
Register for an upcoming COMPLIMENTARY webinar, Agile in the Federal Environment: A Formula for Success. Agile project management has been used successfully for over 10 years, mostly in the commercial space. Implementation of this methodology in the federal environment however, presents unique challenges, in both IT and non-IT domains. Join John Driessnack and Glen Alleman as they discuss their experiences and tips for successfully implementing Agile on federal projects. Topics will include: Determining when Agile is the best approach Developing an acquisition strategy that incorporates the principles of Agile Breaking the Agile language barrier Making Agile processes work in a federal environment Best practices for incorporating Agile on federal programs In this interactive presentation, you will have the opportunity to ask questions, engage in polling and discussion questions, and learn about the tools that will help your projects succeed! Don't Wait! Space is limited. PSST!...Feel free to pass this invitation on to colleagues interested in learning more about these crucial project management skills. About the Speakers John Driessnack, PMP Mr. Driessnack has over 25 years of program management experience. He has served as a DoD major program project manager, Defense Acquisition University Professor, and senior industry consultant focused on integrated program management. Glen Alleman, PMP-RMP Mr. Alleman has over 30 years experience in program management, systems engineering, and software development. He has lead program planning and controls programs for a variety of clients in the areas of aerospace, defense, and energy.	Webinar Details TITLE: Agile in the Federal Environment: A Formula for Success DATE: November 17, 2011 TIME: 1-2 pm, EST SPEAKERS: John Driessnack Glen Alleman PDUs: 1   Beyond the Core Webinar Series – Recent Webinars If you have missed any of the webinars in the Beyond the Core series, click here to download the recorded versions.
8230 Leesburg Pike, Tysons Corner, VA 22182 888.545.8572 Copyright ® 2011 Management Concepts. All rights reserved. Management Concepts is certified to operate by the State Council of Higher Education in Virginia.
If you wish to update your profile or no longer receive emails from us, please click here PX608

Agile in the Federal Government

Introducing Agile to the Federal Government is not only difficult, it is like our friend Sisyphus here to the left. Pushing the stone up hill only to have it roll back down.

It rolls down with a change in administration. With a failure of a major project that used agile. With a change in the whim of the leaders. With an over promise under deliver approach of some vendors.

I'm working on an upcoming webinar on introducing agile into the federal IT space. This has been going on for a year, and we're nearing the outcomes from our efforts for a webinar, a one day course and a three day course.

There is a useful report that outlines both the problem and some solutions around this problem that is worth the investment. Visit the site under the cover picture to the right, read the report, follow the links, and see where this complex topic is headed.

If any domain is in need of agile approaches it is Federal IT. But not any approach will work. All the hand waving, Cum By Ya talk of self directed teams sitting in the same room with their customer, letting the requirements emerge as the money is being spent, probably isn't going to pass the smell test of Congressional oversight of spending the public's money.

Something else is needed. The integration of Agile with the contractual management of the Federal Acquisition Regulation.

As the webinar progresses, the courses development, and the planned book emerges, along with the hands on deployment, I'll post more.

Mary Popendieck Essay

Mary has a new essay titled Don't Separate Design from Development. It's one of those "looking back, to see forward" style of essays. I suspect Mary delivered software successfully because she was a subject matter expert and knew what the customer wanted.

I designed, developed, and programmed control systems - flight systems - and led the development of a fault tolerant process control computer (www.triconex.com), where we built an operating system, IO board controls using INTEL 8051, and a National Semiconductor (a CMOS version of the M68000) set of triple redundant processes. This was done in C with an embedded Real Time Operating system, WindRiver.

I can't remember having any formal specifications. We developed hardward, firmware, bent metal, developed the Control System OS, invented a programming language, developed a programming workstation on the original Compaq computer. All by simply "knowing" what to do because we had done it before, or we were literally inventing the future. We had SME's from our customers (Exxon, Boise Cascade, Shell, Total, and Elf). We had process engineers (ChemE's), we had RT OS developers from our previous defense systems. We had white hot C/Unix developers (the real UNIX, the one that came on a 9-track tape from Bell Labs).

But those days are over for lots of reasons.

1. The deep, and I mean really really deep, subject matter experts are rare these days. When we invented a schedule algorithm to maintain the real time response of all work in the OS, I took the code I had developed for missile guidance loops - keep the target inside the cone of the IR sensor while the AIM9L is on the wing of the F4D, and maintain that "lock" while the F4 flew away and the missile hit the "bad guy" in the ass. That kind of RT scheduling. I have neighbors here in Boulder that still do that for flight vehicle controls for special programs for special customers with three letter names. But in general this kind of software business it's not as common outside of special places in the US, like El Segundoa Blvd in LA.
2. We had direct control over the hardware. I mean at the silicon level. This is not possible today in normal computing
3. We were for the most part actual engineers. Physics for me, with digital signal process experience in grad school on particle accelerators. Others with EE, ChemE, process engineers, all of which had learned to write code. FORTRAN 77, PASCAL, Jovial, assembler, bit-slice (2901) microcode. But everyone was a hard core engineer at heart. One or two PhD's for reliability and safety. But no Computer Science, no home grown developers. Most developers are specialist in the development tools and some of the business process, but the "engineering" mind set is lost. There are some agile self-proclaimed voices that say software cannot be engineered. Which of course is complete BS, but they are out there shouting this nonsense, having never worked systems where "engineering" guided the development of an emergent product - you know, like flying to the moon.
4. The notion of a team was primary to our success. We had all come from large weapons systems, embedded control systems, and industrial environments. All knowing that people die - literally people die with the emergency shut down system fails to shutdown. Piper Alpha had occured a few year before. No prima donnas, no "I'm special because I wrote a book," or I've did one thing at one company and now I'm qualified to tell you what to do somewhere else.

So Mary is right, lots of software can be built without stories and detailed requirements. But that takes place ONLY because all that "stuff" is in the heads of the engineers and developers who have dome it before - succesfully do it before.

But now we work much more complex programs, systems where the lowest level components are well defined from the subject matter experts, but the integration of those components is complex and evolving.

Mary's quotes Gilb and Brooks is right on. In our Aerospace and Defense business this is RARELY the case. Embedded or Software Intensive systems have a tight connections between "engineering" of the systems and the "design" of the systems. These systems are developed incrementally and itertaively. This is the fundamental principle of Systems Engineering, the V-Model, and the Integrated Master Plan / Integrated Master Schedule. As well "process is King," in our domain, so anyone wanting to skip the process can look for work elsewhere.

At the notional level this is how programs are run in the A&D and other Systems Engineering domains. This approach is Rarely found in the enterprise IT would, expect where enterprise IT is inside and A&D firm. When it says detailed design this is context and domain sensitive. On some embedded system, sysMl is used with UML to define all the interfaces between the components. In other domains, detailed design is down to the board level and timing diagram of the firmware controlled system that will be sent to an ASIC fab shop that converts software (Handel-C) to hardware.

The appropriateness of the level of design is always considered, since the programs are on contract for a delivery date, a target budget, and a specific set of performance parameters.

This concept is also missing from enterprise IT - the idea that the project is a "business," with a profit margin, an expected delivery date, and a minimal set of capabilities. I'd suggest that if projects were run that way, better results would appear. Things like "do the appropriate amount" of documentation. Since documentation doesn't fly to orbit, we do what is needed to get the space craft to orbit and that's all. Documentation is needed, but documentation that doesn't support the capabilities - fly to LOE - is a waste. Why this is not well understood outside our domain I don't know.

Bad Project Management is my guess.

Big Program Guidance - a Caution to Agile

The notion that large programs have heavy, unwieldy processes, and that agile methods - both software development and project management - are the solution is many times just that "notional."

Let's start with the top level principle of large program acquisition in the Federal Government

Many if not most large government acquisitions have unwieldy processes, poor track records, and lots of blame to go around. But the first chapter of the Federal Acquisition Regulation (FAR) provides some guidance that can be applied to any program, project, or development effort.

Notice the timely, best value, for the customer, maintaining trust. Add working together as a team, and being empowered to make decisions in their area of responsibility.

Is this guidance followed - rarely. Why? Beacuse guidance alone is not sufficient for success. If that were the case, we woudln't need lighter weight, agile processes to pull us out of the messes we've created on large federal programs.

Agile needs to take this cautionary example - as agile moves into the enterprise, it will be just the latest attempt to lighten the load of processes and formality.

Taking these words as stand alone and looking to agile development and project management methods may provide the mission and vision around improving project performance in smaller domains.

Back from the NDIA ISS-II Agile Conference

The NDIA Information Systems Summit II held in Baltimore was interesting in many ways:

• The government is seeking ways to improve the effectiveness of their IT dollars by (1) Early and continual involvement, (2) multiple and rapidly executed increments or releases of capability, and (3) early, successive prototyping to support and evolutionary approach. This directive from congress to the SecDef is stated in Section 804 of the National Defense Authorization Act .
• The domain of Department of Defense procurement was new to many of the agile participants. This was a surprise when the speakers from DoD were engaging in a conversation about how to bring agile into the DoD and the agile folks started with a clean slate assuming all the agile practices could just be installed and used.
• We had briefings from senior DoD leaders. The deputy CIO's and deputy DISA CTO. These guys have multi-billion dollar budgets. The US Government IT budget is $80B, DoD $38B, most of the services have their own around $7B or so, with DISA itself around $18B. As well the leaders are 2 and 3 star generals or senior government managers.

The prime outcome of the conference is:

• Agile can be integrated into government contracts. There is nothing in the regulations that defined the software development method.
• Agile is a software development method. Project and Program management is established inside the DoD context, so all the agile approaches to project management - mainly leaving the project manager out of the conversation - aren't there. What is there are coaches, Scrum Masters, "product owners" and that.
• The realization that Earned Value and a bunch of other FAR/DFAR guidance is required was news to many of the agile thought leaders. They were unfamiliar with the rigor these regulations are applied and the complexity of the acquisition process.
• While the agile folks see agile as doing everything on the project, the DoD folks defined boundaries where agile can go. That seemed to be news as well.

How to make Agile work in DoD

• Start bottom up. Start with the processes for developing deliverable code. Use Stories for capturing the behavior of software "pieces."
• Use Capabilities Based Planning to define the "architecture" of the product in a way that is familiar to the agile community. This is usually done with Stories, User Stories, Use Cases and the like. There were discussions about the various ways of doing this, what things are called, and how they are developed.
• This is one thing that needs improvement. There is really no standard definition (OK, a few) for what things are called. Epics, Releases, Iterations, etc. I'm going to try and collect these for our next meeting and start a Glossary.
• Keep the agile activities "inside" the work package for a start. This allows the Work Package manager to arrange the work around the release and not have to deal with the structure of the formality of DoD procurement at the actual development level.
• With the sequence of Work Packages around each iteration, and mainatining the BCWS across a rolling wave, Scrum-like activities can start to be directly applied to development.

The basis for all this is anchored in several sources. The best one is of course Dan Wards F.I.S.T. work. Dan's The Fist Handbook, The Simplicity Code, and Rogue Project Leader, should be mandatory reading for any agile advocate working in the DoD world and also the interview with Dan on POGO.

There will be much more on this topic in the coming months. But here's the core theme...

Agile software development can be used in formal and structured environments - like DoD - if the baseline conditions for the program (FAR/DFAR/OMB) are the starting point, then following the guidance from Dan, and then and only then deploying the agile principles.

I'm going to conjecture this approach will result in the broadest deployment of agile development processes. The IT budget of $80,000,000,000.00 completely overwhelms every other IT project collection on the planet. This is where the maximum impact will take place.

The Myth of Traditonally Structured Projects

There is a well worn myth in the agile community, that large complex projects are developed using waterfall methods, where all the requirements are defined upfront, (BDUF), rigid processes are used to execution the program and the outcome is defined before the project starts. This of course is great fodder to the replacement of the "devil" waterfall process with agile processes.

For anyone interested in how a program actually works, here a good book that can be found at the Air University Press book store. "The Development of the B-52 and Jet Propulsion: A Case Study in Organizational Innovation," Dr. Mark D. Mandeles, Air University Press, Maxwell Air Force Base, Alabama. March 1998. The Air University Press has many books on management, leadership, strategy, and history.

Quoted from the preface:

The B-52 and Jet Propulsion: A Case Study in Organizational Innovation is a coherent and nonpolemical discussion of the revolution in military affairs, a hot topic in the national security arena. Mark Mandeles examines an interesting topic, how can the military better understand, manage, and evaluate technological development programs. We see Murphy’s Law (anything that can go wrong, will go wrong) in operation. No matter how carefully the military designs, plans, and programs the process of technological development, inevitably, equipment, organizations, and people will challenge the desired expectations. Mandeles argues convincingly that recognizing the inevitability of error may be the single most important factor in the design of effective organizations and procedures to foster and enhance innovative technology and concepts.

The book focuses on the introduction of jet propulsion into the B-52. This case study illustrates the reality that surprises and failures are endemic to development programs where information and knowledge are indeterminate, ambiguous, and imperfect. Mandeles’ choice of the B-52 to illustrate this process is both intriguing and apt. The military had no coherent search process inevitably leading to the choice of a particular technology; nor was decision making concerning the B-52 development program coherent or orderly. Different mixtures of participants, problems, and solutions came together at various times to make decisions about funding or to review the status of performance projections and requirements.

Sounds like an agile project to me.

Does Agile Need to be Named?

There was a suggestion that Agile should drop the name Agile.

What just when we're starting to get that name syndicated in our large, cumbersome, domain. No Way.

When someone asks whats in the name, I've started using two pictures.

A traditional approach

and  an agile approach

and to focus the idea of agile even more, here is a simple phrase.

What do we mean when we say agile? Being able to turn inside the loop of unfolding events.

The person making that statement wasn't some self styled agile thought leader, or the owner of a leading Scrum training organization, or even one of the original appostles of the Agile Manifesto.

It is Dr. Ashton Carter, Under Secretary of Defense in Acquisition, Technology and Logistics, in the Sep/Oct, 2010 of the journal Defense AT&L.

It's not time to go changing anything aroudn a name. It's time to move Agile Software Devloment into the "real" mainstream - the US Department of Defense. If you want to see software intensive systems that MUST respond to emerging and complex environments, then look no further than software intensive weapons and their support systems.

In our domain and context the anchor paradigm for agile is Col Boyd's OOAD

The components of agile are all here. It is itertaive, adaptive, can be made to be incremental, and  defines the "loop" processes. But most of all it is "field proven." People can be trained to follow this loop process. People can receive direct feedback when they are not following the loop. The loop focuses on the outcomes of work effort.

In Boyd's paradigm there are three critical success factors applicable to project and program management:

1. Insight and vision - without insight and vision there can be no orientation to deal with the present and future.
2. Focus and direction - without focus and direct - implicit or explicit - there can be neither harmony of effort nor initiative for vigorous effort.
3. Adaptability - implies variety and rapidity. Without variety and rapidity, one can not cope with changing and unforeseen circumstances.

Boyd used this paradigm in warfare, focused on fighter aircraft air-to-aor combat. But the concepts are directly applicable to the "combat" of getting the product out the door in the presence of an emerging environment of chaging requirements, emerging technology, internal and external risk, and the politics of people when they are connected to money.

Questions for the Project Manager

In the defense and space business the Control Account Manager (CAM) is as close to a Project Manager as you can find. The role of the CAM is basically:

• Establish and maintain Control Account budgets and schedules.
•  Define the tasks and the sequencing of the schedule for optimizing the accomplishment of the work scope.
• Assure the scheduled activities for the Control Account have a realistic duration and performance logic.
• Maintain the applicable portions of the WBS Dictionary and approve the Control Account Authorization (CAA) and authorized changes to the CAA.
• Proactively manage the personnel performing the Control Account detailed scope of work.
• Identify, negotiate and implement required vertical and horizontal interfaces.
• Manage Customer Expectation Agreements (CEA) and Memorandums of Understanding (MOU) with other organizations and programs.
• Review and approve all resources charged to the Control Account and assure their accuracy.
• Monitor and assess Control Account and Work Package performance.
•  Prepare Control Account variance analyses.
•  Develop, implement and manage corrective actions, as appropriate.

These are deliverbles management functions. They are not focused (primarly) on the techncial aspects of the Control Account (the project inside a program).

Now this may be a distinction from the traditional IT or SW development project, where the PM is a techncial expert and a participant on the development team. And certaintly distict from the agile software development where there is not a difference between the "manager" of the team and the "members" of the team.

But there is important issues with the "less organized" project teams - teams where "managing the work" and many times indistinguishable from "doing the work."

For domains where this distinction is clear and bright, there are a set of questions that need answered to assess the credibility of the project's ability to be successful.There is a set of questions that come from the Defense Contract Management Agency to guide this assessment.

This starts with ...

Organizational Processes.

• Would you please tell me about your area of responsibility on the program?
• How long have you been in this position?
• Is your time totally dedicated to this program, or do you also work on other programs?
• Please show me how you fit into the organization and to whom you report?
• To whom do you report and what are your responsibilities to this person?
• How many people work for you and what do they do?
• How do they report to you?
• How do you know the performance status of their work?
• For how many control accounts are you responsible?
• What is the total dollar value of your control accounts?

Scheduling Responsibilities

• Please explain your scheduling responsibilities?
• Please explain how you schedule your work?
• Did you develop your schedule?
• If so, how did you establish the timeframes or
  durations for your effort?
• Do you directly support any major milestones on the master or intermediate schedules?
• How do you receive schedule information? How do you provide schedule information?
• Do you have detailed schedules below the work package?
• How do detailed schedules below the work package support the work package schedules?
• What process do you use to ensure that your schedules are aligned with the schedules of other CAMs and with the program schedule?
• How are you informed by other organizations of changes in their activities that may affect your control accounts schedules?
• How do you inform others of any changes in your scheduled activities?

Work/ Budget Authorization Process

• What role did you have in formulating the proposal estimate/ budget for this effort?
• How did you arrive at your proposal estimates? Do you have the backup or worksheets from which you derived your estimate(s)?
• Was there a negotiation process for your budgets after contract award? Is your budget adequate?
• Discuss any risks to achieving the work within budget.
• How are you advised of budget? Of tasks? Of schedule? Of changes?
• How are you authorized to begin work? (Ask to see the work authorization documentation.)
• How do you know that the task description in the authorization is correct?
• May we see your control account plan?
• Are your budgets time phased? How is this shown on your control account plan? Do you have any planning packages?
• Are these time phased?
• When must planning packages be detailed planned?
• How do you status your control accounts? How does the performance status of your accounts get into the system?
• What options does your system provide for taking earned value?
• Where on your control account plans is it indicated how you are taking earned value?
• How do you define a work package? What is the difference between a work package and a planning package?
• For how many work packages are you responsible?
• What portion of your work is measurable or discrete effort? What portion is Level of Effort (LOE)?
• Do you have any control accounts which contain a mixture of LOE and discrete effort? What is the highest percentage of LOE within a control account that also contains discrete effort?
• Do you have any LOE control accounts? Please describe the work associated with these control accounts?
• How do you open a control account? How do you close a control account?
• What do your computer runs show when a control account is opened or closed?
• How do you know when a work package is opened or closed?
• Have you ever opened work packages earlier than the scheduled start date? If so, how is this accomplished?
• Demonstrate how earned value is taken in the same way that budgets are planned.
• How do you determine earned value for work-in-process?
• Do you use interim milestones on any of your work packages to measure performance?

Managerial Analysis Process

• What reports do you receive that identifies control and schedule status of your control accounts?
• Which reports do you use most frequently? Why?
• Are budgets, earned value, and actual controls reported in terms of labor, material, and other direct controls?
• What are the variance thresholds of your control accounts?
• Do you have any variance thresholds at the work package level?
• How do you know when you have exceeded a threshold?
• How do rate changes affect your control accounts?
• Who is responsible for rate variance analysis?
• How do you know when you must prepare a variance analysis report?
• Do you have samples of any variance analysis reports? (Determine that these
• show a statement of problem, the variance, its cause and impact, and proposed corrective action.)
• Who receives your variance reports? What action is taken on the reports?
• Do you develop corrective action plans for variances? Who approves these plans?
• How do you follow through corrective action plans to ensure they are carried out?
• Do you ever have mischarges to your control accounts? How are these corrected?
• Who reviews labor hours on time cards charged to your control accounts?
• When/ how do you incorporate your variances into an EAC?
• What does Estimate at Completion (EAC) mean to you? How do you arrive at an EAC?
• How often is your EAC revised?
• Change Incorporation Process
• Have you had any changes to your control accounts? (Ask to see samples of how these are handled.)
• What is the process for making changes to your budgets? Can you rephase or replan work? In what circumstances?
• Have you had budget transfers between your control accounts and Management Reserve and Undistributed Budget? How? Are they traceable?

Grady Booch and DoD Software

Grady Booch spoke in CrossTalk about the current state of DoD software. John Goodpasture has the clip, here's the magazine with the article. I certaintly don't have the credentials the way Grady does to speak of the broader issues of architecture and similar topics.

But, as one working on portions of DoD (and NASA) software intensive programs, I found his approach a bit unsettling. Booch's basic message is DoD is behind the curve and needs to catch up.

• There is a mix of words around IT Infrastructure and weapons in his article. The weapons side is almost always purpose built software. Software designed and constructed for a singular purpose. The Systems Engineering paradigm dominates these products. Some DoD services do better than others when it comes to Systems Engineering. NAVAIR is an example of incorporating system engineering throughout the full procurement cycle. Others do a good job, some "not so good."
• The IT infrastructure problem is larger. COTS systems architecture is incorporated into many systems. From ERP to voice and data systems.

What Booch does not mention - on the weapons side - is DoD is primary a "procurement" house. They buy thing others build.

Booch mentions "open source," but not the Information Assurance (IA) and secure systems aspects of weapons. Vetting open source is likely less costly than writing it from scratch. But it is still a non-trivial process for an integrated weapons platform.

There is a mix of issues in the article. SOA, for one, is being used for large infrastructure programs - NETCENTS2 is one we work - in the Program Management Office. But embedded systems have special needs. Grady mentions languages and their need for improvement, but doesn't isolate the domain or context. Flight systems is another side of the DoD (and NASA) side we work. A missing critical understanding around programming languages is the heritage of the code base. C, C++, and Handel-C are common sources of heritage applications. The avionics suite on Mars missions has decades of V&V. The code base is not going to be converted to the "next new thing." The flight software on vendors "black boxes" is not going to be  re-written in the lastest language. The current approach is "incremental upgrades." The site for the journal Booch writes in manages efforts like these.

It is not well understood that systems like GN&C (Guidance Navigation & Control), C&DH (Command & Data Handling) and GPS (Global Position Systems) all have decades of "proven" reliability and functionality on the code base. The source code for these systems are "trusted" in ways unimaginable in the commercial world.

There is no doubt - especially among those working the real problems - of the difficulties of developing weapon systems software, flight, launch, ground, on-orbit - all have great difficulties. But many times the source of those difficulties starts with requirements and capabilities definitions. These attributes (capabilities and requirements) are many times driven by politics as much as adversaries.

Grady doesn't speak to these issues. GAO does.

What troubles me most is the final answer in the article.

I think the major change is in education. I don’t mean to be critical, but in many ways the DoD’s expertise has, frankly, been outsourced to its contractors. It is not to say that is a horrible, terrible thing, but a lot of the things that happened in old warfighting systems came through intrinsic expertise inside the DoD. I would strongly encourage the increase of education of the DoD’s intrinsic forces with regards to decision engineering and software engineering—and draw back into the DoD more of that intellectual property. Ultimately, delivering for the warfighters is what the DoD is all about, and that requires an intensely educated staff to make that happen. How does one make that manifest? I think there is work to be done in acquisition policy, in processes for delivery in the use of things like DoDAF. I think the DoD itself can lead and should lead this, and it needs to make this change in the interspatial spaces of its training, in its service academies, and in its colleges as well.

DoD is an acquisition organization. It does write "some" software but not much - at least not on the weapons side. DoD "acquires" weapons systems from the industrial base. A good starting point is SoftwareTech News. Read through some recent issues and articles about the "software acquisition" issues. Take a look at the site of the person tasked to manage these acquisitions  - USD(AT&L),
then some related webs sites.

The Issue Is More Complex Than Languages, Modeling, and Architecture

The issue does start with Systems Engineering. But systems engineering starts with definitions of needed capabilities and the development of technical, operational, and managerial requirements.

Booch's article contributes to the discussion of building complex systems in the presence of emerging needs. But it's not about software development - at least in the beginning. It's about defining what DONE looks like in units  of measure meaningful to the stakeholders. This is about Systems Engineering (which Booch states). But it's also about capabilities. This  is where to start. There are processes in place to do this, but not always used.

One place to start this approach is the Rand Corporations work on Capabilities Based Planning. IT Infrastructure could take a lesson from this approach. The weapons side does in some cases. In others "not so much." Future Combat System (one where we work on a piece), Joint Strike Fighter, NASA Orion, and other System of Systems each have varying degrees of success in this domain (SoS).

The Problems Are Great, So Should the Solutions

Huge amounts of waste are in the DoD procurement process. There is no doubt about this. The SecDef speaks directly to the problem and proposes solutions. Please subscribe to the Defense Acquisition Portal to read almost daily news on these issues. At the same time there are professional organizations focusing on the issues of procurement and development of software during those procurements. The Software Engineering Institute is one. There approach is described in Understanding Common Acquisition Problems. Acquisition, like all complex problems, is fundamentally a risk management problem. In order to manage risk, we needed measures of risk and along with that measures of performance. This paradigm is at the heart of Systems Engineering. The problems Grady speaks about are Systems Engineering problem from day one. Architecture, development, operational test and evaluation, deployment, and sustainment are systems problems.

If there is to be encouragement in education let's start with encouraging Systems Engineering as a foundation. Without the system architecture being right, all the subsystems are going to hampered in their ability to provide solutions to the mission capabilities.

The Other Side of Software Based Procurement

All DoD procurements greater than $20M must have Earned Value Management processes. For procurements greater than $50M, a validated Earned Value Management System is mandated. Reform in the Earned Value world is starting to take place. Inclusion of Techncial Performance Measures (TPM) is one path for improvement. Paul Solomon provides insight into this issue

Aerospace Acronyms

The current issue of Avionics Magazine has a nice Acronym guide for Aerospace Terms. This trade publications and many others like it provide some insight into software intensive products. The Defense Acquisition University's Glossary is another.

These provide some insight into the world of software intensive systems.

Veterans Day

Let us remember today all the Soldiers, Sailors, Airman, and Marines we have served with, who are our friends, family, and colleagues.

But more importantly, let us remember those Comrades in Arms that did not return.