Mary has a new essay titled Don't Separate Design from Development. It's one of those "looking back, to see forward" style of essays. I suspect Mary delivered software successfully because she was a subject matter expert and knew what the customer wanted.
I designed, developed, and programmed control systems - flight systems - and led the development of a fault tolerant process control computer (www.triconex.com), where we built an operating system, IO board controls using INTEL 8051, and a National Semiconductor (a CMOS version of the M68000) set of triple redundant processes. This was done in C with an embedded Real Time Operating system, WindRiver.
I can't remember having any formal specifications. We developed hardward, firmware, bent metal, developed the Control System OS, invented a programming language, developed a programming workstation on the original Compaq computer. All by simply "knowing" what to do because we had done it before, or we were literally inventing the future. We had SME's from our customers (Exxon, Boise Cascade, Shell, Total, and Elf). We had process engineers (ChemE's), we had RT OS developers from our previous defense systems. We had white hot C/Unix developers (the real UNIX, the one that came on a 9-track tape from Bell Labs).
But those days are over for lots of reasons.
- The deep, and I mean really really deep, subject matter experts are rare these days. When we invented a schedule algorithm to maintain the real time response of all work in the OS, I took the code I had developed for missile guidance loops - keep the target inside the cone of the IR sensor while the AIM9L is on the wing of the F4D, and maintain that "lock" while the F4 flew away and the missile hit the "bad guy" in the ass. That kind of RT scheduling. I have neighbors here in Boulder that still do that for flight vehicle controls for special programs for special customers with three letter names. But in general this kind of software business it's not as common outside of special places in the US, like El Segundoa Blvd in LA.
- We had direct control over the hardware. I mean at the silicon level. This is not possible today in normal computing
- We were for the most part actual engineers. Physics for me, with digital signal process experience in grad school on particle accelerators. Others with EE, ChemE, process engineers, all of which had learned to write code. FORTRAN 77, PASCAL, Jovial, assembler, bit-slice (2901) microcode. But everyone was a hard core engineer at heart. One or two PhD's for reliability and safety. But no Computer Science, no home grown developers. Most developers are specialist in the development tools and some of the business process, but the "engineering" mind set is lost. There are some agile self-proclaimed voices that say software cannot be engineered. Which of course is complete BS, but they are out there shouting this nonsense, having never worked systems where "engineering" guided the development of an emergent product - you know, like flying to the moon.
- The notion of a team was primary to our success. We had all come from large weapons systems, embedded control systems, and industrial environments. All knowing that people die - literally people die with the emergency shut down system fails to shutdown. Piper Alpha had occured a few year before. No prima donnas, no "I'm special because I wrote a book," or I've did one thing at one company and now I'm qualified to tell you what to do somewhere else.
So Mary is right, lots of software can be built without stories and detailed requirements. But that takes place ONLY because all that "stuff" is in the heads of the engineers and developers who have dome it before - succesfully do it before.
But now we work much more complex programs, systems where the lowest level components are well defined from the subject matter experts, but the integration of those components is complex and evolving.
Mary's quotes Gilb and Brooks is right on. In our Aerospace and Defense business this is RARELY the case. Embedded or Software Intensive systems have a tight connections between "engineering" of the systems and the "design" of the systems. These systems are developed incrementally and itertaively. This is the fundamental principle of Systems Engineering, the V-Model, and the Integrated Master Plan / Integrated Master Schedule. As well "process is King," in our domain, so anyone wanting to skip the process can look for work elsewhere.
At the notional level this is how programs are run in the A&D and other Systems Engineering domains. This approach is Rarely found in the enterprise IT would, expect where enterprise IT is inside and A&D firm. When it says detailed design this is context and domain sensitive. On some embedded system, sysMl is used with UML to define all the interfaces between the components. In other domains, detailed design is down to the board level and timing diagram of the firmware controlled system that will be sent to an ASIC fab shop that converts software (Handel-C) to hardware.
The appropriateness of the level of design is always considered, since the programs are on contract for a delivery date, a target budget, and a specific set of performance parameters.
This concept is also missing from enterprise IT - the idea that the project is a "business," with a profit margin, an expected delivery date, and a minimal set of capabilities. I'd suggest that if projects were run that way, better results would appear. Things like "do the appropriate amount" of documentation. Since documentation doesn't fly to orbit, we do what is needed to get the space craft to orbit and that's all. Documentation is needed, but documentation that doesn't support the capabilities - fly to LOE - is a waste. Why this is not well understood outside our domain I don't know.
Bad Project Management is my guess.
