Some time ago there was a discussion around Black Swans and the knowability of low probability - high impact events. The Black Swan is a metaphor that encapsulates the concept that an event is a surprise (to the observer) and has a major impact. The Wild Swan is a concept that better describes what we encounter in our project work. Without a domain and a context in that domain, any discussion fails to be applicable and turns into a "populist" philosophy story.
For the Back Swan, the event is rationalized by hindsight. The critical understanding here is the surprise to the observer. Francis Bacon suggested our minds are wired to deceive. This mention of the observer rarely occurs in the popular discussion of Black Swans.
Beware the fallacies into which undisciplined thinkers most easily fall--they are the real distorting prisms of human nature.
The term undisciplined thinkers is a Root Cause of most failures. It is common to use the term Black Swan to describe some correlation that we believe (in Bacon's way) cannot be explained.But when applying a little discipline, research, and most importantly fact checking is turns out what was Black may actually be White or possible Gray.
In the current issue of Strategy & Business is an article How to Prepare for a Black Swan: Disruptor analysis can help assess the risk of future catastrophic events. (Free registration required). The clip below provides insight into the concept of a Black Swan and repeats the problem of not understanding the paradigm of black swans - a surprise to the observer.
Let's start with the Fukushima Daiichi power plant black swan. The popular press reported the earth quake was unexpected. Let's look at some articles - NOT from the popular (or populist) outlets. IEEE Spectrum is a monthly magazine of the IEEE. A special issue was dedicated to the accident.
While the earthquake itself is a complex topic, the problems at the power station are more important.
- The seawall had no safety margin.
- The diesel generators in two of the units were in the basement. The other 3 units had them on the 3rd floor. It was cheaper to put them in the basement.
- There was not back-up for the back-up power.
- The spent fuel rods were kept on site because of the politics of the disposal process, just like they are in the US.
- The killer event was that the gravity feed for reactor cooling was enabled when the earthquake struck. It was cooling the core too fast so it was turned off. Several minutes later the wave breached the sea wall, flooded the basement. The valves to turn in the gravity fed cooling loop of course required electricity. The core melted.
The earthquake was possibly unpredictable - we'll have to wait and see what the science says. But the power station and the policies were unprepared for many things, starting with the loss of site power. This is the case in many locations in the US as well. Site power is needed to control the reactor to shut down. So it went down hill from there.
The first discussion of Black Swans was prompted by Rumsfeld's comment about unknown unknowns in regards to Iraq. Like Vietnam (my war) the decision makers had little understanding and little interest in understanding the social and political ramifications of their actions. The country of Iraq is synthetic, created through the British Mandate of Mesopotamia in 1920. Tribal groups were assembled into a country. When the US came and liberated Iraq, those tribes emerged and started killing each other as well as the liberators to the surprise of the observer.
Like Vietnam, students of Iraq informed the decision makers of the hornets nest that would be released when the dictator was removed. See Fire in the Lake: The Vietnamese and the Americans in Vietnam
Black Swans ≠ Unknown Unknowns Only if it is UNKNOWABLE
The role of fault tolerant systems is to survive in the presence of failure. Fault tolerant systems are used for Emergency Shutdown, turbine controls, boiler controls. I have some experience in the domain of Fault Tolerant Systems.
This background informs (colors) my view of the notion of unknown unknowns. If you're going to venture into a realm of high risk, high consequences, you'd better damn well have considered all those improbable events that can cause your system to fail. And then in the end design that system to be fault tolerant.
So here's how it works:
- Acknowledge that Black Swans are surprises to the observer and get more observers. In fact seek out the experts in the field and have them seek out their experts and perform the needed processes to confirm that you have exhausted all the sources of faults in your system or you idea before proceeding. Table 5 of Reliability of Instrumented Safety Systems is an example.
- When the external system fails, the system in control must be not only be fault tolerant, it must fail safe. That is the system cannot fail to danger.
- If you - the observer - were surprised, ask how could that have been? Did you make every effort possible to determine causal factors that would create a fail to danger outcome? Maybe you couldn't afford to do that. Maybe you were incapable of doing that? Maybe you didn't have time to do it? Maybe you had too much hubris (in Rumsfeld's case) to admit you should be doing it. If it didn;t do all these you can't call the outcome a Black Swan.
In the absence of this approach, those pesky black swans are going to bite and bite hard.
A Counter Argument
Matthew Squair's wonderful blog has a post about probabilistic design and aviation safety. Here he restates John Downer's argument that once the risk of an extreme event has been ‘formally’ assessed as being so low as to be acceptable it becomes very hard for society and it’s institutions to justify preparing for it (Downer 2011).
So the revisiting the Black Swan and Fault Tolerant and Fail Safe concepts...
5.(a) … (1) in any system or subsystem, the failure of any single element, component, or connection during any one flight should be assumed, regardless of its probability…
This means we MUST consider that all possible failures will happen and engineer the system to deal with this. This is the failure mode of Fukushima Daiichi. The improbable was possible.
