There is a great blog post on TechWell about Black Swans and project failure. IT project failure, but any project failure is appropriate.
There some important points here on several fronts. For projects to actually fail means there were events that occured that the participants did not see coming. There are several sources of not seeing it coming. The first of course is you didn't look. You didn't look because you choose not to look. Or you didn't know where to look. Our you couldn't recognize the train wreck coming before it happened. Or when you looked the wreck didn't seem as bad as it turned out to be.
The second issue is Black Swans are used improperly in the project domain. Taleb's paradigm is in the sociologically driven finance world where systems operate in the edge of chaos. This chaos is a market process where hidden information drives behaviour. Many time information is not only hidden the people creating and using the information intentionally lie to each other, the public, and the oversight agencies. The result actually is a Black Swan. But of course in the best of example of a Black Swan - the finance collapse - many saw it coming and made a killing. See The Big Short and All the Devils Are Here are two sources describing how many saw it coming, knew what to do about making money and followed through on their plans.
The last thing is, many in project management, especially on the agile side for some reason, use Black Swans as an excuse for not doing their job of looking, planning, mitigating, controlling, and actually managing the processes of a project. All elements of doing stupid things on purpose. This phrase came from an incident at a DOE Nuclear Weapons plant when a fiber optic cable was cut by acident that knocked out the safety and safeguards controls to several buildings. All processes, all drawings, all step-action-plans were followed. But the final decision was made without any actual test of what woudl happen is they cut the wrong cable.
So Are There Black Swans On Projects?
If we're deploying an ERP system, build a web site - even the ACA web site, integrating a collection of legacy systems, probably not. I say probably, because there may be an unknown behaviour of the system that can't be determined before it is put into place.
If we're building a particle accelerator for the first time and haven't developed the control systems for the super-conducting magnets there may be unknowns. But of the power system for those magnets is located under trees that could fall, well that's not a black swan. That's not looking for risk.
The whole Black Swan paradigm comes down to risk management. Risk management has several issues as well. We may not be able to afford to look for risks. When we find them we may not be able to afford the mitigation costs. Or we may not actually have to political will to take mitigation's. The O-Ring and Foam mishaps at NASA are examples of a combination of these.
For IT projects, we're not inventing new physics, we not flying to outer space, we're just writing software for money. So why blame failure of Black Swans? Because it's the easiest thing to do. And as a colleague is fond of saying They Can't Handle the Truth.
So no more Black Swans until you've exhausted every avenue of exploration of the source of project failure. Identified those sources - risks, categorized them, analyzed them, made handling plans, and adjusted your project plans and schedule in the response to them. Then the only thing left are the things that were Unknowable. Only then can you invoke the Black Swan excuse.
Why Estimate?
The answer is simple. If you don't look ahead to what something might cost, when it might be done, what impediments you might encounter, how you're going to measure progress beyond the passage of time and consumption of money, then the pesky Black Swans are going to bite you. Estimating is not looking forward in an attempt to control, but looking forward in an attempt to avoid. Not doing that? You're a train wreck waiting to happen.
