Making decisions in the presence of uncertainty of a future outcomes resulting from that decision is an important topic in the project management, product development, and engineering domains. The first question in this domain is...
If the future is not identical to the past, how can we make a decision in the presence of this future uncertainty?
The answer is we need some means of taking what we know about the past and the present and turning it into information about the future. This information can be measurements of actual activities - cost, duration of work, risks, dependencies, performance and effectiveness measures, models and simulation of past and future activities, reference classes, parametric models.
If the future is identical to the past and the present, then all this data can show us a simple straight line projection from the past to the future.
But there are some questions:
- Is the future like the past? Have we just assumed this? Or have we actually developed an understanding of the future from looking into what could possible change in the future from the past?
- If there is no change, can that future be sustained long enough for our actions to have a beneficial impact?
- If we discover the future may not be like the past, what is the statistical behavior of this future, how can we discover this behavior, and how will these changes impact our decision making processes?
The answers to these and many other questions can be found in the mathematics of probability and statistics. Here's some popular misconceptions of mathematical concepts
Modeling is the Key to Decision Making
"All models are wrong, some are useful," George Box and Norman R. Draper (1987). Empirical Model-Building and Response Surfaces, p. 424, Wiley. ISBN 0471810339.
- This book is about process control systems and the statistical process models used to design and operate the control systems in chemical plants. (This is a domain I have worked in and developed software for).
- This quote has been wildly misquoted, not only out of context, but also completely out of the domain it is applicable to.
- All models are wrong says, every model is wrong because it is a simplification of reality. This is the definition of a model.
- Some models, in the "hard" sciences, are only a little wrong. They ignore things like friction or the gravitational effect of tiny bodies. Other models are a lot wrong - they ignore bigger things. In the social sciences, big things are ignored.
- Statistical models are descriptions of systems using mathematical language. In many cases we can add a certain layer of abstraction to enable an inferential procedure.
- It is almost impossible for a single model to describe perfectly a real world phenomenon given our own subjective view of the world, since our sensory system is not perfect.
- But - and this is the critical misinterpretation of Box's quote - successful statistical inference does happen any a certain degree of consistency we exploit.
- So our almost always wrong models do prove useful.
We can't possibly estimate activities in the future if we don't already know what they are
We actually do this all the time. But more importantly there are simple step-by-step methods for making credible estimates about unknown - BUT KNOWABLE - outcomes.
This know of unknown but knowable is critical. If we really can't know - it is unknowable - then the work is not a project. It is pure research. So move on, unless you're a PhD researcher.
Here's a little dialog showing how to estimating most anything in the software development world.
With your knowledge and experience in the domain and a reasonable understanding of what the customer wants (no units of measure for reasonable by the way, sorry), let's ask some questions.
I have no pre-defined expectation of the duration. That is I have no anchor to start. If I did and didn't have a credible estimate I'd be a Dilbert manager - and I'm not.
- Me - now that you know a little bit about my needed feature, can you develop this in less than 6 months?
- You - of course I can, I'm not a complete moron.
- Me - good, I knew I was right to hire you. How about developing this feature in a week?
- You - are you out of your mind? I'd have to be a complete moron to sign up for that.
- Me - good, still confirms I hired the right person for the job. How about getting it done in 4 months?
- You - well that's still seems like too long, but I guess it'll be more than enough time if we run into problems or it turns out you don't really know what you want and change your mind.
- Me - thanks for the confidence in my ability. How about 6 weeks for this puppy?
- You - aw come on, now you're making me cranky. I don't know anyone except someone who has done this already, that can do it in 6 weeks. That's a real stretch for me. A real risk of failure and I don't want that. You hired me to be successful, and now you're setting me up for failure.
- Me - good, just checking. How about 2½ months - about 10 weeks?
- You - yea that still sounds pretty easy, with some margin. I'll go for that.
- Me - Nice, I like the way you think. How about 7 weeks?
- You - boy you're a pushy one aren't you. That's a stretch, but I've got some sense of what you want. It's possible, but I can't really commit to being done in that time, it'll be risky but I'll try.
- Me - good, let's go with 8½ weeks for now, and we'll update the estimate after a few weeks of you actually producing output I can look at.
Microeconomics of Decision Making
Making decisions about the future in the presence of uncertainty can be addressed by microeconomics principles. Microeconomics is a branch of economics that studies the behavior of individuals and small impacting organizations in making decisions on the allocation of limited resources. Projects have limited resources, business has limited resources. All human endeavors have limited resources - time, money, talent, capacity for work, skills, and other unknowns.
The microeconomics of decision making involves several variables
- Opportunity cost - the value of what we give up by taking that action. If we decide between A and B and choose B, what is the cost of A that we're giving up.
- Marginal cost analysis - impact of small changes in the “how-much” decision.
- Sunk cost - costs that have already been incurred and cannot be recovered.
- Present Value - The value today of a future cost or benefit.
Formally, defining this choice problem is simple: there is a state space S, whose elements are called states of nature and represent all the possible realizations of uncertainty; there is an outcome space X , whose elements represent the possible results of any conceivable decision; and there is a preference relation ⪸ over the mappings from S to X. †
This of course provides little in a way to make a decision on a project. But the point here is making decisions in the presence of uncertainty is a well developed discipline. Conjecturing it can't be done simply ignores this discipline.
The Valuation of Project Deliverables
It's been conjectured that focusing on value is the basis of good software development efforts. When suggested that this value is independent of cost this is misinformed. Valuation and the resulting Value used to compare choices, is the process of determining the economic value of an asset, be it a created product, a service, or a process. Value is defined as the net worth, or the difference between the benefits produced by the asset and the costs to develop or acquire the asset, all adjusted appropriately for probabilistic risk, at some point in time.
This valuation has several difficulties:
- Costs and benefits might occur at different points in time and need to be adjusted, or discounted, to account for time value of money. The fundamental principle that money is worth more today than in the future under ordinary economic conditions.
- Not all determinants of value are known at the time of the valuation since there is uncertainty inherent in all project and business environments.
- Intangible benefits like learning, growth or emergent opportunities, and embedded flexibility are the primary sources of value in the presence of uncertainty.
The valuation of the outcomes of software projects depends on the analysis of these underlying costs and benefits. A prerequisite for cost-benefit analysis is the identification of the relevant value and cost drivers to produce that value. Both cost and value are probabilistic, driven by uncertainty - both reducible and irreducible uncertainty
Modeling Uncertainty
In addition to measurable benefits and costs of the software project, the valuation process must consider uncertainty. Uncertainty arises from different sources. Natural uncertainty (aleatory) which is irreducible. This uncertainty relates to variations in the environment variables. Dealing with irreducible uncertainty requires margin for cost, schedule, and the performance of the outcomes. For both value and cost.
Event based uncertainty (epistemic) which is reducible. That is we can buy down this uncertainty with out actions. We can pay money to find things out. We can pay money to improve the value delivered from the cost we invest to produce that value.
Parameter uncertainty relates to the estimation of parameters (e.g., the reliability of the average number of defects). Model uncertainty relates to the validity of specific models used (e.g., the suitability of a certain distribution to model the defects). There is a straightforward taxonomy of uncertainty for software engineering that includes additional sources such as scope error and assumption error. The standard approach of handling uncertainty is by defining probability distributions for the underlying quantities, allowing the application of a standard calculus. Other approaches based on fuzzy measures or Bayesian networks consider different types of prior knowledge. ‡
The Final Point Once Again
The conjecture we can make informed decisions about choices in an uncertain future can be done in the absence of making estimates of the impacts of these choices has no basis in the mathematics of decision making.
This conjecture is simply not true. Any attempt to show this can be done has yet to materialize in any testable manner. This is where the basic math skills come into play. There is no math that supports this conjecture. Therefore there is no way to test this conjecture. It's personal opinion uninformed by any mathematics.
Proceed with caution when you hear this.
† Decision Theory Under Uncertainty, Johanna Etner, Meglena Jeleva, Jean-Marc Tallon, Centre d’Economie de la Sorbonne 2009.64
‡ Estimates, Uncertainty and Risk. IEEE Software, 69-74 (May 1997), Kitchenham and Linkman and "Belief Functions in Business Decisions. In: Studies in Fuzziness and Soft Computing, Vol. 88, Srivastava and Mock
