The continued lack of understanding of the underlying probability and statistics of making decisions in the presence of uncertainty to plague the discussion of estimating software.
All elements of all projects are statistical in nature. This statistical behaviour - reducible or irreducible stochastic processes - creates uncertainty.
Event based uncertainties have a probability of occurrence and a probability of the impact once that uncertainty becomes a relaity. These are Epistemic uncertainties - epistemology is the studdy of knowledge. Espitemtic means knowing or not knowing in this case/ We can buy knowledge. This is the core concept of agile paradigm/ We are buying down risk by building software to test the uncertainties of the project deliverables. This is the basis of saying agile is about risk management. But I suspect those saying that without being able to do the math as we say in our domain, don't realize what they are actually saying.
The natural occurring variances are aleatory. They are always there, they are irreducible. That is they can't be fixed. Work effort and duration is aleatory. The ONLY fix for aleatory uncertainty and the resulting risk is margin. Cost margin, schedule margin, technical performance margin. You can't buy the fix to aleatory uncertainty.
Found a book today Discover Probability: How to Use It, How to Avoid Misusing It, and How It Affects Every Aspect of Your Life: How to Use It, How to Avoid Misusing It, and How It Affects Every Aspect of Your Life, Arieh Ben-Naim. World Scientific.
This is one of those must read book for anyone working in a domain where probability and statistics dominates the decision making process. Unlike other books How To Measure Anything, The Flaw of Averages, How Not to Be Wrong: The Power of Mathematical Thinking, which is very good books - but populist in that they contain little in terms of actual mathematics. this books is in between. Lots of narrative, but math as well. Not like Probability Methods of Cost Uncertainty Analysis: A Systems Engineering Perspective but in the middle.
In The End
you can't make decisions in the presence of uncertainty without estimating the outcome of your decision in the future. Using empirical data is preferred. But that empirical data MUST be adjusted for future uncertainty., past variances, sampling errors, poor representations of the actual process and the plethora of other drivers of uncertainty. Having small, simple samples without variances, and most of all confirming the past actually does represent the future - and doing that mathematically not just announcing it - is needed for any estimates of the future to have any credibility. Otherwise it's just an uninformed bad guess.
