In the YouTube presentation from the previous post, titled Predicting the Future, there was a statement made about minute 9:08 where it is stated the only way that you can estimate is if you can look into the future and that requires precognition, which we don't have. For those of us earning our living by looking into the future, this brings a smile.
Starting with machines that look into the future every day. The airtraffic control system looks into the future and throttles the traffic to avoid congestion. Not so well many times because of externalities. But the algorithm of NextGen does this. Positive Train Control does something similar. These two programs I'm familiar with, having work on parts of them in the past. Guideing missiles to their target requires looking into the future to rapidly determine where the target is going to be a few seconds from now, and being there as it arrives. I'm very familiar with this type of estimating having written lots of code the guide that missile.
In the project management domain, forecasting future performance of the team's productivity and quality and taking corrective actions to avoid undesirable outcomes is what we do for a living. This field is called Program Planning and Controls (PP&C). We are PP&C for software intensive programs as well as things that fly, drive, or swim away using that software.
So let's start with the basis of how to forecast the future given the past, a model of the past, or some representation of the past. We make several assumptions. Without accepting these, it's a short discussion.
- The underlying dynamics of the system under control is describable mathematically. This dynamic nature can be stationary or non-stationary. But the underlying system has to behave in such a way that an equilibrium can be achieved at some point. Disruptive systems are difficult to control and making estimates in the presence of disruption is a waste of time.
- The basis of our forecast needs to contain a small number of parameters as possible needed to represent the source of the forecast. This is called parsimony. But care is needed here. This is NOT Occakm's Razor. That's a 700 year old notion long before Rev. Bayes came up with Bayesian statistics. So we need a model with the right number of parameters.
- Here's a notional example of what R can do when forecasting the future.
Where can we learn about all this stuff? For forecasting project performance - cost, schedule, technical performance, labor utilization, quality of product,defect absorption rates, etc. one place is Time Series Analysis and Its Application with R, 3rd Edition. This is my favorite book, because it has working examples of the analysis we need right there in the book, with the R code and the supporting math. And since we're essentially lazy, we can get to the answer without too much effort.
For those who don't like math in the same way they don't like estimates, well this is going to be a problem. Because to make credible forecasts of future behaviour from past behaviour - time series forecasting - math - statistics and probability actually - is required. No way out of it, sorry.
Now with this base established, what are so rules for actually making forecasts?
- We need indicators of performance. Leading indicators are best, but lagging will do if they don't lag too far back. Knowing that the horse has run out into the pasture by seeing her from the kitchen window is a lagging indicator. Knowing the horse has run down the road by getting a call from the sheriff is a little too lagging.
- We need to know what we actually want to measure. The really important point made in the video is that customers don't buy story points, they buy stories. Our core marketing brochure says it a little different. In fact customers don't buy stories, they buy the capabilities the stories enable. This notion of capabilities is not found very often in the IT world. But it is the essence of how we do business in the defense and related industries. The JCIDS is a place where capabilties are identified before any project work is ever done.
- The last thing is we must establish what kind of estimate we want for the future. What is good enough. If you're estimating the cost of a 8 week project that involves a few people, you're wasting your !@#$ing time. Don't do it. Listen to the #NE crowd and don't do it. But if you're estimating the cost, schedule, performance, weight of a flight avionics package for a Mars lander, then that's another thing.
Let's look at this example.
- We have an idea that we'd like to try out. Let's fly through the tail of a comet. Actually have the comet fly by us is more like it.
- How much does that cost? How long will it take to get a machine that will do that? Do we even know how to do this? It's never been done before by the way. How many people will we need on our team? What new physics will we have to invent to make this thing work?
- All good questions about the future. The unknown. It is a knowable unknown, but for now it is unknown.
- So here's the little rub. We need to ask for a sum of money that is not too much and not too little. How much money? We also have to have a credible plan to build this machine on a deadline. The comet is coming on a deadline and it is leaving on a deadline, and it is not coming back for a really long time. We also need to know estimates of our ability to produce on fairly fine grained boundaries. This is much more than a business estimate. It's a project success estimate. If we can't make the machine on time, we fail. We can always go beg for more money. So how many features are needed, how many can we build in the period of performance of the contract. This is called software development and oh yea hardware too.
- Here's the actual mission.
So now what? How can we develop a credible - see that term credible, it's important - forecast of cost, schedule, and needed technical performance. How do we do this? We build a model. Ignore completly those who will quote George Box that all models are wrong, some are useful. In fact ignore anyone who speaks about the use or misuse of models who does not actually have Time Series Analysis, Forecasting and Control, George E.P. Box and G. M.Jenkins on their shelf and has at least once opened it to look at the Box -Jenkins auto regression algorithm.
This is the answer. There are tools, tons of tools, some are even free (R for example), that can be used to make forecasts of the future.
Now back to the term credible. This means that there is a well defined confidence level for the forecast. The error band on that confidence level is also well defined. But the critical piece here is there is a causality between the past and the future, not just a correlation. In the YouTube presentation, near the end when he is showing the differences between Story Points and Items as an example of better forecasting, he states - and may not have even realized it - that explaining the reason why needs more work. Correlation does not mean Causality. Correlation models are very risks, because they are probably wrong.
So all this comes back to a core concept.
- We can forecast the future and do it every day.
- We may not need to forecast the future - make an estimate - if doing so costs more than the value at risk. The value at risk is a parameter of estimating. How much to spend on estimating is driven by how much we have to lose.
- We can only forecast short term items in most instances. But at the same time, we can forecast very long term items as well. I live in the town where several neighbors are climate modelers. They can forecast the weather a few days out. We live in the mountains, so mountain weather is sporty at best. They can forecast (estimate) years, decades, centuries out. They make estimates of the future given a model of the past. To say we need precognition is silly.
- But we do have to have some assessment of past performance. When asked how much will it cost to a person who has never in their life built a Mars Lander flight avionics package, you'd probably not have much confidence in their number.
The very notion of No Estimates fails miserably in the absence of a domain, a context, a value at risk, and other parameters. So without starting out with the applicability of the concept discussion, it rapidly goes in the ditch. But unlike some in the #NE camp, it is not up to those on the outside looking in the accept the concept in the absence of a domain and context. If #NE'ers want to move beyond their own tribe - and maybe they don't - some form of domain, context, and process is needed. The YouTube example is a very nice start. Which by the way doesn't say no estimates it says no story points.
