There is a quote from George Box that is used - misused actually - by many populist blogers and authors that goes like this...
All models are, some are useful
Of course this quote is used to avoid asking and answering questions around models, forecasting, and assessment of possible future states of systems, a project being a system.
The actual quote is from Science and Statistics, George E. P. Box, Journal of the American Statistical Association, December 1976, pp. 791-799.
So the actual reading would suggest "all model are wrong" ... and the correct answer cannot be obtained by making the model more elaborate. A simple model is needed.
But how simple? That's always the challenge.
The nonsense that #NoEstimates is the answer is of cource just as foolish as overparameterized and overelaborated models. This of course is lost on both sides of the modeling discussion.
There is a popular quote from George Box about all models are wrong, but some models are useful. This quote is many times used by people suggesting some new or innovative approach to a problem that has been around long time.
While this quote has a pithy ring to it, I'm almost certain those using the quote do so without actually reading the book where is was used.
In the early 1970s econometric models had been constructed for a number of countries using time series data. They were largely static, with responses to change assumed to take place within one period, irrespective of whether it was a year or a quarter.
The time series models of Box and Jenkins stood in stark contrast to these naive and static models. The Box and Jenkins used a single variable in isolation and dynamics played the central role. A few studies compared the two approaches and concluded that univariate time series models provided the more accurate short-term forecasts.
This was a turning point because it implied that dynamics were more important for understanding short-run movements than economic relationships as then understood. The emphasis in time series econometrics therefore shifted from modelling large simultaneous systems to taking account of dynamic interactions.
Box and Jenkins were influential not so much because what they said was new, but because they said it well. Their contribution was to show how the dynamic properties of observed series could be matched to those of theoretical models.
The Project Management Point
Models are a critical component of any credible forecast of cost, schedule, and technical performance. Without these models it is actually Guessing as so many critics of estimating are fond of stating. With credible models, forecasting becomes a tool used to increase the probability of project success.
So next time some self-proclaimed person uses Box's quote, ask if they have his book in the shelf and on what page that quote appears.
There have several rounds of how to use analogies and how not to use analogies in the past few years.
These involved notions like agile is an untended garden. Actually and untended garden and called a weed patch.
Or we can't really stop and develop a strategy, because we're always putting out fires. Actually firemen rarely put out fires. They spend the majority of their time preventing fire with fire safety, inspections, fire inspections. Their job is to never have fires start.
And of course for false analogies of the double pendulumas a stand in for chaos and unpredictability. Since the equations of motion are easily defined - an exercise for any upper division physics student - and a MathLab plugin, you can plot the path of the double pendulum.
And my favorite the attractor analogy, in which it is presumed that some how in chaos theory the attractor attracts. Without understanding that those pretty pictures of the attractor are the result of the underlying equations for the system.
So with that in mind, there is a new book from one of my favorite authors, Douglas Hofstadter, Surfaces and Essences. In the book Hofstadter makes the case for analogy as the fuel for creative thinking. Using Robert Oppenheimer's quote...
whether or not we talk about discovery or of invention, analogy is inevitable in human thought, because we come to new things in science with what equipment we have, which is how we have learned to think, and above all how we learned to think about the relatedness of things.
But as always we need to take care to assure that those analogies we use to expand the conversation, don't violate the laws of physics, gardening, or mathematics.
Cost estimating methods have been around for a long time. The current processes found in agile use a points system, sometimes a Fibonacci series to bin the values of the points.
The challenge with this approach is the estimate in agile is not monetized so we can't really tell if the Total Allocated Budget (TAB) is sufficient for the project at any point in time, unless the capacity for and the quality of the ourcomes is steady - that is Level of Effort.
With the LOE approach, the capacity for work is the critical measurement needed for estimating the cost at completion. As well continuous updating of this capacity for work is needed and correctly done on good agile projects.
But there are other issues with this LOE approach on larger projects:
Do we know the variances in the capacity for work and how those variances will impact the final Cost at Completion?
Do we know the interdependencies between the various work products and how they impact the final cost?
Do we know the Aleatory uncertainty - the natural occurrence that can't be fixed and has to have margin.
Do we know the Epistemic uncertainty - the event based risks that need a risk handling plan?
So the examples like that found at Projects @ Work, don't really consider any of the underlying uncertainties in estimating. Without the next level down - statistically adjusted estimates of the work effort, the capacity for work, the quality of that work, and the interdependencies between those work activities and their products, the simple and maybe even simple minded approaches to estimating have limits to scaling.
This is one of those topics where everyone is right in some way, depending on the domain, context in the domain, and scale of that domain. As agile enters the larger acquisition community, where we're spending other peoples money - maybe 100's of millions of dollars, care needs to be taken when applying un-monetized, non-probabilistic, non-joint probability (cross correlations between work element) and non-stochastic forecasting models. The real world is not that simple.
Forecasting involves making projections about future performance on the basis of historical and current data.
The chart below is a synthetic (same as fake) Cost Performance Index at the program level 2½ years. The area on the right is the forecast of possible values of the CPI given the historical behaviour, with tuning parameters and smoothing functions.
The next step is to use real program performance data over some realistic period of time - say 4 year - and use H-W (or a similar forecasting tool), to confirm that a less that complete assessment, say take 2 years of data, forecast a 6 month value, and see of the actual 2 year, 6 month value come out close the to the forecast.
This is all done in R, with very simple commands, once the time series for the CPI is formatted and partitioned properly.
As well, there is a flap over the use of Excel for modeling
econometric processes. Which was addressed more in the NPR segment. More on Excel
and modeling later.
In the original paper -
which has now been updated, there are several key words that
trigger red flags in any credible statistical analysis of dynamic
systems like the economy or complex programs. There is an assessment of the
original paper. Let's start with the response to the original paper
from the original authors addressed to the author of the critique.
This
brings us to the core conceptual issue, which Herndon, Ash and Pollin argue
greatly biases our results. They argue that we (they) use an “unconventional weighting
of summary statistics.” In particular, for each bucket, we (they) take average growth
rates for each country and then take an average of the result.
Averages
without variances are not very useful. We all know the story of measuring the most
likely temperature in two locations Cody Wyoming and Trinidad-Tobago and
the misrepresentation of Averages.
Since
economies and projects are driven by stochastic processes,
we must model their data as random.
Everyone
claiming to forecast the future or make correlations between random
processes must have on their shelf and have demonstrated to have
read How to Lie with Statistics,
Darrell Huff.
So ignoring for the moment the very naive statement by two Harvard
professors of averaging the averages in the absence of the variances, and then
making projections of the correlations, there are other problems.
Using
Excel for modeling can certainly be done, but there are much
better ways to model statistical and probabilistic processes - R for one. MathLab is
another.
Excel
also suffers from the in auditability problem. If you build
a spreadsheet and send it to me to make management decisions, I have to look at every cell to determine the right
formula, right range for that formula, and right data is in the right rows
and columns for that formula to work. A pain in the ass for simple
spreadsheets. A nightmare for larger ones. This is simply bad
modeling. Especially since R is free and an academic version of MathLab or Mathematica is
under $300. Come on Harvard professors, earn your reputation.
So What Does This Mean for Project Performance Analysis?
There are pictures in the links above to show that projects are
statistical processes. So here's the bottom line:
All
numbers in projects and econometric models are random numbers.
Correlations
between the source of these numbers may also be random or at least
changing over time - nonstationary stochastic processes.
Models
that don't consider this information are probably not that useful and may
actually - and I'll be crass here - BOGUS.
If
we are ever to produce credible models of how projects work - cost, schedule, risk, and technical performance models - they must be credible
statistical models. That means averages and averages
of averages are simply not allowed without the variances and
even higher order moments and the correlations between the generating functions of
these random variables.
So however, this turns out with the Bad Math people
at Harvard, we've got to do better.
The
current Earned Value Management processes don't consider the statistical
nature of the performance indices. This is bad. Here's a simple and very understandable paper "Performing Statistical Analysis on Earned Value Data."
As
well, the calculations of future performance use cumulative numbers
which wipe out the varainces, the current period which is a point sample
to compute the Estimate At Completion. This is not only bad it is naive
math.
And
then we are surprised when things don't turn out as expected.
Some final thoughts
Correlation
is not causation.
All
numbers are random numbers drawn from a known or possibly unknown
population. If you don't know the population statistics your assessment of the numbers is
different than if you do.
All
random numbers need variance, standard deviation, and higher order moments
to be considered credible as sources of any analysis.
Drawing
a picture of the average of anything is very sporty
without the proper statistical analysis.
Read
Huff, reread it, read it all the time. Also go buy a good statistics and analysis book.
Start with Advanced Statistics Demystified.
If
there is an academic paper that is being used for public policy has not been
peer reviewed, and then throw it away. The Reinhard and Rogoff paper
is the basis of Paul Ryan's economic plan. He may have really good points,
but it built on sand.
Same
advice from self-proclaimed experts in anything, especially project
management. If there is a PhD thesis that has ZERO references in CITESEER,
ignore it.
Read more in the linked references below to see how bad statistics can go even more bad.
Download R, download the free books, learn how to think and act with credible statistics. Learn how many sample you need, learn how to assess the needed population statistics confidence levels, learn how to make decisions based on confidence levels not absolute numbers. We have a 70% confidence of completing on or before a date, or we have a 70% confidence or completing at or below of specific cost must be the answers when management asks about cost and schedule performance.
This is from an article about the application of Bayesian Statistics to a civil suit in the UK over the source of a building fire.
The idea that you can assign probabilities to events that have already occurred, but where we are ignorant of the result, forms the basis for the Bayesian view of probability. Put very broadly, the 'classical' view of probability is in terms of genuine unpredictability about future events, popularly known as 'chance' or 'aleatory uncertainty'.
The Bayesian interpretation allows probability also to be used to express our uncertainty due to our ignorance, known as 'epistemic uncertainty', and popularly expressed as betting odds. Of course there are all gradations, from pure chance (think radioactive decay) to processes assumed to be pure chance (lottery draws), to future events whose odds depend on a mixture of genuine unpredictability and ignorance of the facts (whether Oscar Pistorius will be convicted of murder), to pure epistemic uncertainty (whether Oscar Pistorius knowingly shot his girlfriend).
When we build probabilistic models of project performance - cost, schedule, and technical - we assume we understand the underlying statistical processes that drive these probabilistic generating functions. These are the aleatory uncertainties in duration, cost, and performance. We define the Probability Density Function in the Monte Carlo Simulator. Then we apply that to the network of work activities (the Integrated Master Schedule), to produce confidence outcomes for completing on or before a planned date and at or below a planned cost. This is all fine and dandy. But we really don't know the underlying drivers that create coupling, correlation, and cross correlations between the work activities, cost, and technical performance. These can be model by discovering the drivers in the network.
For the Epistemic uncertainties we need another modeling tool. The current tools don't actually use Bayesian statistics, rather they use Monte Carlo Simulation and treat the Probability of an Event as an aleatory process integrated with the other PDF's, ranges, and their shapes (Kurtosis and Skew).
We're missing the tools needed to construct a credible epistemic model of how the program works. Using the Integrated Master Schedule (IMS) as the topology for work, the probabilistic behaviours of the work elements at each node - cost, schedule, and technical performance compliance of the products - and the coupling and cohesion of the nodes. With this information - assuming it is credible, which is a HUGE assumption - we could model the behaviour of the program and ask what if questions.
When we get into discussions about policy, spending money, especially government money, and the outcomes from that spend, we as a nation are ill equipped to handle the facts. Not because the facts tell us thing we don't want to hear - that's another topic. But because the facts are not understandable by the general public.
This is No Way What So Ever a Denigration of the General Public, but rather that the complexity of everyday life have overtaken our ability to deal with them. Let's start with H. G. Wells
"Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write." H.G.Wells
So here's an example of "everyday" life in the life of scientific based policy making that is likely lost of most of those needed to support the policy.
We hear all the time about how project managers can't make credible schedules or generate credible cost estimates for IT project or even other types of projects - mega project construction for example.
I'd have to say Flyvbjerg is a bit over the top calling estimaters fools and lairs, but he does have a point about the in-credibility of cost and schedule estimates.
We about how building such estimates is a waste of time and we should switch to doing everything in an agile manner and let the requirements emerge over time while spending the clients money to discover them.
This is an exaggeration I know and it literally doesn't work that way even for agile processes. But it brings out the point that developing credible estimates of cost and schedule is actually hard and requires careful consideration to the processes.
The first thing to recognize is there is no such thing as a credible estimate in the absence of the underlying statistical and resulting probabilistic processes being applied to that estimate. The notion that we can forecast the duration of a work activity or the cost of that activity without a probabilistic confidence to that forecast is näive at best and delusional at worse.
Maybe if Flyvbjerg had used the work "delusional" it would set better with me. Liar implies malicious intent. Fool implies no one else was looking after the work and a "fool" was left in charge of the estimiating process.
What To Do About This?
Here's an example of the probability distribution of a cost estimate for a project.
This is from a model project that has a cost loaded IMS. The model will be only as good as the IMS is good and the resource loading is good. So this approach ONLY works when the model is right. That's a problem of course, but for now let's assume the model is right, since it came from a VERY credible source. David Hulett and I work the Risk and Opportiunity policy segment of the US DOD, along with others from government and industry.
The 50th percentile cost is $2,296,898 (a small project in the A&D world BTW). That means there is a 50/50 chance the project will cost more than that or less than that. That number is the median - the middle of the range. If we want to know what the number is with an 80% confidence, than it is $2,333,153. That says there is an 80% confidence that the cost of the project will be $2.3M or less.
When we speak about schedule, we can use the same terminology. In this case there is an 80% confidence the project will complete on or before 11/06/2015.
These graphs are generated from a Monte Carlo simulation tool applied to a resource loaded Integrated Master Schedule (IMS). If there is any doubt as to why you MUST create a resource loaded schedule, please please put those doubts away. These two graphs are the source for the Joint Confidence Level showing the over all probability distributions for both cost and schedule for the project.
This picture should convince anyone that the "joint" probability of completing on or before the target date (I didn't say what that was) and at or below that planned budget (didn't say what that was either) needs to be modeled in a way that would be considered credible.
Outside of a small domain of aerospace and defense this is rarely if ever done. Most IT projects have some made up budget and schedule. Their strategy is based almost entirely on HOPE with no underlying assessment of the statistical and probabilistic drivers of the cost and schedule, let alone the technical aspects coupled with risk.
And we wonder why IT projects come in late and over budget.
Turns out of course large defense and space programs have the same problem - poor estimating, politically motivated estimating, naive estimating, etc. But they are required to produce data like this, so we know it is bogus sometimes early enough to fix it - sometimes not.
But the real root cause of most serious overages for A&D comes from other sources. Those can be found in the Nunn McCready breach reports at RAND in Volume 1 and Volume 2. IDA has there assessment as well. There are some professional provocateur voices that conjecture all this can be fixed with a good WBS, a scaled down Earned Value Management System, and "firing the program managers when they go over cost and schedule." That of course is probably not the real solution. Read the RAND reports for the root causes of major overages. Read the Feb 18th issue of Aviation Week, "Slaughterhouse Five: If sequestration can rewrite U.S. strategy It's time to kill the sacred cows in industrial policy." F-35's current cracked first compress blade grounding and the world wide subcontracting effort that provides the program with nearly invincible support in congress.
We only wish it were as simple as many would have us believe. But sadly it is one of those wicked problems that we simply have to manage in the presence of. Many have suggested solution - credible ones, not the cocka mamy ones - but those solutions usually don't address the political forces in play when you are working with the public's money.
Why I've Followen in Love with R
In the programming language R, drawing the joint probability plot is done with a simple command
> pairs(X)
produces a scatter plot of the variables defined by the columns in X. Every column in X is plotted against everyother column of X and the resulting n(n-1) plots arranged in a matrix with constant plot scales over the rows and columns of the matrix.
Trending is a critical tool in forecasting the future performance of a project. "We've be doing this well for awhile now, what's going to happen in the future?" is a very common question for project management to ask of the "project controls" staff.
Two indices in Earned Value are the Cost Performance Index (CPI) and Schedule Performance Index (SPI). They are calculated from the Earned Value Management numbers of BCWP (Budgeted Cost of Work Performed), BCWS (Budgeted Cost of Work Scheduled) and ACWP (Actual Cost of Work Performed). This post is not about Earned Value, but about statistical analysis of Earned Value numbers. You can find good tutorials on EVM in several places, including A Gentle Introduction to Earned Value.
What is critical is the forecast for where the project is going with the information from the past. In the standard approach static information is used to compute a forecast of the future. This information is a cumulative measure of CPI/SPI for all past periods and the current period of performance. Together these are used in a linear formula to forecast future performance - again in a linear, non-probabilistic manner.
All of this has lead me to think about the probabilistic processes needed to improve the credibility of project performance measures. I discovered through our BioChem son, the programming system R. R is a functional programming language and system for statistics and probability.
Along with R the US DOD has rolled out an XML format for capturing the monthly Integrated Performance Management Report. The time series of CPI and SPI are available monthly in electronic readable form. With this data forecasting using R and the myriad of tools can be done.
Using a simple functional command line,
will produce a nice picture of the time series data with forecast value at defined confidence levels.
The next step is to do this on actual project performance data (using SPI/CPI) and see what can be reveiled about future performance.
The problem to be solved is simple:
The current Earned Value Management reporting hides all the variances at the work package and control account level through the cumulative reporting. The data is there, it is just not reported.
The current period data also cumulates the variances from the work packages and then adds that data to the past reporting.
There is not assessment of the statistical nature of the underlying data, so the result is a scalar, linear report of a underlying stochastic process, hiding all the real drivers by ignoring the variances and summing to the top. In Darrell Huff's How to Lie with Statistics, this is a recommended practice.
There is no adjustment of the future forecast for the past probabilistic behaviour. Let alone risk, technical performance, measures of effectiveness, and compliance with Key Performance Parameters (but JROC and program). There is just a linear projection of the future from the rolled up past, hiding the variances, and ignoring all the excursions that created those variances.
Mathematically this means that what drove the variances in the past is assumed to no longer be driving the future. Those sources of variance are washed out. This may be the case from corrective actions for the Epistemic uncertainties (uncertainty due to lack of knowledge). But the Aleatory uncertainties (uncertainty due to randomness or luck) that drive the variances are not subject to corrective actions. They are baked into the system and no amount of new knowledge or corrective effort is going to make them go away.
This is simply bad data analysis in any discipline I've ever worked, except program controls.
There is another round of discussion of the Chaos aspects of software development projects and the models of this chaos.
The chaotic ones that are totally unpredictable like a double pendulum
A double pendulum (two pendulums attached to each other) is also a simple system. It is easy to make and easy to understand. And yet, it undergoes unpredictable chaotic motion due to a high sensitivity to the initial setup of the pendulum.
The common paradigm of the Double Pendulum is misunderstood and misapplied to the problems of modeling systems.
A double pendulum, is one pendulum suspended from another, shown below. The path of the bottom element (m2) is a
potentially chaotic system. When certain parameter ranges undergo a slight
change in one of the initial starting conditions there can be dramatic effect on the
subsequent motion of the pendulum. As a result the motion of a double pendulum
extremely difficult to predict without a model of the system. The external observation is the pendulum exhibits seemingly random or chaotic behavior.
The first step in solving for the motion of m2 is to understand the difference between chaotic motion and unpredictable motion. The double pendulum can behave in a chaotic manner. But the motion of m2 as well as m2 is predictable. These terms get mixed up and we need to keep them straight if we're going to apply them to paradigms outside of the double pendulum.
Chaos can described as:
Being sensitive to initial conditions.
Possessing topological mixing, meaning the system evolves over time.
Possessing a density of periodic orbits, meaning every point in the space is approached arbitrarily closely by periodic orbits.
Why Is This Important for Project Management
When we use analogies in project management to explain a situation, we need to actually have the right analogy. The Double Pendulum is a common explanation of chaotic behaviour. But the Double Pendulum, while behaving chaotically, is a deterministic system driven by the solution to the Hamiltonian equations of motion. This is called Hamiltonian Chaos.
The Hamiltonian description of motion uses momentum as the variable.
Depending on the initial conditions the system behaves regularly or show chaotic behaviour. The solution to this equation can be found in Java at Physics Department of the University of Buffalo.
So we can see that the solution to the double pendulum is not unknown. The plot of the path can be known by running the simulation for the needed period of time.
What this means is using the Double Pendulum as an analogy for chaotic behaviour of projects requires careful wording and application. Human systems don't not have Hamiltonian's, double pendulums do. With Hamiltonian, we can write a simualtion to show the motion of the pendulum, run it and see whatit does - Exactly what it does. We can run the simulation forward in time and Predict what it will do sometime in the future.
No mysteries about the path of the pendulum parts, they are completly specified by the solutions to the Hamiltonian. They may look chaotic, unpredictable, and wandering all over the place. But the equations of motion are explictly defined in the Hamiltonian solution and its simulaiton in our favorite language, Java or Mathematica.
Bayesian: being, relating to, or involving statistical methods that assign probabilities
or distributions to events (as rain tomorrow) or parameters (as a population
mean) based on experience or best guesses before experimentation and data collection
and that apply Bayes' theorem to revise the probabilities and distributions
after obtaining experimental data.
This is Bayes Theorem. In the context of Bayes, probability speaks the the degree of belief. We believe that the cost of our project will come in under $10M if all the work goes as planned.
Bayes' theorem links the degree of belief in a proposition before and after accounting for evidence. For Bayes to be used in project work, we need evidence of what happened in the past. Were there any projects of a similar nature that came in under $10M.
This approach, while possibly counter intuitive, is distinctly different from taking a guess about what the project would cost. It says that to have any credible estimate, we need to know something about what happened in the past.
In project work, a Bayesian network is formed as a probabilistic graph - a directed graph. This less than definitive data about the project represented in the directed graph (the sequence of work activities), can apply Bayesian statistics to determine probabilities in the presence of uncertainty.
From Bayes An Essay towards Solving a Problem in the Doctrine of Chances. By the Late Rev. Mr. Bayes, F. R. S. Communicated by Mr. Price, in a Letter to John Canton, A. M. F. R. S. Philosophical Transactions (1683-1775)
When we talk about measuring things in project management, there are four types of outcomes for two attributes of that measurement.
The measurement can be accurate and the measurement can be precision. We want it to be both. But without knowing if it is both, we're probably not going to get what we expected.
This is translated into our estimating processes like this. We need to know the accuracy of the estiimate as well as the precision of that estimate.
It does no good to have a precise estimate, that is not accurate. Or to have an accurate estimate that is not precise. Both are needed, but we also must know if we have accuracy and precise in the presence of natural variability of the underlying processes.
So when someone says I estimate that this project or this part of the project will be $127,700 and take 6 months, you can ask them back accuracy and precision questions.
Lynda Bourne has a nice post titled Prediction is Difficult. In the end she speaks about the human failings of predicting and the difficulties of getting senior management is accept these predictions.
First I'd like to suggest a change from prediction to forecasting. The forecast includes a range of possible outcomes and the confidence level on that range. This is one root cause of senior managements lack of understanding about business numbers. They assume (in my experience) that project numbers are like accounting numbers. They are not of course, but as project managers we don't make that clear by using ranges and confidence intervals on those ranges, so what else they to think.
How To Speak About Project Numbers
When we speak about cost, schedule, and technical performance it MUST be in the following manner:
We have a 70% confidence in completing on or before September 13th, 2013
Our cost for this project has a 70% confidence of be $1.6M or less
The sustained throughput of the data channel will be 950Mb other the Time Triggered Gigibit bus with a 80% confidence.
How Do We Develop These Confidence Discussions?
The very first we DON'T, as Lynda suggests is ask some subject matter expert. SME is the least confident answer. The highest confidence answer in our domain is called Past Performance.
We've done the 17 times before and have a statistically sound basis of our estimate, and have a 80% confidence that this time the duration will be 18 weeks or less.
The source of this data may not be directly available, so the next best source is Reference Class Forecasting. A good place to look to see how this is applied in project management is Bent Flyvbjerg's "From Noble Prize to Project Management." (Follow the links from his wikipedia page to find many other discussions about Reference Class Forecasting).
All Point Estimates Are Wrong, Without a Variance and a Confidence on that Variance
Making single point estimates - this project will complete on August 17th of this year, is simply wrong. There is not one ounce of credibility in that statement. Without stating upfront what the range of possible finish dates, and confidence levels for each of those range (binning this is called) or have a Probability Distribution Function (PDF) that shows the confidence of an on or before completion, the production or forecast is simply bogus.
Once in awhile, I come across an concept I didn't think I would. Something I've bee fooling around with that turns out to be exciting in a special kind of way. Here's one of those things.
How do you construct the Golden Ratio with a simple pen and paper, a compass, and a ruler?
Here's how.
Draw a triangle with dimension of 100 and 50
Take the compass and scribe an arc anchored on B through C
Take the compass and scribe an arc from the intersection of the previous arc along the line segment between A and B
The ratio of distance between the two line segments between A and C is the Golden Ratio = 1.6180223 ...
There several ways to do this, but this approach came when fiddling around with Visio.
is a license to stop thinking intelligently about models and how they are used for nearly everything we do. Models are simply aids to thought, especially thoughts about actions in systems that are complex. Models are used to understand more about the system and, in that way, can help the decision maker make decisions that will bring about better results.
I wonder if those self-proclaimed voices have actually read Box's book Robustness in the Strategy of Scientific Model Building, or had to build models of complex systems prior to constructing physical or virtual systems, or worked in a area where modeling of systems was the basis for making management decisions with other peoples money.
Box's book is about the philosophy of robust procedures. It argues that the circa 1979 emphasis by statistical researchers on ad hoc methods of robust estimation is mistaken. Classical methods of estimation should be retained using models which more appropriately represent reality. Attention should not be confined merely to discrepancies arising from outliers and heavy tailed distributions but should be extended to include serial dependence, need for transformation and other problems. Some researches of this kind using Bayes theorem are discussed.
So without actually reading the book, applying the classical estimating processes, examining the Bayesian processes, or incorporating these statistical models with the prior conditions that drive the modeling parameters, the use of Box's quote is just a glib toss off quote so common now with the 3.0 version of actually doing the hard work to solve hard problems.
Of course models are useful. The notion they are not is naive and misinformed. There is a model of the air traffic control system that is used everyday in our Enroute ATC here in Longmont, to get aircraft across the US and into the ATCT/TRACON Center at DIA. There are models of drug toxicity used for clinical trials, models of vibration damping used for stability control systems, models of cost and schedule adherence, models of environmental cleanup processes, models of climate change, models of capitalization of assets, models of nearly everything we depend on in our modern life.
What George Box was speaking about is how the underlying statistical processes are used or misused. This is the frequentist versus Bayesian conversation, that is never ending. This is also the continuous discussion in journal articles based on statistical inference. So stop reading the grocery store populist magazines and start subscribing to Science and Nature
If those Management 3.0 and Agile thought leader voices would actually read the book, go take a statistical forecasting class at the university, stop reading the populist repeating of bad quotes, read the Bayesian statistics text books used on our biopharma, probabilistic risk analysis, and non-linear dynamics programs, then go themselves work a program using their new found knowledge, maybe they'd not be so glib about All Models are Wrong (because that is a falsehood), some models are useful (a truth if you actually understand and can apply modeling).
In our daily work we depend on several modeling tools and the models they produce:
So time to buck up and do some homework on applying modeling to complex problems. In the end when those thought leaders complain about "populist" models like OODA they can't seem to grasp that George Box was actually talking about mathemtcial and computation models. Not their simple non-mathematical models. The difference between populist thought and mathematical and scientifica thought is in the populist world you can't calculate anything, so you have to have the dicussion using arm waving.
This week end my cycling partner and I rode a century ride from Gunnison Colorado to Crested Butte, Colorado for a fund raiser for our son's collegiate cycling team. It was 100 miles, and 8,000 feet of climbing. When we were at a rest stop before heading up to Kebler Pass, before going down to Crested Butte, I asked what's the road like ahead. Since we had been on dirt, with our road bikes for the last 26 miles at 5% to 6% grade, I was interested on how much longer our stay in Hell was going to be.
Oh, it's not bad, it's paved around the next corner, you go over the summit than down a paved road, with a bit of dirt before you get into Crested Butte.
Well, not bad meant 6% to 7% and a little bit of dirt meant 7 miles on a rapid descent before getting to the town limits.
NEVER EVER USE ORDINAL VALUES TO SPEAK ABOUT ANYTHING IMPORTANT
A little bit for a 21 year old collegiate racer, is a whole lot for a 60'ish century rider.
The First Problem in Project Management Rankings
I come to competency assessments once in awhile in our domain. There is some discussion around these on LinkedIn and I'd like to explain why these approaches are flawed in a fundamental way.
Not that we don't want or need someway to assess the competency of project managers or brain surgeons for that matter. But the matter of how the assessment takes place through filling out surveys is fundamentlly wrong. This is not opinion, it is primary survey guideance.
First is the notion of Ordinal and Cardinal numbers used to rank something.
An Ordinal number is just a number. They can be added, subtracted, multiplied. They are an extension of natural numbers. 5 is an ordinal number. So is 1.
Tell me using a number between 1 and5, how much do you like chocolate ice cream is an Ordinal Scale.
A Cardinal number is also a natural number, but it measures something.
2 Red Tailed Hawks sitting on our back fence looking for something to eat, is a Cardinal number.
Their dinner may consist of 6 field mice, is a Cardinal number.
So Here's The Problem
Say there is a ranking of projet complexity that asks the question What are the implications of this project? There is a 1 to 4 scale, and a 2 is Moderate and says there are some limited social implications due to public visibility of the new system.
So what is the unit of measures- the Cardinal value - of "some," "limited," and "visibility."
Here's another example.
A project complexity measure might be the number of interfaces. Interfaces of the technology, interfaces of people. The question is Project Interfaces Ranked a 2, meaning fairly large number of interfaces do to the number of locations.
How big is Large, 10, 100, 1,000,000? Doesn't say.
Here's my favorite, because it's my favorite topic.
The unit of competence is about Planning the work. The performance criteria - for being competent is - defines project deliverable using a Work Breakdown Structure. The evidence for this competency assessment is the WBS and a list of project objectives.
I've seen more WBS's that arepure crap than I care to remember. Build the WBS is a very difficult process if you don't know what a good one looks like. Reading MIL-STD-881C and the examples in the back on WBS's for various domains is child's play compared to having to actually do one.
So Here's the REAL Problem
Using uncalibrated scales to capture levels of competency or levels of anything is a non starter. If you don't have a calibrated scale into which to place your observed or assessed measurements, then you will have no credible basis on which to make suggestion. How big is Big, How steep is steep, how competent is competent?
This happens in Risk Management as well, where the probability of occurance and the conseqeuntal outcomes are Ordinal numbers, usually 1 to 5. Sometimes there are scales on the probability of occurance. Ignoring all that bad approach, here's an example of risk ranking with Cardinal numbers
When we see rankings of attributes or levels of competency for Project Managers and there is no Cardinal descriptions in domain specific language, then close the page and move on. The big push to have competency based PM and NOT have the scales calibrated on a domain specific basis is a waste of time.
One classical example is that Impact measure. A Rank 2 due to limited social implications due to public visibility of the new system means something much different to a nuclear waste site remooval project than it does to the extension to the mall project. In mall project the impact of rank 2 means we have to work with the city and neighbors not to disrupt too much of the ongoing business, maybe look after for wildlife, and stay inside the design codes. For the nuclear waste site it means let's no kill anybody, poision the ground for 10,000 years and not destroy the environment along the way.
There are the usual reactions to George Box's quote about models...
Essentially, all models are wrong, but some are useful.
... to make the wrong headed position that we can't use models for our benefit.
Here's a better basis for our needs ...
This model will be a simplification and an idealization, and consequently a falsification. It is to be hoped that the features retained for discussion are those of greatest importance in the present state of knowledge
One of the features that distinguishes applied mathematics is its interest in framing important questions about the observed world in a mathematical way. This process of translation into a mathematical form can give a better handle for certain problems that would be otherwise not possible.
This is called Modeling. It combines formal reasoning with intuitive insights.
The simplest models can be powerful descriptions of the real world and at the same time demonstrate the naive assumptions of some self proclaimed thought leaders. Newton's laws of motion, formulated several centuries ago, provide an early example of mathematical modeling.
F = ma = m dv/dt = dx2/dt2
This is a simple model where acceleration is described as the derivative of the velocity of the object being moved by the force. Note that the second derivative of the position of the object is now NON-LINEAR as all second derivatives are.
So the notion that we've all be taught to think in linear modelsis of course nonsense.
But What Does This Have To Do With Managing Projects?
The elements of projects, the processes used to manage them, the variables of cost, schedule, technical performance, and yes the people are non-linear by their very nature. No need to buy that notion that we've all been taught the world was linear and it's a new discovery it's not, unless of course you were asleep in the freshman physics class.
Not only that; all the world is a non-stationary stochastic process. This means everything is random and described by a random process whose probability distribution shifts in time. This really means the thing that is causing the randomness changes the values it creates as time moves from left to right. The weather is a nonstationary stochastic process. The stock market, the traffic flow in a major city, supply chains.
So if we are modeling a project, we must come to understand if we are seeking predictive outcomes and not modeling the underlying system as a nonstationary stochastic process, we're going to be disappointed with the outcomes of our model. We'll be late, over budget, and the thing we're building will probably not work as predicted.
We'll be using the naive interpretation of Box. All models are wrong, some are useful.
And oh by the way the quote from those who claim all models fail should be told to the designers of the very complex system of getting Curiosity on Mars, where many of the systems could only be modeled, not actually tested. And listen to the engineers as they talked about confirming their models worked when Curiosity sent back the first 64x64 picture of the dust settling with the message I'm Here.
Listen to the first words here. The forecast (model) miss distance was 232 meter over the planned landing. The landing spot was modeled. Then listen to how they abandoned an update to their model at minute 2:35, since the previous one was within spitting distance of the planned landed spot.
All the world's a model, it's just how much fidelity we need for our efforts and how much fidelity we want to search for or are willing to pay for.
One More Thought
If we go back to the opening quote from Turning and the Morphogenesis example, we can address a very common populist misunderstanding about self organizing systems.
There is a popular quote floating around
Self-Organization is a process of attraction and repulsion in which the internal organization of a system, normally an open system, increases in complexity without being guided or managed by an outside source.
The premise is there is organization without management. But of course like most populist descriptions of complex systems, they miss a fundamental concept. Those repulsions and attractions are GUIDED by the models, equations of motion, and rules of physics, chemistry, and biology. They are MANAGED by these rules.
Morphogensis is the process whereby form and pattern evolve in a biological system. There are two simple models for the purposes of this post. Both models start with an undifferentiated and spatially homogeneous distribution of some biological substance. This distribution represents an unstable equilibrium state, and in both models a slight disturbance permits nonhomogeneous spatial pattern to develop over time. This pattern emerges through the rules of interaction defined by physics and chemistry - in this example.
Read the paper and learn that the notion of no external control is a false analogy. There are always rules and laws for the interaction of the individual components of the system. We may not be able to see them, or even understand them if we saw them. But they are there. To be otherwaise would defy the laws of physics.
This is the challenge for using actual science analogies for what is very sketchy, untested, anecdotal, and possibly untestable opinions about how human systems work. Be careful. Human systems are complex, but when populist explanations starting with actual science enter in, take a deep breath, and put the models in the stack called notional. These models may or may not be useful. The irony of course, is the populist approach fulfills Box's statement, the models are wrong.
There are several shelves of books about many topics in our house. Project Management and related topics are in the Office. The list of books below have specific attributes. They have not only been read, they have been marked up, reread, and used in practice. I have many books that are read, but not marked up. Other books that have been read, marked up, but not put into practice. And a hand full of books that have all three attributes plus one more. The contents of the book has changes how I do my job.
There are other books that I've read, marked up, and concluded the content was either not applicable to the problems I work on, or in a few cases complete nonsense. By this I mean the content is not only not appropriate to the problems I work, but if applied will result in disappointment.
So here's my must have list. This list is not populist in nature. By populist I mean a book or paper in which you CANNOT calculate some outcome. A populist book is something to read if you are just coming to the topic and want to discover the lay of the land.The books listed here are nit populist books
This list is in no particular order, other than they were taken off the shelf in this order. One other attribute is if I know or met the author.
General Project Management
These books are about the general concepts of managing projects, including agile projects. They are from authors who are or have recently benn project and program managers. This is important for the simple reason that who wants to read about something from someone that actually doesn't practice what is in the book. By the way, project management techniques that do not address agile approaches are not that useful anymore. As well, authors with 1996 style credentials have likely been passed by the current approaches as well.
The Handbook of Program Management, James T. Brown - I have met Dr. Brown. He is a former Program Manager at NASA. This is a book about program management. Program Management is about managing collections of projects. It is a hands on instructional advice type of book.
The Radical Elements of Radical Success, Dan Ward - this is a book from a person practicing Program Management. Dan is an Air Force procurement officer, just back from theater. It's one of those books that causes me to think. Dan and I have had long discussions about the issues in government procurement. Here's a pre-defined search for Dan's work
Project Management the Agile Way, John C, Goodpasture - I know John professionally, but have never been face to face. John and I share many common principles. This book is published by Management Concepts, which I do work for. MC publishes books, provides courses, and services for the Federal Government. John's book shows you how to deploy agile processes in the presence of the governance based environment. One where you are spending other peoples money and are accountable for success to external parties, like the Federal Government, a Board of Directors, and share holders.
Flawless Execution, James D. Murphy - I cam across the After Burners at a PMI conference. The speakers are all ex-fighter pilots. There approach is logical, concise, and clear. The perfect paradigm for managing projects. This approach is not for everyone. It holds all members accountable for their actions, builds teams in ways not found in any other approach and is not one of those touchy feely styles.
Beyond the Hype, Robert G. Eccles and Nitin Nohria - this is one of those must read for anyone interested in processes. Many if not most of the new ideas are actually old ideas in new cloths. Or worse, poor copies of old ideas promoted by self aggrandizing authors who know how to market. This book speaks to the problem of not knowing the past, so you
Reinventing Project Management, Aaron J. Shenhar and Dav Dvir - this is a good starting point for PM in the IT environment. With field proven approaches, the books provides the step-by-step activities needed to manage projects in the Enterprise business domains.
The 7 Habits of Highly Effective People, Stephen R. Covey - this is one of my populist books that has direct applicability to project management. Our method - Performance Based Management - contains most of the content of Covey's book is for Project Manager, including the Performance Based Management process we use on our programs.
The Seven Secrets if How to Think Like A Rocket Scientist -Jim Longuski describes the processes needed to manage complex projects. It's a check list of items rarely thought about in the normal project management paradigm. This is done, because the domain Jim works in has projects that blow up and kill people of not properly managed.
A Bee in a Cathedral- Joel Levy's book is here because using analogies in the project management world is very common. And most often the analogies are bogus. read this to see how to use analogies properly, since they are powerful tools to get points across, reveal understanding.
Leadership
There are so many books on leadership in the project management space. I'm simply burned out listening to all the voices on leadership from folks who speak and write about the topic, but who have not actually lead people under circumstances that require real leadership. Here's the books that you need from those people who have.
How NASA Builds Teams - Pellerin led the Hubble Space Telescope program and speaks to how that worked and didn't work.
Augustine's Laws - Norman was the leader of Martin Marietta here in Denver. These Laws are general business processes but for high tech firms.
In the aerospace and defense business Program Management is a Systems Engineering paradigm. This is because the technical architecture of the program has to be matched with the programmatic architecture. The product that is being designed must also be built and shipped. This means the value stream flow of work needs an architecture as well. This is the purpose of the Integrated Master Plan (IMP) and the Systems Engineering Management Plan (SEMP)
These books are the basis for being a good project manager. Here's a list of specific topics, meaning more technical, needed for project success.
The Art of Systems Architecting, Second Editon, Mark W. Maier and Eberhardt Rechtin - everything is a system. No those silly descriptions of those populist systems where you can't design or calculate anything. But system we encounter in the real world. The air traffic control system, the banking system, the internet, the enterprise IT systems, manned space flight, building the 787. Systems like that. Rechtin is the father of the approaches used to build real systems. Yes, people are in these systems. And people are an important part, but in these types of systems, the components of the system are the basis of success.
Risk Management
There are lots of risk management books, articles, websites, and Bloggers out there. Here are the two you should own and read.
Risk Happens!: Managing Risk and Avoiding Failure in Business Projects, Mike Clayton - I've corresponded with Mike for several years. This is a great book for starting the process of risk management. It follows the steps defined in many locations - the best is the SEI Continuous Risk Management (CRM) model. There are simialr guides from DOD and DOE. PMI has a Risk Management Practice Guide, but like most things PMI they have their own way of doing things and rearrange the work processes.
As an aside determining risk drivers is a critical success factor for any Risk Management process. Typically risk management means making lists of risks, ranking them, and doing some kind of handling. This is not only naive, but it is wrong. You must determine which risks drive which undesirable outcomes of your project. Start with A Framework for Categorizing Key Drivers of Risk and Taxonomy-Based Risk Identification. I worked with one of the authors.
Effective Risk Management: Some Keys to Success, Edmund Conrow - I've worked with Ed on a proposal and he turned me on to several concepts that I use daily. The first is the serious error in the PMBOK about multiplying the probability of occurrence with the ranked outcome to produce a risk score. This is not mathematically possible, since both are probability distirbution functions. The Method of Moments can help if you don't have a Monte Carlo Simulator. But if you don't buy Ed's book and read how to do Risk Management the right way.
There aren't really a lot of books about Earned Value Management. But there is a ton of misinformation about how EV is used, no used, applied, and misapplied. So these books and sources should be the starting point
Performance Based Earned Value - Paul's book came out a few years ago, and the ideas have been with me ever since. The notion that Technical Performance Measures are part of Earned Value was present in the C/SCSC days, but was lost in 748-B and needs to come back. See Paul web site of the same name for good background.
Earned Value Management Intent Guide - is the National Defense Industry Association's offical document for US Government Earned Value Management. This guide tells you how to comply with the 32 Guidelines of Earned Value Management. For most projects outside the government this is not applicable. But you should read this anyway, just to have an understanding of what the largest implementations look like.
There are other sources of Earned Value, even in the agile world. But care is needed. To properly use Earned Value you must be able to do several things:
Maintain a baseline of the capabilties at least - the end outcomes of the project should not be changing
Know the total budget - the end budget and some notion of how this is spread over time.
Have some measure of physical percent complete and know how much physical percent complete you should be at when you measure it.
If you can't do these, you shoulded be considering Earned Value for your project
Statistics and Math
Project managers need to know somethings about probability and statistics. All the numbers in project management, cost, durations, head count, technical performance are random numbers. In the absence of knowing how to use statistics and probability, you're going to be late and over budget before you start.
How to Lie With Statistics- every PM must have this book, must have read it, and must put it to use. When you have a conversation with anyone on the project, have this conversation in the context of this book.
Then get Advanced Statistics Demystified or Statistics for Dummies and learn about sampling, variances, inferences, and other models of how your project is going to be working even if you don't know that is working that way.
Recent Comments