I posted a comment (which got me off on the wrong foot, BTW) about the Faster Better Cheaper concept on a previous post, about FBC "not being that good from my experience."
FBC was a NASA concept of moving programs forward in "faster, better, and cheaper" ways, that resulted in bouncing two spacecraft on Mars.
Dan Ward responded that I needed to produce evidence that FBC was not good to address his conjecture that it was good. Ignoring for the moment all the issues around communicating what "good" or "not good" means, here's THE core problem when we start to talk about a method or a paradigm being good or not-good.
What are the units of measure of Faster, Better, and Cheaper?
- How fast? Faster than we've gone in the past? How fast can we go? What is fast enough? How can we recognize we're going as fast as we can?
- What is the tangible measure of Better? Better than what? Better than before? Better than we can ever be?
- Cheaper? How cheap can we get? When is cheap counter to quality? If we want cheap, what else do we give up?
None is these units of measure are talked about in Faster, Better, Cheaper.
Now for the Hard Part
When there are sets of numbers in tables - say comparisons between doing it one way (FBC) and doing it another way (Not FBC), how many samples are needed to draw conclusions?
The answer is found in the Clopper-Pearson method. This "test" defines how many samples are needed for a given confidence level from a bi-nominal distribution (FBC and not-FBC) with an unknown binomial probability.
In English this means how many projects to I need to sample - some doing FBC, some not doing FBC, before I can draw any conclusions about the impact of FBC. IF I don't know the distribution of their success up front.
Using the Central Limit Theorem or knowledge about the population distribution (which is missing in the NASA reports), an estimate can be made regarding the percentage of sample means that fall within a certain distance from the population mean. In this case if, I have a collection of projects using and not using FBC what are the statistical attributes of these projects regarding success?
The number of samples needed is n0 = (Z2 p x q) / e2
- The level of confidence desired, Z,
- The sampling error permitted e and
- The true proportion of success p.
- q = 1- p
In an unknown population distribution the selection of the three unknowns above is often difficult. Once the desired level of confidence is chosen and the appropriate Z computed from the normal distribution, the sampling error e indicates the amount of error that is acceptable. The third quantity, the true proportion of success p is actually the population parameter the measurement is attempting to quantify.
So the real question is:
How can a value for the true proportion be stated for the value of the sample proportion that is being measured?
When sampling values from the proportion from an unknown distribution (non–normal), some constraints must be placed on the sample size so that the expressions in the proceeding section are valid. For small sample sizes, a lower limit is set for a finite population and a specified confidence interval. Details of these constraints are given in [Yama67] pp. 89–95. The sample size limits were tabulated by C. J. Clopper and E. S. Pearson in [Clop34], [Kend61].
If the proportion estimate is to have a 95% confidence with a 5% error, (these figures are general guidelines described in [Yama67].
The normal approximation for sample and the absolute sample size n follows the guidelines below [Lawr68].
- p = 0.4, n > 50
- p = 0.3 or 0.7, n > 80
- p = 0.2 or 0.8, n > 200
- p = 0.1 or 0.9, n > 600
- p = 0.05 or 0.95 n > 1400
This means that for an 20% / 80% confidence interval, 200 samples are needed if the underlying population statistics are not known.
For small samples of data, as in the NASA reports on FBC, drawing conclusions that are statistically sound is pretty much a non-starter and the conversation reverts to opinion and verb point / counter point. Hence my comment that FBC is a "bad idea," since it can not be shown to be statistically effective for managing projects.
Looking at a dozen projects with huge variances in technology, mission goals, development teams, contractors, operators and principle investigators is not that useful.
This is also the basis of the "non value added" problem with things like the Standish Report, or anyone who conduct a survey of say their students to determine the outcome of a bi-nominal distribution - Yes/No
References
- [Yama67] - Yamane, T., Elementary Sample Theory, Prentice Hall, 1967.
- [Clop34] - Clopper, C, J. and Pearson, E. S., “The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial,” Biometrike, 26, pp. 104, 1934.
- [Kend61] - Kendall, M. G., and Stuart, A., The Advanced Theory of Statistics, Volume 2, Inference and Relationships, Hafner Publishing, 1961.
- [Lawr68] - Lawrence, W. A., “Proposed Methods for Evaluation of Current Trouble Location Manuals – Case 36279–133,” Internal Memo, Bell Telephone Laboratories, December 9 1968.
