In the agile world, people have datasets from their projects, their friends projects, or made up projects. They use these datasets to support some position. Ranging from Story Points don't work, the Cone of Uncertainty is a hoax, all projects fail, or what ever their favorite cause celeb is for the month.
A Story About Data, Models of Data, and Decision Making
As a graduate student (twice in two different fields) we always started the first semester with a design of experiments course. In this course, we learned about statistics beyond the physics and engineering (systems engineering) needs and how to sort out bad data from good data. While that course and the experimental particle physics work that followed was useful, a school of hard knocks experience always helps.
In the early days long before laptops and small electronics, I used a PDP-8/m, to write Fast Fourier Transform algorithms in Macro-8, the assembly language in under 4K of memory, to process the outputs of an analog to digital converted capture data off the accelerator. This was my starting point for becoming a software engineer rather than a physicist, by the way.
Hooked up to the experiment was a series of oscilloscopes with Poloride cameras attached to capture the signal of something interesting to the Principle Investigators. First-year grad students are labor by the way.
With a set of pictures in hand and a pile of green bar printer outputs, we'd go down the PI's office with Data. The PI was a Nobel Laureate and while he enjoyed the company of grad students and students in general, he had a low tolerance for speculation about anything.
Nice pictures boys and girls (yes we had female grad students). Now go back and get me 5 more of those. Then disassemble and reassemble the experiment and get me a few more. When that's done, I want you to call my colleague at a university on the east coast (we were on the west coast) and have him repeat the process on his equipment to see if you're on the same page with the data. If that turns out to be true, then I want you to go back to your offices and find a couple of theorist grad students and get them to explain WHY you should be seeing that data. When that's done and your pretty picture still hold up, go write a letter to PhysRevLetters and see if people crap all over your idea. If not, come back and we've got something to explore. No get out of my office
This has formed the basis of all I've done since, when I hear or see a set of numbers used to explain some outcome. From early days of writing digital signal processing algorithms in FORTRAN 77 for missile defense system radar - a direct transfer of the math and science of signal processing in particle physics after I abandoned my graduate degree to current day process improvement efforts in program performance management.
If you start with the end in mind and go looking for data to support that end, you're going to end up with egg on your face.
So here's the point
When we hear I have data showing such-and-such is true AND the reason that data shows that is not provided as well - run away, it's bogus.
Root Cause Analysis is a powerful tool for finding the cause of an observed symptom. It is also a powerful tool for finding the cause of an observed outcome.
- WHY is the Uncertainty on your project NOT reducing as needed and NOT following the paradigm that increases the probability of project success by reducing uncertainty as the project progresses? Your project shows it doesn't reduce - why doesn't reduce? No answer? Stop talking and go find out why.
- You're bad at estimating - WHY? What don't you understand about making decisions in the presence uncertainty? What class did you not attend? What book did you not read?
- Our management misuses information we provide them. WHY? Are they incompetent? Did you provide the wrong information?
When you hear some conjecture that violates an established principle, ask WHY. No answer, ignore that conjecture.
