Like the Chaos surveys of Standish, business surveys suffer from the same simple and simple minded statistical flaws.
The article was about Netflix and how it is struggling against other providers in the customer service side.
While providing good streaming services - the customers like them - the customer service side is lower than the competition.
As an aside our house is a Netflix user, mainly for the TV shows our college students watch at their houses. Multiple streams on a single account for the ~$8.00 per month is a good deal if you live in a college style manner.
But as always the statistical aspects of these surveys is suspect at best. The first approach is to conduct a Z-Test, which can answer the question -
Did Netflix score significantly lower than the population mean, using the samples from those using Netflix alone?
Since the underlying data is not present - like the Standish data - there is no way to conduct this test. But let's look at the average numbers - the scores - and see if the Netflix number is statistically significant.
The Standard Deviation of the mean of 71, with 8 samples, is 3.6. That means that the difference between Netflix's score and the Mean is 2, within the STD. Amazon Instant's 74, is also within 1 STD - higher of course.
So what can be deduced from this statistical reporting.
- No single point value is of use without the variances.
- Reporting of performance, be it customer service or cost and schedule variance cannot be trusted without knowing the underlying statistical processes.
For example,
- What are the statistical properties of each sample? Size, variance?
- Are the samples unique? Are there repeated users of multiple services?
- What are the shapes of the probability density functions for each supplier?
- What are the number of individual supplier samples and the total samples?
- What is the total population of users?
Without this information no credible assessment can be done to see if Netflix's lower customer satisfaction rating is statistically significant.
So remember...
A single point value has little worth in the absence of the underlying statistics.
Stakeholder: What's the planned duration of this project? Project Manager: 14 months. SH: What's the variance? PM: What do you mean? SH: Oh Boy, you don't have a clue do you of the actual duration of your project. Come back with the variance of cost, schedule, and the technical performance of the products you're supposed to be building for me.
