I apologize for the long title.
This week end my cycling partner and I rode a century ride from Gunnison Colorado to Crested Butte, Colorado for a fund raiser for our son's collegiate cycling team. It was 100 miles, and 8,000 feet of climbing. When we were at a rest stop before heading up to Kebler Pass, before going down to Crested Butte, I asked what's the road like ahead. Since we had been on dirt, with our road bikes for the last 26 miles at 5% to 6% grade, I was interested on how much longer our stay in Hell was going to be.
Oh, it's not bad, it's paved around the next corner, you go over the summit than down a paved road, with a bit of dirt before you get into Crested Butte.
Well, not bad meant 6% to 7% and a little bit of dirt meant 7 miles on a rapid descent before getting to the town limits.
NEVER EVER USE ORDINAL VALUES TO SPEAK ABOUT ANYTHING IMPORTANT
A little bit for a 21 year old collegiate racer, is a whole lot for a 60'ish century rider.
The First Problem in Project Management Rankings
I come to competency assessments once in awhile in our domain. There is some discussion around these on LinkedIn and I'd like to explain why these approaches are flawed in a fundamental way.
Not that we don't want or need someway to assess the competency of project managers or brain surgeons for that matter. But the matter of how the assessment takes place through filling out surveys is fundamentlly wrong. This is not opinion, it is primary survey guideance.
First is the notion of Ordinal and Cardinal numbers used to rank something.
- An Ordinal number is just a number. They can be added, subtracted, multiplied. They are an extension of natural numbers. 5 is an ordinal number. So is 1.
- Tell me using a number between 1 and5, how much do you like chocolate ice cream is an Ordinal Scale.
- A Cardinal number is also a natural number, but it measures something.
- 2 Red Tailed Hawks sitting on our back fence looking for something to eat, is a Cardinal number.
- Their dinner may consist of 6 field mice, is a Cardinal number.
So Here's The Problem
Say there is a ranking of projet complexity that asks the question What are the implications of this project? There is a 1 to 4 scale, and a 2 is Moderate and says there are some limited social implications due to public visibility of the new system.
So what is the unit of measures- the Cardinal value - of "some," "limited," and "visibility."
Here's another example.
A project complexity measure might be the number of interfaces. Interfaces of the technology, interfaces of people. The question is Project Interfaces Ranked a 2, meaning fairly large number of interfaces do to the number of locations.
How big is Large, 10, 100, 1,000,000? Doesn't say.
Here's my favorite, because it's my favorite topic.
The unit of competence is about Planning the work. The performance criteria - for being competent is - defines project deliverable using a Work Breakdown Structure. The evidence for this competency assessment is the WBS and a list of project objectives.
I've seen more WBS's that arepure crap than I care to remember. Build the WBS is a very difficult process if you don't know what a good one looks like. Reading MIL-STD-881C and the examples in the back on WBS's for various domains is child's play compared to having to actually do one.
So Here's the REAL Problem
Using uncalibrated scales to capture levels of competency or levels of anything is a non starter. If you don't have a calibrated scale into which to place your observed or assessed measurements, then you will have no credible basis on which to make suggestion. How big is Big, How steep is steep, how competent is competent?
This happens in Risk Management as well, where the probability of occurance and the conseqeuntal outcomes are Ordinal numbers, usually 1 to 5. Sometimes there are scales on the probability of occurance. Ignoring all that bad approach, here's an example of risk ranking with Cardinal numbers
When we see rankings of attributes or levels of competency for Project Managers and there is no Cardinal descriptions in domain specific language, then close the page and move on. The big push to have competency based PM and NOT have the scales calibrated on a domain specific basis is a waste of time.
One classical example is that Impact measure. A Rank 2 due to limited social implications due to public visibility of the new system means something much different to a nuclear waste site remooval project than it does to the extension to the mall project. In mall project the impact of rank 2 means we have to work with the city and neighbors not to disrupt too much of the ongoing business, maybe look after for wildlife, and stay inside the design codes. For the nuclear waste site it means let's no kill anybody, poision the ground for 10,000 years and not destroy the environment along the way.
