Limitations of metrics as success measurement
I’d like to step back on what success means. Let’s take the example of a teacher. By measuring what is a teacher considered successful? Good average class grade? Good students mood? Whatever idea you come up with, you’ll have to acknowledge very soon that either it cannot be measured quantitatively (students mood) or if it can (average grade), it has limitations (grades cannot really be compared if applied on different topics, teachers could be tempted to bias their grade to be seen as more successful).
Being a “metrics-driven organization” also means that the mission you’re trying to achieve is reduced to your metrics. I hope I’m not shocking anyone if I say that this is a wrong idea in many ways. Many things are not accurately “metricable”. Acknowledge it and accept it or you’re just being intellectually dishonest.
Metrics done right
Metrics in themselves are not an issue. Metrics are necessary for (at least) two reasons: first, it gives an insight to the outside world on what you do, how your activity evolves over time (not only in the numbers but also in the evolution of the metrics you choose). Second, it’s good for people motivation. Having your success measured and being able to say “we were there, we are here and we moved toward the right direction” is a powerful tool. Also, in periods of doubts, it’s a good thing to have a very concrete answer to “where are we going now?”.
What is an issue though is deciding of a metrics and not being crystal clear on where numbers come from, what their source of bias is, what is the limitation of what is being measured. Having a decent study of this prevents (or at least reduces) misinterpretations.
It has been decided that there will be one metrics: contributors. Three categories are being considered: coders, templaters, instructors. For each, a brief description is provided. But I’m sorry, even when making a big effort, I can’t figure out numbers based on these vague definitions.
The question is being asked at 26’15”: how do we define a contributor? I doesn’t find its answer in the video unfortunately, while it’s a the core of the issue being discussed. I’ll try to elaborate on the question and get to the specifics: “when do we consider that one person is an additional contributor?”. If never is defined what the definition of “one more contributor” is, then the metrics has no meaning and as a consequence, “being (un)successful” is meaningless as well.
I’m not saying it’s an easy task, but for sure it is a necessary one, so I set up an Etherpad to try to define what a contributor is. Please join the discussion.