These days, organizations have a lot of information coming at them and it can be hard to see clearly through the spreadsheets, dashboards and data visualizations. It gets difficult to measure what is important. Why?
Because sometimes the things that are easy to measure aren’t that important and sometimes the things that are hard to measure are very important.
In an information deluge world, when a measurement already exists it can become important by default. Even if it doesn’t quite get at what we want it to, it’s better than nothing… right?
Before we know it, we’re using these measurements to make clever charts and pretty diagrams, looking at those numbers over time and making decisions based on the ups and downs. It makes us feel productive, we have something to point to, we can give ourselves targets and meet goals.
The problem is that just because we have data doesn’t mean we are measuring what is important or answering the questions that matter.
To measure what matters we need to know what matters. We have to start by saying “What do we care about? What do we value?” It sounds easy, but I’ll tell you – most of the time what an organization values isn’t found in a spreadsheet.
I recently did some work for the excellent Software Carpentry Foundation where we asked the hundreds of volunteer instructors what they got out of their time teaching technical seminars for scientists. From the short survey, we learned that instructors enjoyed networking with the diverse international community, learning to be better teachers and teaching the content.
Luckily, Software Carpentry has some pretty good metrics for most of this. You could argue that they might need more creative ways to measure networking and community. But still, you can point to how many teachers there are, how many workshops were offered, how many people were engaged.
What was surprising was that SWC instructors also consistently valued the online materials they use to teach their classes which are freely available to everyone. These materials are lessons about topics for scientists who want to learn R or Python for instance. Instructors told us is that they liked sending the information to colleagues or students who couldn’t make it to a workshop or wanted to learn material on their own.
Software Carpentry Foundation had always created their lessons in an open source manner, but knowing that people value this material and that members of their community pass along the materials changes the landscape. Now they can look at the lessons as something that offers an impact instead of just the workshops. It gives them more outcomes to measure their impact.
We have incredible opportunities when we get to play with data, but we need analysis that has a targeted inquiry about what matters, to whom and why. Without knowing what we care about, how can we measure what matters?