There is a small number of meta-questions about knowledge management that people regularly grapple with. The most obvious is “what is knowledge management?” After that, the next most frequently asked must be “how do you measure KM success?” I have found at least 23 answers (or challenges) to that question, and there are undoubtedly more. I recently found an interesting commentary on the measurement game in a different context, which might shed some light on the matter.
I maintain a watching brief on the higher education sector in the UK. Partly for nostalgic reasons, partly to see trends that might affect our future lawyers, and partly because serendipity is part of this job and I think that only comes with practised observation. So I couldn’t miss Jonathan Wolff’s recent insight into the way in which the UK funding and quality agencies monitor universities.
Suppose you have applied for a job, any job. You are at one of those macho interviews where the panel members compete to see who can make you sweat the most. And this is the winning question: how do you plan to monitor and evaluate your own performance in the role? …
Suppose your job is in business of some sort and, ultimately, you are employed to make the company money… In the end, the only thing that matters, then, is the profit you bring in. But it may take some time to build up a client base and to gather the dosh. It would be foolish to say that in the short term you should be judged on how much profit you make for the company. Rather you should monitor your activity: how many meetings you have taken, how many letters and emails you have sent, how many briefings you have been to. But, of course, that is only for openers. If the meetings don’t result in business, then you are wasting your time. So in the second phase of monitoring, you stop counting meetings and start counting things like contracts signed, goods shipped, turnover generated, or any other objective sign of real interaction.
But, once more, this is only an interim goal. You are there not to generate turnover, but profit. And once you have been around long enough that is the only thing that matters. In the third and final phase you count how much you make for the company, and stop worrying about meetings, letters or contracts signed. Who cares about how many of these there are if the bottom line stays juicy enough?
Pithily put, and accurate too. (Perhaps one should expect nothing less from a professor of philosophy at the institution inspired by Jeremy Bentham.) Unfortunately, Wolff’s tale does not end there. Our universities are stuck at the first stage — they can only monitor and measure the most obvious stuff they do. They haven’t worked out how to demonstrate how well they do at their core tasks: educating students and producing excellent research. They know that those are the bottom line (the profit equivalent), but they cannot measure how close they get to it.
The lesson from business is that over time, if you can’t count the right thing, counting the wrong thing isn’t a substitute. It isn’t even just a distraction. It is the road to ruin.
As a result, our universities are trapped in an immature relationship with their market and their paymasters. My memory of that relationship is that it was characterised (on both sides) by petulance, truculence and pedantry. I don’t think things have changed much in the last seven years.
Where does that leave KM? We go through the same phases. In the early days we demonstrate the value of our work by showing people the simple numbers — this many documents created, stored or accessed; that many people involved in knowledge sharing. Later on, we can look at the quality of this stuff — how good are these documents, is there good feedback on knowledge sharing. Ultimately, though, we need to work out what our bottom line is: what are we here for and how good are we at delivering that value. In any given organisation that may take a while, but if we stick at simple measures we shouldn’t be surprised if our paymasters and clients see us as an irrelevance. If we can show the impact of our work on profitability, we should always aim to do so (and loudly). Nobody is going to blow our trumpet for us.