All Life Is Problem Solving

Joe Firestone’s Blog on Knowledge and Knowledge Management

All Life Is Problem Solving header image 2

Why Don’t We Write Much About KM Metrics?

August 31st, 2008 · 8 Comments

burningofparliament1834JWMTurner

From the beginning of KM there’s been remarkably little focus on metrics and measurement. In particular, there’s been remarkably little focus on metrics of KM impact. This lack of focus is in line with a certain anti-scientific orientation that has appeared in KM associated with the philosophies of post-modernism and social constructivism. It is also in line with a rejection in the field of the idea that KM projects need to be justified by pointing to concrete results, and the adoption of a position which seems almost to say that KM is like the furniture in an organization. It’s hard to measure its impact; but, without it, an organization is hard-pressed to survive. Further, the neglect of metrics and measurement is also in line with the difficulty and unpleasantness of developing frameworks and architectures for measurement in an applied social systems field like Knowledge Management. The key terms of KM are abstractions. Change in them is not directly or easily observable. To relate changes in our experience to changes in these abstractions, we often need complex measurement models and there aren’t many KM practitioners who have the background and training to develop such models.

 

There are still other difficulties to note. Dave Snowden talks about the extreme reactivity of many indicators and the ease with which they can be “gamed.” He is right. Simple indicators can be gamed and this also argues for more complex measurement models that are non-reactive and impossible to game because gaming them would require too high a price for the “gamers” to pay in their everyday organizational interaction. In addition, the most important thing to measure in KM is the impact of KM interventions. This, however, introduces another difficulty, because measuring impact requires measuring change over time and also doing some modeling of influence relations. Nor is this all. To measure impact we also need to be able to project a counter-factual: the expected result of a scenario in which we don’t intervene, and compare that with the measured state of the target we’ve been trying to influence after we intervene. All this requires a methodological and technical sophistication, which we have rarely seen employed in KM to date. Nevertheless, it is all necessary to measure impact.

 

Finally, yet another difficulty in measurement is caused by the persistent tendency in KM to confound KM activities and outcomes with knowledge processing and its outcomes. It’s easy to understand this problem by looking at the three-tier model below.

tnkm ref

When we do see metrics in KM projects, they often relate KM activities and direct outcomes to effects on business processes and their outcomes. That is, KM activities are related to business metrics, but such studies don’t develop any measurement models or metrics relating KM to the middle, knowledge processing and outcomes, tier. The problem with that is a failure to trace impact through the middle tier, making it harder to show that any post- KM intervention outcomes are actually due to KM. Now sometimes this omission is not important in justifying one’s project. For example, in the Partner’s HealthCare case it’s very hard to deny that reductions in the negative impact due to errors in ordering prescriptions was due to the KM intervention re-structuring the ordering process, and eliciting a growth in the problem recognition and problem solving surrounding it. In other cases, however, particularly those involving the much more common ecological approach to KM, the relationships among KM, knowledge processing and outcomes, and business processing and outcomes. is much more complex and is not disentangled in the cases. So, it is much harder for them to establish KM impact, either positive or negative in character.

 

In spite of all these difficulties besetting the task of developing measurement models and metrics in KM, I don’t think the field will progress very much unless this development takes place. If we can’t show impact we won’t be able to claim impact. And if we can’t claim impact, no one will ever take KM seriously. So, I think we had better begin to spend a lot more time on both doing and writing a lot more about measurement and metrics, and we ought to do that immediately, so that when the time comes to evaluate the newly minted Web 2.0-based interventions, we can say just how successful they are without yet another generation of arm-waving.

Tags: KM 2.0 · KM Methodology · KM Techniques · Knowledge Making · Knowledge Management

8 responses so far ↓

  • 1 frysystems // Sep 3, 2008 at 10:42 pm

    The key for me lies in Knowledge Processing where it does become possible to measure coverage of the underlying explicit knowledge base, and also rightness and timeliness (with a bit of effort as you cover elsewhere).
    It is all about Web 3 (I am in line with the Fuji-Xerox concepts here). Web 2 does little more than implement a framework of tools for possible knowledge exchange. Metrics on that are useful in looking at usage; but without some analysis of content; you are not measuring Knowledge Management.
    I am almost inclined to paraphrase Mark Twain on weather:
    “Everybody talks about it; but nobody does anything about it!”

  • 2 Joe // Sep 4, 2008 at 1:43 am

    Welcome, Ian, and thanks for your comment.

    I agree with what you say and I also think that without analyzing content you can’t really measure changes in knowledge processing and KM impact.

    In my research, I’ve been developing a conceptual architecture for measurement. It’s a very detailed conceptual framework covering organizational adaptive functioning, including the three-tier model, decision processing, and a few other primary categories dealing with special kinds of decision making. The framework is currently in the form of an Expert Choice (see http://www.expertchoice.com) software template I call the Open Enterprise Template. I introduce people to it in KMCI’s CKIM Workshop and provide the Template as one of the Workshop takeaways along with the right to use or customize any part of it.

    In short, I’m trying to do something about the measurement problem.

    Best,

    Joe

  • 3 frysystems // Sep 4, 2008 at 7:18 am

    I have been tracking your thoughts via Richard Vines, who I met at ACTKM last year. I “use” Richard to help interpret some KM stuff, and I find he is an excellent thinker.
    My orientation is quite IT related, but at the advanced end, and I came to KM as a “dumbing down” of my work in AI (if that is not too cruel).

    For over a year I have struggled with your “ecological approach” but have finally found a place for it in my own thinking, which is entirely driven by the commercial world and its realities.
    On his recent visit to US, I sent Richard with the mission to work out the following hypothesis:
    “Is it DANGEROUS to only impliment the ECM Model and not allow for a co-commitant ecological approach as a metric/parallel/ sense making approach?”
    Consider my major client who makes money from conducting training sessions. How could you sensibly and commercially integrate the ecological approach? And what are the consequences if you don’t??
    I reckon, 1 year on in my work with Richard, I have an inkling of a glimmer of a possible resolution.
    Please keep up the “good thinking”
    Please try and make it to ACTKM – it is the most marvellous experience, populated by many of the people appearing on your blog references.
    Last year I spent 2 days with David Gurteen, Luke Naismith, Shuan Callahan, Richard Vines, Arthur Shelley. Matt Moore etc

  • 4 Joe // Sep 5, 2008 at 12:57 am

    Ian,

    Thanks for your last comment. If at all possible I would certainly like to come to the ACTKM meeting. I have more fun with my ACTKM correspondents than any of my other groups and would certainly like to have a face-to-face with everyone. As you know Richard came here in June and we had a great time, including hatching many future plans and also developing the common view necessary to finish our XML interoperability paper, which we are just putting the finishing touches on now.

    On the ecological approach, I’ve given considerable thought to how to implement that, though of course the devil is in the details of each project. Also, I, too have had some background in AI approaches and used to write some AI-related papers during my first few years in KM. They’re available at http://www.dkms.com. Of course, my book, Enterprise Information Portals and Knowledge Management, KMCI Press, Butterworth-Heinemann, 2003, envisioned a knowledge portal concept integrating a distributed cloud of Intelligent Agents.

  • 5 KM 2.0 and Knowledge Management: Part Ten // Sep 14, 2008 at 12:51 am

    […] formulation, knowledge production, and knowledge integration. In addition, I’ve explained the Three-tier Model as specifying the context of […]

  • 6 National Governmental Knowledge Management: KM, Adaptation, and Complexity: Part Nine, Funding KM Programs and Projects Across the National Government // Mar 3, 2009 at 11:16 pm

    […] Pre-specify expectations and measures of performance (success and failure) relative to the KM or knowledge life cycle targets of the proposed program or […]

  • 7 Some Quick Thoughts on Reasons for KM Failure // Apr 25, 2009 at 12:04 am

    […] The relative absence of knowledge processing metrics in KM. We can track our own actions and expenditures in KM and keep metrics on those things, and we also […]

  • 8 Benchmarking or Measurement Validity? // Jun 14, 2009 at 9:00 pm

    […] and conceptual frameworks might well be correct, regardless of the quality of their benchmarking, if impact measurement best practices within the KM industry are themselves inadequate. Based on my own very long experience in the social sciences, and exposure to many different kinds […]