All Life Is Problem Solving

In Parts Two and Nine of this series, I talked about the “strategy exception error,” and the need to overcome it in the quest for quality knowledge processing across all areas in the Federal Government including the strategy function itself. Another important aspect of reaching this goal, as well as ensuring the quality of knowledge processing itself, is the Knowledge Accountability Office’s (KAO) function of evaluating the impact of KM and knowledge processing activity across the decentralized, partially self-organizing clusters of KM activity in the Federal Government. In this evaluation function it would serve as the mechanism of Government-wide KM performance accountability to the legislative fiduciary (the Congress). In this blog, I’ll discuss some of the difficulties the KAO will have to cope with in evaluating KM impact.

Conceptually the idea of KM impact is clear enough. Here’s a visual.

In an earlier blog, I used this visual to ask why there was very little modeling of KM impact going on, and also asked how we could expect to make the case that KM works without modeling impact. Well, the obvious answer to the first of these questions is that there are difficulties in the way of KM impact analysis and measurement and this raises the possibility that the answer to the second question is that we can’t expect to “make the case” unless we overcome these difficulties. Here are some of the most important ones.

— The indirectness of the relationship between KM activity and business outcomes (I highlighted this relationship in a previous blog about the three-tier model. The value chain is: KM activity -> KM Outcomes -> knowledge processing activity -> knowledge processing outcomes -> business processing activity -> business outcomes. To trace the impact of KM activity on business outcomes, we need to trace it through the value chain while recognizing that the arrows represent propensity rather than causal relationships, and also that the value chain exists in the broader context of a value network incorporating many other intermediate influences reflecting the complexity of an organization and impinging on the value chain at every point along the path to business outcomes);

— The relative absence of metrics dealing with the second tier of the three tier model (We can measure what we do in Knowledge Management and we can tell what the direct KM outcome is, but we lack a structure of metrics to tell us what changes have occurred in various knowledge sub-processes such as individual and group learning, knowledge claim formulation, knowledge claim evaluation, knowledge sharing, etc. The same applies to the area of knowledge outcomes in which we have no ready metrics to handle changes in the quality of knowledge, its degree of relatedness within the distributed organizational knowledge base, its legitimacy, the legitimacy of Knowledge Management, and other aspects of knowledge outcomes);

— The complexity and indirectness of the relationship between ecological characteristics directly impacted by KM and enhancement of knowledge processes (KM outcomes include changing the ecological aspects of knowledge processing. That’s what one does, for example, when one creates a Community of Practice, and provides it with social computing support. That’s also what one does when introduces an Enterprise Information Portal to provide content management and collaboration support. It’s not easy however, to specify the relationships of such ecological changes to enhancements in various aspects of knowledge processing. There are many different kinds of communities and their relationships to enhancing knowledge processing will differ and are highly contextual. There are many different kinds of portals and their relationships to enhancing knowledge processing are also highly contextual. The general idea is that the relationship between any ecological change resulting from KM, and the knowledge processing it was intended to enhance is always highly contextual and dependent on the place of the ecological change in the history and evolution of the complex system involved);

— The problem of evaluating rather than simply describing impact (Apart from describing the impact of KM in terms of the extent of movement toward goals or desirable states, there’s also the problem of evaluating the significance of such an impact. This is important because the degree of impact is a different thing from the importance of that impact to us, and to prioritize among alternative future KM efforts we need not only to be clear about their expected impact, but also about the importance of that impact to us)

The importance of these four difficulties is that the KAO will need to develop methodology for handling them, and will also have to evaluate funding proposals for supporting KM programs and projects from the viewpoint of how well the proposals plan to cope with the difficulties. Also, there are, I think, three primary approaches to KM and they differ in the extent to which the four difficulties are important in impact evaluation. In my next two blogs in this series I’ll discuss the three approaches and also the implications of the combination of the evaluation difficulties and the different approaches to KM in shaping the organization of the KAO’s evaluation function.

To Be Continued