Evaluation

	Table of Contents
p. 1 of 7 BACK | NEXT Preface I. Introduction II. The Setting of Knowledge Audits III. Auditing Knowledge IV. Auditing the Lessons Architecture V. The Survey of Perceptions VI. Picking Investments in Knowledge Management

Preface

The annual sovereign and nonsovereign lending volume of the Asian Development Bank (ADB) is typically about $6 billion, with technical assistance usually totaling about $180 million a year. This does not include cofinancing and the counterpart funds that the governments of developing member countries associate with ADB's operations, which amounted to about $8.2 billion and $4.9 billion, respectively, in 2006 for loans totaling a higher-than-average $7.5 billion. Plausibly, the return on investment in lesson learning for operational and developmental impact is likely to be high, and maximizing it is a legitimate concern. There are signs of improvement in evaluation-based learning in ADB. This holds for policies, strategies, country programs, and projects, including their design, implementation, results, and associated business processes. One notable area is greater attention to country, sector, and thematic areas. But the use of evaluation for learning may be less important than that of other inputs, such as self-evaluation and training, and evaluation results may only marginally support policy, strategy, and operational changes.

Since ADB created a post-evaluation office in 1978, the function of operations evaluation has moved from recording history to shaping decision making throughout the project cycle. Recent advances, to name a few, were sparked by the Board of Directors' decision in 2003 to enhance the independence and effectiveness of the Operations Evaluation Department, by ADB Management's endorsement of the Paris Declaration on Aid Effectiveness of 2005, and by the department's resolution in 2006 to leverage knowledge management tools in support of lesson learning.

But even improved and well-used evaluation systems do not generate learning automatically. Evaluations generate learning through stakeholder involvement and via communication in a dynamic organizational context. If ADB wants to maximize its learning to ably serve a fast-changing region, it should pursue strategies that let the two modes supplement each other. Learning must be related to what people know and what they need to know to perform their jobs well—and hopefully better. Crucially, material must be tailored to the audiences targeted, and produced in a form that they can use. But managers must also actively support use of evaluations because applying knowledge has costs. Besides time and money, the latter can include the need to unlearn previous practices, and disruption to established relationships. Only if these two conditions hold will the costly gap between evaluation and planning be bridged.

In ADB, and most likely in other development agencies, the construction of knowledge is rarely examined, and there has been a dire absence of work to find out what helps or hinders the transfer of knowledge through evaluation studies. Auditing the Lessons Architecture brings to light the contribution that knowledge audits can make to organizational learning and organizational health, notwithstanding the psychological and social barriers that organizational culture can throw up. With deserved emphasis on the organizational context for learning, the booklet shows with a real-life example how knowledge audits open opportunities in strategy development, management techniques, collaboration mechanisms, knowledge sharing and learning, and knowledge capture and storage. The knowledge audit methodology described in the booklet can support systematic identification and analysis of knowledge needs, products and services, flows, uses, users, and gaps from the perspective of learning lessons, necessary to tie in with the department's audiences. Responses to the online, questionnaire-based survey of perceptions conducted as a first exercise in 2007 provide ready and multiple entry points against which the department can take measures to that intent, as well as a comprehensive baseline assessment. They evoke a common sentiment that the department can do better in knowledge sharing and learning, and probably also in knowledge capture and storage. Staff of the department believed also that gaps exist in the management techniques that the department applies for knowledge management. In these three priority areas, the Five Competencies Framework adopted in Learning Lessons in ADB¹ provides a springboard for action. The reader should always bear in mind that the survey of perceptions carried out is only one of the miscellany of investigations that can make up a knowledge audit. The Operations Evaluation Department chose to kick off knowledge auditing with a survey of perceptions because this tool quickly helps identify gaps in knowledge management. The customer is always right.

Learning Lessons in ADB, the companion volume to this booklet, guides the knowledge management initiatives of the department. It was published to share, mainly with ADB staff, a formal knowledge product that has implications for development thinking. This booklet is likewise offered as a resource and reference guide to ADB staff. It may also give the department direction and help it make the right decisions. The emphasis that it places on meaning, management, and measurement can help ADB become a better learning organization. Ideally, knowledge audits should take place regularly, and the booklet can support the development of in-house expertise. (Carrying out a knowledge audit is certainly within the capacity of able employees.) Much as Learning Lessons in ADB, the booklet will also appeal to the international evaluation community and people having general interest in knowledge management and, more specifically, in knowledge audits and their organizational setting. Auditing the Lessons Architecture was written by Olivier Serrat, Senior Evaluation Specialist (Knowledge Management), under my overall supervision. Adele Casorla analyzed the results of the survey of perceptions.

R. Keith Leonard
Officer-in-Charge
Operations Evaluation Department
Asian Development Bank

____________________
¹ADB. 2007. Learning Lessons in ADB. Manila. Available: www.adb.org/documents/reports/learning-lessons-adb/strategic-framework-2007-2009.asp

Back Auditing the Lessons Architecture	Next I. Introduction