Evaluation

	Table of Contents
p. 2 of 7 BACK | NEXT Preface I. Introduction II. The Setting of Knowledge Audits III. Auditing Knowledge IV. Auditing the Lessons Architecture V. The Survey of Perceptions VI. Picking Investments in Knowledge Management

I. Introduction

1. Organizations¹ — defined as social arrangements by means of which two or more persons pursue collective goals—are among the most significant structures through which society acts out its economic and social life. Among these, development agencies spend billions of dollars every year on thousands of interventions that aim to reduce poverty through innovation and societal change. But what development results do they deliver? Not knowing whether or not a development intervention actually made a difference, or not acting on the lessons² of that experience, does not just waste money: It denies the poor critical support to improve their lives.

2. In development agencies, much data, information, and knowledge³ are needed by policy makers to decide what resources to assign to what development interventions; by personnel tasked with making decisions on impact, outcome, outputs, costs, financing, implementation, and other key design features; and by agents faced day after day with the challenges of implementing the interventions. The knowledge base needed for good policymaking, design, and implementation originates from many sources. Typically, development interventions include monitoring⁴ and evaluation⁵ of inputs and outputs along the results chain, building knowledge about processes and institutions, and providing evidence of accountability. The knowledge generated from these is not a luxury.

3. The reflective conversation element in evaluation, in particular, is a foundation block of organizational learning,⁶ and concern for the effectiveness of the evaluation function and its feedback mechanisms is pertinent: It is essential to transfer increased amounts of relevant and high-quality knowledge into the hands of policy makers, designers, and implementers. This booklet examines the setting of knowledge audits and underscores the role that they can play to

• better position evaluation as a resource linked to policy, strategy, and operational efforts
• share accountability, acknowledge risk, and reward adaptive learning
• create more space and structure for learning
• focus with more intent on different levels of learning

Because knowledge audits are a new and immature practice,⁷ the booklet also offers pointers on their conduct and explains the methodology developed by the Operations Evaluation Department in 2007 to strengthen the lessons architecture of the Asian Development Bank with regular inputs. Lastly, it presents the results of an online, questionnaire-based survey of perceptions conducted in 2007—as a first application of one tool of knowledge auditing—and highlights the main conclusions from that.⁸ Success hangs on knowing what works: When we seize opportunities to learn, benefits can be large and widespread.

____________________
¹The word is from Greek organon, literally, tool, instrument.
²In development work, a lesson is knowledge derived from the implementation and evaluation of an intervention that can be used to identify strengths and weaknesses. This knowledge is likely to be helpful in modifying and improving functions. Lessons are of two types: operational and developmental. Irrespective, they usually relate to planning, positioning, implementation, and management.
³Data are discrete and objective facts, measurements, or observations that can be analyzed to generate information. Information is data that have been categorized, analyzed, summarized, and placed in context in a form that has structure and meaning. Knowledge is a combination of data and information, to which is added expert opinion, skills, and experience.
⁴Monitoring is the continuous collection of data and information on specified indicators to assess the implementation of an intervention in relation to activity schedules and expenditure of allocated funds, and progress and achievements in relation to its intended outcome.
⁵Evaluation is the periodic assessment of the design implementation, outcome, and impact of an intervention. It should assess the relevance and achievement of the intended outcome; implementation performance in terms of effectiveness and efficiency; and the nature, distribution, and sustainability of impact.
⁶Learning is a characteristic of an adaptive organization. An adaptive organization is able to sense changes in signals from its environment, both internal and external, and adapt accordingly.
⁷Very little literature on knowledge audits goes beyond superficial discussion of what they might involve. Large companies have crafted proprietary knowledge audit methodologies: They are not publicly available but can be acquired for a fee. But no universal template can exist for the same reasons that few knowledge management frameworks have reached prominence and a wide audience: Knowledge management requires complex, interrelated changes in leadership, organization, technology, and learning, all of them specific to the enterprise. In spite of everything, it is nonetheless possible to extract insights from existing literature and use common sense to customize an up-to-standard knowledge audit methodology, as this booklet reveals.
⁸Evaluation at ADB has two major dimensions: (i) self-evaluation by the units responsible for particular development interventions, and (ii) independent evaluation by OED. Both are governed by principles of usefulness, credibility, and transparency. The second, as the name indicates, is made stronger by the additional principle of independence, and the associated attributes of impartiality and freedom from bias. The survey of perceptions pertained to independent evaluation.

Back Preface	Next II. The Setting of Knowledge Audits