Evaluation

	Table of Contents
p. 3 of 7 BACK | NEXT Preface I. Introduction II. The Setting of Knowledge Audits III. Auditing Knowledge IV. Auditing the Lessons Architecture V. The Survey of Perceptions VI. Picking Investments in Knowledge Management

II. The Setting of Knowledge Audits

A. Learning Organizations

4. A knowledge advantage is a sustainable advantage that provides increasing returns as it is used. However, building a knowledge position is a long-term enterprise that requires foresight and planning. In the knowledge-based economies that emerged in the mid- to late 1990s, the organizations with the best chance to succeed and thrive are learning organizations that generate, communicate, and leverage their intellectual assets. In The Fifth Discipline, Peter Senge labels them "… organizations where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free, and where people are continually learning to see the whole together."⁹ He catalogues their attributes as personal mastery, shared vision, mental models, team learning, and systems thinking (the fifth discipline that integrates the other four).¹⁰ Command of these lets them add generative learning to adaptive learning:¹¹ They seldom make the same mistake twice. Organizational learning promotes organizational health:¹² As a result, organizational performance is high.¹³ Referring further to Peter Senge, Figure 1 displays the core learning capabilities of organizations as a three-legged stool—a stool that would not stand if any of its three legs were missing. Figure 2 provides a matter-of-fact, multidisciplinary argument for why one might want to create a learning organization.

5. Other authors¹⁴ see learning organizations in different ways and the search for a single, all-encompassing definition of the learning organization is attractive but frustrating. In the final analysis, the most useful description is likely to be that which each organization develops for itself: That should be a well-grounded, easy-to-apply definition. Box 1 suggests an alternative way of looking at learning organizations, namely by considering what key characteristics might be. An important feature to bear in mind is that for associated benefits to arise a learning organization must be organized at five, sometimes overlapping levels:

• individual learning,¹⁵
• team learning
• cross-functional learning
• operational learning
• strategic learning

B. Organizational Learning

6. In the final analysis, other definitions of learning organizations share more with Peter Senge's than they disagree with but it should not be assumed that any type of organization can be a learning organization. In a time of great change, only those with the requisite attributes will excel. Every person has the capacity to learn, but the organizational structures and systems in which each functions are not automatically conducive to reflection and engagement. There may be psychological and social barriers to learning and change. Or, people may lack the knowledge management tools with which to make sense of the circumstances they face. In this sense, the learning organization is an ideal towards which organizations must evolve by creating the motive, means, and opportunities.¹⁶

7. The literature on learning organizations is oriented to action and geared to the use of strategies and tools to identify, promote, and evaluate the quality of learning processes. In contrast, that on organizational learning concentrates on the detached collection and analysis of the processes involved in individual and collective learning inside organizations. That is to say, organizational learning is the activity and the process by which organizations eventually reach the ideal of a learning organization. The dividing line between the two is the extent to which proponents emphasize organizational learning as a technical or a social process. Figure 3 exemplifies single-loop and double-loop learning, the technical view expressed by Chris Argyris and Donald Schön.¹⁷

8. Jean Lave and Etienne Wenger think that learning is inherently a social process that cannot be separated from the context in which it takes place. They coined the term "community of practice" in 1991 based on their work on learning theory in the late 1980s and early 1990s (even if the phenomenon to which it refers is age old). Learning is in the relationships between people. Social learning occurs when persons who share an interest collaborate over time to exchange ideas, find solutions, and build innovations based on ability, not hierarchical position. Jean Lave and Etienne Wenger argue that communities of practice are everywhere and that we are generally involved in several of them—at work, school, or home, and even in our civic and leisure activities: We all are core members of some groups and at the margins of others. Naturally, the characteristics of communities of practice vary. But they can be defined along three dimensions:

• what they are about (their domain)
• how they function (their community)
• what capabilities they produce (their practice).¹⁸

9. More recently, communities of practice have been associated with knowledge management as organizations recognize their potential contributions to human and social capital¹⁹ as well as to organizational performance. Communities of practice can drive strategy; spawn new ideas for products and services; transfer good practice²⁰ and decrease the learning curve of new employees; respond more rapidly to specific client needs—requested or anticipated—for certain information; solve problems quickly; minimize organizational knowledge loss (both tacit and explicit); reduce rework and prevent "reinvention of the wheel;" develop professional skills; and help engage and retain talented individuals. Even with the help of community-oriented technologies,²¹ however, harnessing them in support of organizational development is not easy. Communities of practice benefit from cultivation, but their organic, spontaneous, and informal nature makes them resistant to supervision and interference. Importantly, there is an intimate connection between knowledge and activity, and knowledge workers²² have a strong need to feel that their work contributes to the whole. To get communities of practice going, leaders should

• identify potential communities that will enhance the organization's core competencies
• provide supportive infrastructure
• use nontraditional methods to measure their value

In a learning organization, leaders are designers, stewards, and teachers.²³ Fundamentally, they should move from managing to enabling knowledge creation: Communities of practice are voluntary, and what will make them successful over time is their ability, within an enabling environment, to generate enough excitement, relevance, and value to attract, engage, and retain members. Depending on their maturity, communities of practice fall in one of two self-reproducing patterns of organizational performance, as illustrated in Figure 4.

C. Organizational Culture

10. The principal competitive advantage of successful organizations is their culture. Its study is a major constituent of organizational development—that is the process through which an organization develops the internal capacity to be the most effective it can be in its work and to sustain itself over the long term. Organizational culture may have been forged by the founder; it may emerge over time as the organization faces challenges and obstacles; or it may be created deliberately by management. It comprises the attitudes, experiences, beliefs, and values of the organization, acquired through social learning, that control the way individuals and groups in the organization interact with one another and with parties outside it. Standard typologies include communal, networked, mercenary, and fragmented cultures. These are determined by sundry factors²⁴ that find expression in organizational structure, making structure itself an important culture-bearing mechanism. The discourse on organizational culture can be esoteric: Figure 5 delineates ten components that, together, influence organizational culture. Identifying discernible elements of culture allows organizations to determine features that can be managed to help implement and sustain constructive organizational change. But just as none of the ten components in the figure shapes organizational culture on its own, none can individually support desired improvements.

11. Organizational culture varies more than any other corporate asset, including large and tangible information and communications technology infrastructure. It is said to be strong where employees respond to stimuli because of their alignment with it. Conversely, it is said to be weak where there is little alignment, and control is exercised with administrative orders. Regardless, if an organization is to succeed and thrive a knowledge culture must develop to help it deal with its external environment.²⁵ But organizational culture is hard to change in the best circumstances: Employees need time to get used to new ways of organizing. Defensive routines pollute the system, more often than not unwittingly, and undermine it. The dynamics of culture change must be considered an evolutionary process at individual, group, organizational, and interorganizational levels, to be facilitated by psychologically attentive leaders who do not underestimate the value of selection, socialization, and leadership. People cannot share knowledge if they do not speak a common language: And so there is a serious, oft-ignored need to root learning in human resource policies and strategies.²⁶

12. Observers recognize a correlation between the orientation of organizational culture and organizational learning.²⁷ Indeed, the inability to change organizational behavior is repeatedly cited as the biggest hindrance to knowledge management. For this reason, even if the need to take a hard look at an organization's culture extends the time required to prepare knowledge management initiatives, the benefits from doing so are likely to tell. Organizations that are more successful in implementing knowledge management initiatives embody both operations-oriented attributes and people-oriented attributes. Typically, a learning culture is an organizational environment that enables, encourages, values, rewards, and uses the learning of its members, both individually and collectively. But many cultural factors inhibit knowledge transfer. Box 2 lists the most common frictions and suggests ways to overcome them. Most importantly, when sharing knowledge, the method must always suit the culture as that affects how people think, feel, and act.

D. Learning for Change in ADB

13. The work of ADB is aimed at improving the welfare of the people in Asia and the Pacific, particularly the 1.9 billion who live on less than $2 a day. Despite many success stories, the region remains home to two thirds of the world's poor. However, the nature and pattern of Asia's growth process from the 1990s, analyzed in 2007 by the Eminent Persons Group that gave its views on the future of the region and made recommendations on the role of ADB,²⁸ is leading to fundamental changes in the demand for ADB's products and services. Put bluntly, it is in ADB's self-interest to invest in lesson learning to maintain its status as a relevant development agency in the region, and continue to influence global, regional, and national policy debates. It must manage for development results²⁹ with significantly increased effort, and embrace knowledge management. It will find it easier to do so if it moves from static measures of output to metrics that place a premium on adaptability and flexibility.

14. The experience that ADB has gained is its most important organizational asset. It must be safeguarded and used to inform operations in and services to developing member countries. The operations and services of ADB should reside in finance, knowledge, and coordination. They should move from "make-and-sell," at the simplest level, to "sense-and-respond" in ways that are increasingly satisfying to stakeholders. ADB's completeness of vision and ability to execute would be strengthened by a more systematic approach to knowledge management. In turn, this would enrich the quality and operational relevance of knowledge products and services; communicate know-how at the start of a development intervention—allowing it to move forward with less ongoing input; highlight problems earlier; increase the likelihood that others will volunteer beneficial information; allow tasks to be shared or delegated better; avoid duplication of work; speed up business processes; and create a positive atmosphere and stronger team spirit. Harnessed knowledge would bring the field and headquarters closer together and ensure that ADB's operations and services are grounded in and customized to local realities. It would also promote better partnerships within ADB and—through knowledge networks—outside ADB.

15. Evaluation has always been about learning—about how to be accountable, how to be transparent, and how to learn from experience. OED³⁰ applies itself to help ADB become a learning organization both in headquarters and in the field. Its work program³¹ is being reinterpreted in a more clearly defined results framework to

• conduct and disseminate strategic evaluations (in consultation with stakeholders)
• harmonize performance indicators and evaluation methodologies
• develop capacity in evaluation and evaluative thinking

With strategic evaluations, the department actively promotes sharing of lessons of experience to encourage higher organizational performance for development relevance, effectiveness, efficiency, and sustainability. In support, it also builds systems to ensure prompt follow-up of actions taken in response to evaluation findings and recommendations. Since January 2004, the department reports to the Board of Directors of ADB, not the President, through the Development Effectiveness Committee.³² Behavioral autonomy, avoidance of conflicts of interest, insulation from external influence, and organizational independence have advanced the department's mission. In addition, the department established a knowledge management unit in 2006 to leverage operational and developmental wisdom, both internally and externally, and increase learning. Independent Evaluation at the Asian Development Bank looks to a future in which knowledge management plays an increasingly important role in operations evaluation.³³ Figure 6 shows how the informal evaluation community of practice hosted by the department animates organizational learning.³⁴

16. The vast resources invested in development work make it imperative to learn from experience and avoid repeating mistakes, and the stakes for demonstrating the use of learning to improve organizational performance have never been greater.³⁵ Without a doubt, knowledge should be everyone's business. Learning Lessons in ADB articulates the results-based, medium-term strategic framework within which OED will do so and guides the department's knowledge management initiatives. It set the stage for regular knowledge audits, beginning in 2007, to provide fact-based assessments of where the department must cluster its efforts in the result-based framework of ADB.³⁶ Specifically, knowledge audits are to be used deliberately to

• identify knowledge needs for policy, strategy, and operational efforts
• draw up an inventory of existing knowledge products and services
• recognize gaps in knowledge products and services
• analyze knowledge flows within the organization and knowledge exchanges with outside agencies
• identify blockages to knowledge flows
• create knowledge maps
• suggest agenda for organizational change

The knowledge audits will also permit formulation of annual business plans to deliver outputs steadily against OED's interface with other departments, DMCs, and the international evaluation community.³⁷ Thus, the process of planning will draw on previous learning and apply that to new or changing situations to anticipate situations before they happen, rather than just reacting to what happens.

17. Acting on Recommendations and Learning from Lessons in 2007: Increasing Value Added from Operations Evaluation³⁸ proposed other steps including adhering to strategic principles, sharpening evaluation strategies, distinguishing recommendation typologies, refining recommendations, reporting evaluation findings, and tracking action on recommendations. Recognizing that knowledge that has not been absorbed has not really been transferred, it is hoped that synergies from the independence of the evaluation function and the application of knowledge management to it will boost readership and awareness of OED's knowledge products and services; raise their quality, influence, and use; and encourage the emergence of mechanisms to improve operations evaluation and its outreach. Figure 7 suggests that completeness of vision and ability to execute make all the difference between being a niche player and being a leader.

____________________
⁹Peter Senge. 1990. The Fifth Discipline: The Art and Practice of the Learning Organization. New York: Currency Double Day.
¹⁰According to Peter Senge, personal mastery hangs on clarifying personal vision, focusing energy, and seeing reality. Shared vision is built by transforming personal vision into common vision. Mental models are put together by unearthing internal pictures and understanding how they shape actions. Team learning grows from suspending judgments and fostering dialogue. Systems thinking fuses the first four disciplines to create a whole from distinct parts.
¹¹Generative learning concentrates on transformational change that changes the status quo. This type of learning uses feedback from past actions to interrogate the assumptions underlying current views. At heart, generative learning is about creating. Adaptive learning focuses on incremental change. That type of learning solves problems but ignores the question of why the problem arose in the first place. Adaptive learning is about coping.
¹²The notion of organizational ill-health is easily understood and needs no explanation. As long ago as 1962, Warren Bennis identified three dimensions of it: (i) adaptability, (ii) coherence of identity, and (iii) the ability to perceive the world correctly. The point here is that organizational learning can provide a necessary and valuable contribution to organizational health by advancing the shared values, clarity of purpose, institutionalized leadership, technical capability, open and honest channels of communications, and ability to deal constructively with conflict. All are qualities that employees expect from their work nowadays.
¹³Organizational performance comprises the actual outputs or results of an organization as measured against its intentions. It is commonly examined in terms of relevance, effectiveness, efficiency, and sustainability. The forces that drive these are organizational context, organizational knowledge, inter- and intra-organizational relationships, and the external environment.
¹⁴Mike Pedler, John Burgoyne, and Tom Boydell. 1996. The Learning Company. A Strategy for Sustainable Development. London: McGraw-Hill. Pedler et al. argue that a learning company is an organization that facilitates the learning of all its members and consciously transforms itself and its context.
¹⁵Individual learning is not covered in this booklet, even if it is the starting point of the learning organization and something that a learning organization should certainly encourage. Employees who are willing and able to learn new things are very important to an adapting organization. Without them, there will be no new products or services. There will be no growth. Specifically, learning organizations need skilled, enthusiastic, entrepreneurial, resultsoriented, and improvement-minded individuals. To describe how individuals learn, David Kolb has framed a wellknown experiential learning model: (i) doing, (ii) reflecting, (iii) connecting, and (iv) testing. Learning cycles can begin at any stage, depending on individual learning styles, but typically originate from doing. Reflective practitioners can choose to strengthen their ability at each stage to become all-round learners. Nevertheless, a learning organization is more than a collection of individuals who are learning—individual learning is a necessary but not a sufficient condition for organizational learning.
¹⁶A motive is a reason for doing something. Here, the motive is understanding learning and why it is important. The means are models, methods, competences, and support. Opportunities are in the space made available for learning, with implications for prioritizing time.
¹⁷Single-loop learning asks a one-dimensional question to elicit a one-dimensional answer. Double-loop learning turns the question back on the questioner. It encourages people to take personal responsibility for their action and inaction, and reveals information that can produce real change. Double-loop learning is potentially far reaching and can lead to what has been termed triple-loop learning—challenging the principles and assumptions of an organization, which requires an open if not robust exchange of views. Pessimists argue that bureaucracy and learning are mutually exclusive because bureaucracies all too often subjugate initiatives to their operating routines and mindsets, even though mindsets themselves must change for learning to occur. This argument, if it holds, underscores the importance of the evaluation function in bureaucracy.
¹⁸Etienne Wenger, Richard McDermott, and William Snyder. 2002. Cultivating Communities of Practice: A Guide to Managing Knowledge. Harvard Business School Press: Boston, Massachusetts.
¹⁹Human capital refers to the stock of productive skills and technical knowledge embodied in labor. Social capital refers to connections within and between social networks.
²⁰A good practice is defined as anything that has been tried and shown to work in some way—whether fully or in part but with at least some evidence of effectiveness—and that may have implications for practice at any level elsewhere. Three possible levels of good practice flow from this: (i) promising practices, (ii) demonstrated practices, and (iii) replicated practices.
²¹In a fast-changing market, numerous community-oriented technologies have emerged. They include (i) the desktop, with portal-like applications for managing participation in several groups; (ii) online project spaces for joint work; (iii) website communities; (iv) discussion groups; (v) synchronous meeting facilities, online auditoriums, conference rooms, and chat rooms; (vi) e-learning systems; (vii) expert profiles; and (viii) knowledge repositories. The advantages of one over another have to do with time and space, participation, membership, value creation, connections, and community development.
²²The knowledge worker, a term coined by Peter Drucker back in 1959, is anyone who works for a living at the tasks of developing or using knowledge. See also Olivier Serrat. 2001. Managing Knowledge Workers. ADB, Manila. Available: http://www.adb.org/documents/periodicals/ero/2001/managing_knowledge.asp.
²³Olivier Serrat. 2001. Marry in Haste, Repent at Leisure. ADB, Manila. Available: http://www.adb.org/documents/periodicals/ero/2001/marry_in_haste.asp
²⁴Factors determining organizational culture include innovation and risk taking, aggressiveness, outcome orientation, team and people orientation, attention to detail, stability, competitiveness, the diversity of members, and the age of the organization.
²⁵Shared vision and a sense of community, fueled by a positive outlook, are generally recognized as the principal drivers of a culture's receptivity to change. This bears out the earlier discussion of learning organizations and organizational learning.
²⁶The success of organizational learning lies also in its being incorporated into the recruitment and selection process, staff learning and development, and the performance management and rewards system, as well as being part of mobility and reassignment, including promotion.
²⁷In an ideal world, studies of the effects of organizational culture on organizational performance would develop theories and point to evidence that links specific aspects of culture to specific aspects of performance through specified intervening variables. Regrettably, only a very modest beginning has been made on this agenda.
²⁸ADB. 2007. Toward a New Asian Development Bank in a New Asia: Report of the Eminent Persons Group to the President of the Asian Development Bank. Manila. Available: http://www.adb.org/documents/reports/epg-report.pdf
²⁹ADB. 2007. Special Evaluation Study on Managing for Development Results in the Asian Development Bank: A Preliminary Assessment. Manila. Available: http://www.adb.org/evaluation/reports.asp?s=1&type=4
³⁰Details of OED's work are at: http://www.adb.org/evaluation/
³¹Available: http://www.adb.org/oed/workprograms.asp?p=opereva2
³²ADB. 2003. Enhancing the Independence and Effectiveness of the Operations Evaluation Department. Manila. Available: http://www.adb.org/documents/policies/enhancing-oed.pdf
³³ADB. 2007. Independent Evaluation at the Asian Development Bank. Manila. Available: http://www.adb.org/evaluation/documents/independent-evaluation/independent-evaluation-adb.asp
³⁴Treating knowledge as a resource opens up opportunities for learning across each time segment of the knowledge management cycle based on reviewing, conceptualizing, reflecting, and acting. Most often, the actions will be a combination of generic operations that involve developing, distributing, combining, and consolidating knowledge. In OED, the experience of practice is captured in evaluation studies. It is synthesized in knowledge products and services, often enriched by lessons from other sources through innovation. These are channeled as publications, presentations, press releases, and media resources, often via the internet, so that experience improves practice. The informal evaluation community of practice in the department animates these throughout the learning cycle of practice; experience; synthesis and innovation; dissemination; and uptake with one-time, near-term, and continuous efforts.
³⁵Capturing and storing lessons to inform practice has appeal. Operations are costly, and lessons that are learned can raise quality and save time and money. To reap such benefits, many organizations have invested in softwarebased knowledge management systems. The technologies at hand offer remarkable capability but seldom work as well as anticipated. Commonly, the overriding difficulty is that the lessons that populate databases are mundane and are of questionable value to users. They are not connected to the social processes that underpin and condition knowledge sharing and learning, such as development of inter- and intra-organizational relationships that match the supply side of knowledge to demand.
³⁶To be exact, OED did conduct a survey of users of evaluation as long ago as 1997, to which 107 ADB staff responded. Interviews with ADB's Management also took place. The paper-based questionnaire used was modest, concentrating on knowledge of evaluation work and the quality, relevance, and usefulness of evaluation studies. The interviews focused on feedback mechanisms and evaluation capacity development. Respondent ratings on the quality, relevance, and usefulness of evaluation reports, which encapsulate perceptions best, were downbeat, especially regarding timeliness, ease of access, and feedback mechanisms. However, the absence at that time of a strategic framework for knowledge management provided little framework for actions in response.
³⁷The IEC referred to comprises the co-members of the Evaluation Cooperation Group, namely the African Development Bank, European Bank for Reconstruction and Development, European Investment Bank, Inter- American Development Bank, International Monetary Fund, and the World Bank Group. The community includes two observers—the United Nations Evaluation Group and the Evaluation Network of the Development Assistance Committee of the Organisation for Economic Co-operation and Development.
³⁸ADB. 2007. Acting on Recommendations and Learning from Lessons in 2007: Increasing Value Added from Operations Evaluation. Manila. Available: http://www.adb.org/evaluation/arealist.asp

Back I. Introduction	Next III. Auditing Knowledge