Evaluation

	Table of Contents
p. 6 of 7 BACK | NEXT Preface I. Introduction II. The Setting of Knowledge Audits III. Auditing Knowledge IV. Auditing the Lessons Architecture V. The Survey of Perceptions VI. Picking Investments in Knowledge Management

V. The Survey of Perceptions

A. Survey Definition

31. The survey that opened OED's first knowledge audit aimed to gain insight into how people within the four interfaces for lesson learning perceive the department's knowledge management activities. From the results, OED measured awareness of and identified gaps in the department's knowledge products and services. The Five Competencies Framework was used to assess the department's organizational performance. This framework identifies these areas of organizational competence for knowledge management as

• strategy development, wherein tools are used to help an organization achieve a particular goal in knowledge management through a long-term plan of action
• management techniques, which covers a range of practices from assessing the forces for and against desired organizational changes to assessing managerial approaches to mistakes, in order to do things right
• collaboration mechanisms, which pertains to facilitating effective practices in working with others
• knowledge learning and sharing, which means using techniques to learn from and improve future activities
• knowledge capture and storage, wherein routines are applied to ensure that essential knowledge is retained by an organization

For each interface, survey questionnaires covered only the areas of competence deemed most relevant.

32. The survey adopted a variant of the Likert scale⁴⁷ to evaluate the perceived level of organizational performance per competence area, wherein respondents specify the extent of their agreement with a statement. Five choices were given per question to gauge perception of OED's competences:

• never
• seldom the case
• sometimes the case
• often the case
• always the case

Two methods were used to determine overall perception of organizational performance in each area of competence: The first was based on the responses of the majority per question. The second established an objective measure by computing the weighted average score in order to account for the perception of the entire sample.

33. Frequent online consultations cause fatigue, and the design of the survey tried to make sure that the questionnaires would be well received by respondents. Before they were deployed, the questionnaires were tested by co-workers to ascertain that the questions would be interpreted in the manner expected. Questions were kept to a minimum, with the understandable exception of those targeted at OED staff, and kept directly relevant to respondents. None took more than 10 minutes to complete, and respondents were advised that this would be the case. They were also told why they should answer the survey. Reminders were sent. No identifiable answers were disclosed. Given the varying sizes, natures, locations, and degrees of interest of the populations targeted, OED considers that reasonable numbers of survey responses were received from three of the four interfaces, namely

• OED-OED
⁴⁸
• OED-ADB
⁴⁹
• OED-IEC
⁵⁰

Sadly, no responses were received from evaluation agencies in DMCs,⁵¹ with implications for the tools that can be applied to that interface in the future. Thus, the discussion that follows has to do with the results of the surveys in three interfaces only.

B. Survey Results

34. Box 6 gives a snapshot of the perception of the performance of OED in each area of competence by respondents from each interface. Respondents from OED thought that the department is doing well in the areas of strategy development, collaboration mechanisms, and knowledge capture and storage. But the department is "on the fence" in knowledge sharing and learning, and its competence with management techniques must get better. Respondents from other departments felt that OED does well only in strategy development. They were ambivalent with regard to collaboration mechanisms. They recommended that the department should deploy more efforts in the areas of knowledge sharing and learning, and knowledge capture and storage. Respondents from the IEC felt that the department is doing well in all three areas of competence regarding which their opinions were sought. Appendix 2 amplifies on the results of the survey of perceptions by interface and area of competence. Appendix 3 catalogues the survey questionnaires and the responses to these.

35. Figure 10 compares the scores in three areas of competence across the three interfaces. It illustrates the belief of ADB and OED staff that the department's weakest area of competence is in knowledge sharing and learning. This reflects unmet demand for knowledge products and services from these interfaces. The IEC, on the other hand, believes that the area of competence in which the department needs to improve most is in knowledge capture and storage. This may imply that people outside ADB may have difficulty accessing knowledge products and services, or most likely that the evaluation pages at adb.org are not yet sufficiently developed. In fact, these were revamped extensively from mid-2007 and updated daily thereafter. They now attract considerable traffic. But this is a comparatively recent phenomenon, and few in ADB or outside were aware of its new advantages at the time of the survey of perceptions. The lesson learned is that it takes time to attract faithful audiences and users.

36. The IEC gave OED the highest marks in all three areas of competence, while ADB staff gave the department the lowest. Respondents from OED and the IEC perceive that the department does best in terms of collaboration mechanisms. However, ADB staff think that it performs best in strategy development. This means that there is a high level of collaboration for knowledge management within the department and with the IEC, but there is a need for better collaboration efforts with ADB staff.

37. The perceptions of all respondents in the three areas are consistent. They all see OED's performance to be slightly above mid-level. This implies that respondents appreciate the achievements that have been attained in knowledge management so far, but believe that further improvements should be made. However, care must be taken in interpreting this result, as the Likert scale is subject to distortions such as central tendency bias.

C. Associated Initiatives

38. The survey of perceptions substantiated the basis of the knowledge management initiatives that OED introduced throughout 2007. It clarified the need for others. Among the new knowledge products and services developed that year, Learning Curves⁵² are handy, two-paged quick references designed to feed findings and recommendations from evaluation to a broader range of clients. Success Rates⁵³ present condensed information on successful ADB projects. The Evaluation Information System⁵⁴ is an online database of lessons, recommendations, and ADB Management responses. The department hosts the secretariat of the Evaluation Cooperation Group—it has also overhauled ECGnet,⁵⁵ the group's communication tool. Evaluation Alerts are targeted information bytes delivered to personal mailboxes. Methods and guidelines for using plain English, disseminating findings and recommendations, and conducting exit interviews have been prepared. The evaluation pages on adb.org⁵⁶ have been refurbished from top to bottom. They are updated daily and are now one of the most accessed first-level directories in adb.org. In 2007, OED formulated regional technical assistance for capacity development in monitoring and evaluation, expected also to suggest an ADB strategy for evaluation capacity development.⁵⁷ The department advertises its knowledge products and services on ADB Today⁵⁸ and adb.org (and through other channels and at several venues) with one-time, near-term, and continuous efforts. The survey of perceptions suggested other opportunities, now nearing completion: They include Evaluation Chats,⁵⁹ a communication tool that will underpin the formal establishment in 2008 of an evaluation community of practice focused on the conduct and dissemination of strategic evaluations, harmonization of performance indicators and evaluation methodologies, and development of capacity in evaluation and evaluative thinking. OED's knowledge management initiatives are framed deliberately to increase value-added from operations evaluation, and are managed with knowledge performance metrics. Client feedback is sought regularly through a variety of means.

39. Appendix 4 particularizes by interface and area of competence the knowledge management gaps elicited from and pointed out by respondents.⁶⁰ The responses constitute a comprehensive baseline assessment of long-lasting value. They provide ready and multiple entry points against which OED can take measures to foster lesson learning. Progress can be judged by means, among others, of the framework of organizational competence for knowledge management discussed earlier. Box 7 identifies the knowledge management tools that might be leveraged to fill the knowledge management gaps identified, and emphasizes with shading the areas of organizational competence found wanting. Appendix 5 organizes by category the common knowledge management tools that can be leveraged to develop organizational competence for knowledge management, and explains their main attributes. (The list presented is not exhaustive.)

40. Some of the tools prioritized are complex and call for significant planning and resources if they are to be utilized effectively. Extension to OED staff of opportunities for learning and development may also be warranted, to develop technical skills, competence, expertise, and knowledge in support of associated knowledge management initiatives. Among these, the formal establishment of an evaluation community of practice, continuous development of social technologies, and institution of staff profile pages show promise. The introduction of peer assists, challenge sessions, and after-action reviews and retrospects would be more demanding, but with associated knowledge management initiatives help close the knowledge gap most commonly cited by survey respondents, that in knowledge sharing and learning.

____________________
⁴⁷Details of the Likert scale are at http://en.wikipedia.org/wiki/likert_scale. A Likert scale is usually composed of an odd number of points measuring positive or negative responses to a statement.
⁴⁸The targeted number of respondents totaled 39, equal to 100% of OED's total staffing complement of professional staff, national officers, and administrative staff. Thirty one staff (80%) responded.
⁴⁹The targeted number of respondents totaled 1,196, equal to 100% of ADB's professional staff (minus professional staff in OED). One hundred and forty five staff (12%) responded.
⁵⁰The targeted number of respondents totaled 85. Ten staff (12%) responded.
⁵¹Twenty four respondents in evaluation agencies in 14 DMCs were targeted. The countries were Afghanistan, Azerbaijan, Bangladesh, People's Republic of China, India, Lao People's Democratic Republic, Mongolia, Nepal, Pakistan, Papua New Guinea, Philippines, Sri Lanka, Uzbekistan, and Viet Nam. The electronic addresses of respondents in evaluation agencies in other countries could not be procured in time for the survey of perceptions.
⁵²Available: http://www.adb.org/evaluation/reports.asp?s=1&type=15
⁵³Available: http://www.adb.org/evaluation/reports.asp?s=1&type=16
⁵⁴Available: http://evis.adb.org/
⁵⁵Available: http://www.ecgnet.org
⁵⁶Available: http://www.adb.org/evaluation/
⁵⁷Progressively more, evaluation ownership must move from ADB to DMCs.
⁵⁸ADB Today is a daily e-information resource for all ADB staff in headquarters, resident missions, and representative offices. It is the main internal communication vehicle to keep ADB staff abreast of events and activities of bankwide interest. It is produced and edited each working day by the Department of External Relations with inputs from other departments.
⁵⁹Available: http://www.adb.org/evaluation/chats.asp. Membership of Evaluation Chats is open to OED staff. Staff of other departments and other interested parties can be granted access upon request.
⁶⁰The greater the number, type, or size of a current or future knowledge gap, and the more precarious a knowledge base may be because of a dynamic or uncertain environment, the more forceful the knowledge management initiatives required.

Back IV. Auditing the Lessons Architecture	Next VI. Picking Investments in Knowledge Management