Independent Evaluation - Completed Capacity Development Projects

Country Project Name Lessons Identified
Regional Capacity Development for Monitoring and Evaluation
  • Baseline situational analysis of M&E status and needs was useful in helping design relevant training courses.
  • The training courses, which started with a Training of Trainers program, was relevant as it would help create a local pool of trainers capable of delivering country-based M&E downline training workshops to relevant staff of concerned agencies in their respective DMCs, which will in turn help sustain and internalize the M&E knowledge and capacity gained.
  • There should be no rush to incorporate outputs/components which could be adversely affected by various forms of restrictions in the countries unless these shortcomings could be addressed first.
  • The technical assistance (TA) project should work more closely with concerned resident missions, each of which should provide a focal point to support TA implementation.
  • A follow-on experience sharing workshop should be provide at the end of the TA to find out the extent of “actual” utilization/applications of the M&E knowledge gained from the training provided by TA and the extent of retention of the officials trained under the TA in doing M&E-related tasks.
Bangladesh Strengthening the Post-evaluation Capability of the Monitoring and Evaluation Division in Bangladesh
  • Constant interaction had been maintained at the senior most level in the executing agency, the Implementation Monitoring and Evaluation Division (IMED) of the Ministry of Planning, throughout the implementation of the TA. This immensely enhanced IMED's degree of ownership over the TA.
  • The new Post-evaluation Wing instituted in IMED is the result of senior level commitment clearly highlighting the priority attached to post-evaluation activities by the Government.
  • In the context of Bangladesh, bringing such institutional changes and having new staff positions approved are difficult feats to accomplish. These become possible only because the TA could obtain IMED's commitment and involvement throughout.
  • ADB needs to work very closely with the executing agency at every level, including the policy making level, so that their commitment is assured for the success of TA implementation as well as for long-term sustainability of TA gains.
China, People's Republic of Developing the Performance Evaluation Capability of the People's Bank of China
  • In the design of TAs, greater flexibility should be provided to accommodate policy changes and unforeseen circumstances during implementation.
  • The duration of international consultant input should take into account the country's procedural requirements and language constraints.
  • The time frame for implementing on-the-job training programs should be realistic to avoid delay in TA implementation.
China, People's Republic of Strengthening of Evaluation Capacity in the People's Republic of China
  • Institutional and leadership support from the Government are essential to sustain full and effective implementation of the TA. The central purpose of making performance evaluation an important tool for development, project management, and decision making should be properly understood by decision makers.
  • To implement a performance evaluation system and associated training in the People's Republic of China (PRC), appropriate consideration for the local language and understanding of the local culture and traditions are essential to facilitate faster dissemination of information and generate greater acceptance.
  • To establish a Technical Committee with a Chairman of equal rank with other technical committee members is not advisable in the PRC.
China, People's Republic of Project Performance Management Capacity Building in the People's Republic of China
  • Familiarity with local conditions and practices can contribute to effective delivery of seminars/workshops/training. Training should provide for interactive discussions and hands-on practice cases using local problems and experiences. Classroom interpretation is also important for training effectiveness. Care taken in translating materials and procuring the service of a skilled interpreter may not avoid inefficiency in classroom delivery and comprehension.
  • Benefits in learning should match resources spent, and be evaluated against the TA objectives for training and learning.
  • Changes in national processes for monitoring and evaluation require institutional and leadership support at all levels of participating agencies.
China, People's Republic of Strengthening Results-Based Monitoring and Evaluation in Key Projects in the People's Republic of China
  • Although evaluation is a mandated function of Key Projects Inspectorate Office (KPIO), it is not clear that, at the national level at least, KPIO will be leading its adoption in PRC.
  • Providing training to many provincial and city officials was a strength of the TA as provinces have considerable M&E autonomy and a number have already moved to adopt results-based M&E.
  • The use of a private sector service provider to develop the M&E software and to support it through a user-pays principle is a successful model that should be closely followed—it has the potential to provide a more sustainable solution (it is already showing signs of this) although it may not be without its problems as well
  • The pilot project or case study approach proved once again to be very successful in the PRC context—it facilitates “learning by doing”, provides a rich source of training material, and it may help overcome the in-built reluctance to make data available.
Nepal Strengthening the Post-Evaluation Capability of the National Planning Commission Secretariat in Nepal
  • The focus of the TA on training staff from the Central Monitoring and Evaluation Division of the Government's National Planning Commission and Secretariat, other line agencies, and consulting firms was appropriate. This experience highlights the importance of needs assessment as a basis for formulating programs to build institutional capacity.
  • Strengthening post-evaluation capability at the national level will be effective only to the extent that a similar capability is created at the important line agencies as well. Future TAs should take this into consideration.
Nepal Strengthening the Project Performance Management System in Nepal
  • Stakeholders full participation is essential at both the TA preparatory and implementation stages to cultivate a sense of ownership among the participants and to forestall lack of motivation among implementing staff to initiate TA activities.
  • A thorough assessment of the beneficiaries; capability/institutional environment prior to TA design would ensure the design's appropriateness and the demand for the TA outputs, and help avoid overly optimistic objectives. In the case of Nepal, for instance, attempting to computerize poorly performing manual monitoring systems may not be the best solution.
  • The lack of data encoders and of programming skills necessary to modify or correct program bugs impinges upon the sustainability of project performance management system in the long run.
Papua New Guinea Strengthening the Ex-Post Evaluation Capability of the Department of Finance and Planning in Papua New Guinea
  • The training experience provided timewise was inadequate and should require at least four weeks to be done properly.
  • There was a need for a more organized program with more attention to developing an overview of post-evaluation systems and developing specific skills such as interview techniques during the first week of the course.
  • The strategy for strengthening post-evaluation capabilities should be given more attention with more involvement from the senior personnel in the program.
  • Knowledge of the longer term development impact and sustainability of the programs and projects can help decision-makers develop policies and institutions necessary to achieve development objectives.
Philippines Strengthening the Post-Evaluation Capability of the National Economic Development Authority in the Philippines
  • Careful study of the post-evaluation environment during TA implementation can help identify factors that will affect sustainability.
  • A phased, medium-term system building program and work plan that takes these factors into account and commits the counterpart agency in charge of post-evaluation to sustain the effort would be a useful TA output to monitor further progress.
  • When counterpart staff participate in ADB missions, assignments should be planned, and at the end of the mission, the mission chief should prepare a performance assessment to guide the participant's growth.
Philippines Pilot Implementation of the Project Performance Management System in the Philippines
  • A partial/piecemeal donor approach to implementing changes in national monitoring and evaluation (M&E) process will not be effective unless it is done in a broad M&E agenda/framework and sustained by institutional and leadership support at all levels of participating agencies.
  • The exercise should have been less donor-driven and with stronger local initiative.
  • The National Economic Development Authority of the Government should take a lead role in providing direction, training, and on-the-job assistance and follow up to implement and replicate the project performance management system/results monitoring and evaluation approach to projects in other departments.
  • Manpower training must be institutionalized in order to develop and maintain the necessary depth of understanding and competence in M&E concepts.
  • A project should be designed using the log frame.
  • A project should not be approved for funding until the objectives, indicators, and targets have been clearly defined and a M&E system established.
  • Funding for M&E should be anticipated and included in the project budget.
  • It is difficult to undertake retrofitting of log frames for existing projects.
Philippines Strengthening Results Monitoring and Evaluation in the Philippines
  • Level of commitment. A TA of this magnitude and strategic importance should be negotiated with the highest executive authority of both implementing agencies in government and the operations department in ADB. TA objectives must match time, resources, staff support, and policy commitments required to attain these.
  • Two-stage TA input. To ensure quality at entry, a comparable organizational/policy reform TA should use a phased approach with initial assessment and planning followed by implementation.
  • Conceptual complexity of Logical Framework and indicator development. There is need for an authoritative “forum” or focal point to review cases in doubt as logical framework terminology is full of differences, points of view, and “schools of thought”.
  • Additional reports and monitoring and evaluation (M&E), especially on a regional basis, are not popular with project management. Results monitoring and evaluation must be sold on its value to management as basis for internal M&E, including financial monitoring, and for external reporting. In adopting results monitoring and evaluation, it would be better to introduce it in new projects.
  • Review of Logical Frameworks. Critical review of logical frameworks at start-up and during implementation is important. The same is true for regular monitoring of assumptions and risks at every stage of implementation. Institutional preconditions not met at project start-up and even to a certain extent during implementation that adversely affect compliance with targets should be identified.
  • Future assistance to the National Economic Development Authority on official development assistance programming and monitoring. This needs to start with an assessment of actual organization and functional set-up, more than the mandated functions.
Sri Lanka Strengthening of Post-Evaluation Capability of the Ministry of Policy Planning and Implementation in Sri Lanka
  • Some difficulties were faced trying to run the training course and preparing the manual at the same time. In the future, some consideration should be given to separating these two activities.
  • To develop the skills of the Sri Lankans further, it may be necessary to provide in the future some in-depth training in financial and economic analysis, sampling methodologies, and also in the art of report writing.
  • The program managers need to show real interest in management information systems. Otherwise work done by computer programmers may receive less attention, if not ignored.
  • It is important that staff from the Evaluation Coordinating Unit undertake on a regular basis some evaluation assignments to let the culture of evaluation grow within it and at the same time its staff functions are clearly divided between evaluation and coordination.
Sri Lanka Strengthening Project Performance Evaluation Capability of the Ministry of Plan Implementation, Ethnic Affairs and National Integration in Sri Lanka
  • The successful implementation of the TA depends largely on
    • conscious effort by ADB staff in maintaining close contacts and relationship with the implementing agencies and the periodic informal monitoring and acilitation role;
    • the political commitment accorded to it;
    • availability of strategic information; and
    • skilled manpower to carry out the tasks.
  • The continuous effort to create and sustain awareness among policy makers, planners, and senior public servants about the importance of post-evaluation as a lesson learning tool toward improving performance accountability enables the institutionalization of post-evaluation system in the public sector project cycle.
  • Advanced training of officers creates a multiplier effect especially where trained officers act as resource speakers in monitoring and evaluation programs.
Thailand Strengthening the Post-Evaluation Capability of the Office of the Auditor General in Thailand
  • A demand-driven TA designed to meet the felt needs of a recipient agency tends to promote its sense of ownership and full commitment, thus facilitating the implementation of the TA and the achievement of its objectives.
  • Flexibility reflected in the design of a TA and during its supervision encourages fuller and more efficient utilization of the funds provided in meeting the TA's objectives.
  • The use of the Advance Facility under ADB's Guidelines for Disbursement of Technical Assistance Grants proved to be useful and effective in expediting the disbursement of TA funds.
  • In developing member countries where English is not an official language, it is useful to conduct the training program in the local language to facilitate the transfer of knowledge.
  • Local reputable higher learning institutions can be good sources for recruitment of domestic consultants.
Thailand Institutional Strengthening of the Development Evaluation Division of the National Economic and Social Development Board in Thailand
  • Since the expertise and techniques transferred by the international consultant are quite innovative, regular monitoring and follow-up after the completion of the TA is essential to maintain the momentum of the project.
  • Both technical and analytical skills acquired under this TA should be maintained as institutional memories, and constantly updated.