Report of the Task Force on Improving Project Quality

Table of Contents

p. 21 of 30 BACK | NEXT

Executive Summary

I. The Setting

II. Key Variables Affecting Project Quality

III. Assessment of Current Bank Practices and Areas for Improvement

A. Project Processing (Project-at-Entry)

B. Project Implementation

C. Feedback Mechanisms

IV. Findings, Recommendations and Action Plan

V. Implications and Monitoring Arrangements

Report of the Task Force on Improving Project Quality : III. Assessment of Current Bank Practices and Areas for Improvement

C. Feedback Mechanisms

84. Feedback is not a specified stage in the project cycle. However, it is expected that, at every major stage of the project cycle, the process of feedback on lessons learned from previous projects concerning implementation should take place systematically. Given the importance of this process, particularly from the point of view of enhancing project quality, the Task Force undertook a review of key feedback processes within the project cycle. The findings of the Task Force are presented in this section.

1. Country/Sector Monitoring

85. In the ongoing ESW undertaken by the Bank, many opportunities arise to obtain feedback that can potentially enhance the quality of future Bank operations. Such information includes key country developments, the Bank's experiences in policy dialogue and project implementation, and related lessons learned. In the preparation of the GOSS, the Bank has the opportunity to address the issue of whether the previous operational strategy has been a success and has been adequately implemented through the COPP. Specific project design and implementation experience can be usefully fed back into the formulation of the to-be-revised COSS and subsequent COPPs. The Task Force considers that these opportunities for feedback are not used adequately within the Bank. The reasons range from the limited resources allocated to the preparation of the COSS and related ESW, to the lack of coordination and communication between the Programs Departments and other concerned departments and offices such as Projects Departments, CPSO, AGSD, PEO and EDRC.

86. The Task Force recommends as follows:

All COSS and COPPs should explicitly incorporate a thorough review of the problems encountered in implementing the previous Bank strategy and projects in the country concerned.
COSS and COPPs should draw such feedback from key in-country sources as well as from PEO and other concerned departments/offices within the Bank.
PEO should undertake more frequent country assessments from the point of view of the Bank's post-evaluation experience.

2. Benefit Monitoring and Evaluation (BME)

87. An effort has to be made to establish a closer link between project implementation and BME by progressively integrating the data compilation and monitoring aspects of BME into the Bank's project management and administration information system. With the increased emphasis on participation, adoption of the process approach and fostering ownership of the operations by the beneficiaries, EAs and the borrowing countries as a means to improve project quality, there is a need to rethink the purpose and design of BME. The main purpose of BME remains the monitoring and evaluation of the performance of an investment operation (project, program, or sector loan) throughout its entire economic life. BME could be redesigned with the identification of a limited number of simple, easily understandable and monitorable quantitative and qualitative indicators, information on which can be readily compiled by the project implementing/executing agencies or other local agencies (see Box 8). The indicators should ideally encompass monitoring of both processes (e.g., degree of participation of beneficiaries, including their contributions and the effectiveness of social organization and cohesion) as well as outcomes (e.g., benefits accruing in the form of production, employment, income, improved health and skill formation). The Task Force recognizes that BME during implementation will be particularly useful in types of projects in which benefits start accruing along with implementation. However, the indicators should also include significant project milestones during implementation and operation so that BME may be expanded into a more general monitoring tool.

Box 8 � DMC Views on the Evaluation of Bank Projects

DMCs confirmed their belief in the need for an objective and comprehensive monitoring and evaluation system. Some of the more significant views and suggestions expressed in this connection were:

The Bank should give greater emphasis to strengthening in-country monitoring and evaluation capacities; thus, requests were made for related institution development and training assistance.
The Bank should develop simple, relevant and easily manageable monitoring and evaluation systems, suited to the institutional capacities of each DMC; realistic parameters and mechanisms for monitoring should be established at appraisal in consultation with the implementing agency.
Lessons from Benefit Monitoring and Evaluation should be pursued systematically by the Bank and the implementing agency to ensure improvements in implementation and project sustainability.
The Bank's Project Completion Reports and Project Performance Audit Reports should be comprehensively discussed with all of the implementing agencies concerned before they are finalized.

88. To improve the effectiveness of the BME system, the Task Force recommends as follows:

BME should be viewed as an important tool for monitoring and evaluating the performance of an investment operation throughout its economic life. Significant project milestones during implementation and operation should be identified in the exercise. The adoption of a logical framework approach in project preparation, processing and implementation would assist in the identification of key indicators and critical milestones.
The indicators should ideally encompass monitoring of both processes (e.g., degree of participation of beneficiaries) and outcomes (e.g., benefits accruing in the form of production, employment and income).
To ensure that the chosen indicators are carefully monitored and that timely corrective action is taken, a list of these indicators should be included in the RRP, with appropriate provisions incorporated in the loan agreement.
The Project Administration Committee (PAC) notes should be modified to incorporate information on these indicators on a regular basis. CPSG could act as the Bankwide coordinator for the modified BME.

3. Project Completion Reports (PCRs)

89. A PCR is normally prepared 6-12 months after substantial completion of the physical components of a project. No record of the performance rating of the PCR is kept by the implementing department (unlike with PPARs), and while PCR findings may well be used for inputs into future projects of a similar nature, their present use in this respect is not fully satisfactory .Considering the time and effort exerted in preparing the PCR, as well as the cost, the Task Force strongly feels that greater utilization of PCR findings should be made in Bank operations.

90. Considering that PPARs are prepared about 18 months to two years after completion of a project, and that they cover less than 50 per cent of all completed projects, the PCRs can be improved to provide a valid alternative to the Bank for assessing project performance, especially in terms of documenting lessons learned. This would be particularly so if the experience with the modified BME system discussed earlier was adequately analyzed in the PCRs � in which case, the PCRs would be akin to comprehensive "implementation completion reports." In addition, the revised PCRs could provide an assessment of the project's transition to operations, identifying any necessary remedial action. The BME indicators should also be re-examined for use in monitoring operations and assessing the development impact. This would make the PCR a more forward-looking document. Despite these improvements in the PCR, an independent post-evaluation report from PEG is still needed.

91. To improve the PCRs, the following are recommended:

The present format of the PCR should be reviewed by both PEO and the Projects Departments so as to derive a format that would form the basis of a future PPAR, thus avoiding the current duplication. The revised format, containing as up-to-date information as available at the time of preparation on financial, economic and social benefits, would have the advantage of covering 100 per cent of completed projects as well as providing a more timely assessment of the performance of these projects.
Experience with the suggested BME system should be comprehensively documented in the PCAs.

4. Post-Evaluation

92. The scope and timing of the post-evaluation exercise has an impact on its feedback value. PEO prepares PPARs for only a sample of projects, within two years of PCA circulation. At the time when the PPAR is prepared, investment costs are relatively well known, but operating benefits have only just begun to emerge. Thus, at present, the post-evaluation process does not provide an adequate empirical basis Ion which to draw firm conclusions on the development impact of projects. This is a major shortcoming and calls into question the relevance and utility of the exercise for planning future Bank operations in a DMC or in a sector across DMCs. On the other hand, re-evaluation studies (carried out about five years after PPAR completion to better assess the development impact) and impact evaluation studies (to assess the impact of the Bank operations in a DMC sector or subsector), which provide better measures of development benefits and their sustainability, have covered only 14 and 56 projects, respectively. These projects comprise only 16 per cent of all completed projects for which PPARs have been prepared. As regards the evaluation of program loans, other than commenting on compliance with loan covenants, the lack of a logical framework of objectives and associated verifiable, time-bound progress indicators in RRPs precludes any sound basis for post-evaluating program performance. In its current format, policy-based lending cannot be easily post-evaluated.

93. The Task Force recognizes that the adoption of the logical framework will be particularly useful in the post-evaluation of social sector projects and nonproject loans. It recommends that it be adopted as a post-evaluation tool in relevant projects to supplement the use of the economic internal rate of return.

94. Although some attention is given in PPARs to the institutional context of projects, the attention given to this subject needs to be increased. Unless the post-evaluation process interprets project performance in terms of the overall institutional environment of the project and also of the Bank, post-evaluation findings will have limited relevance for improving project performance by changing that environment. Too few PPARs comment on internal processes of the Bank or the EA as factors for improving project performance. When they do, institutional factors affecting project performance are often not translated into practical remedial actions.

95. With the adoption of country-focused operational programs, there is a need to provide a framework to monitor and evaluate whether the programs are formulated in line with the strategic objectives of the Bank and are consistent with the country strategy and effective in responding to the needs of the country. BME findings can be fed back to improve the quality of programming work. Moreover, the Bank's current feedback system has been limited to the findings from post-evaluation reports and studies undertaken by PEO. Although useful lessons arise from activities carried out by other offices and departments, they have not been systematically fed back into the operational system of the Bank. In addition, PEO's activities need to be carried out in closer coordination with the operational departments in areas such as selection of projects for post-evaluation and identification of thematic topics for special studies. This would make post-evaluation findings more relevant to the needs of the operational departments.

96. The Task Force recommends the following 1or the conduct of the Bank's post- evaluation activities:

PPARs should cover a maximum of 30 per cent of completed Bank projects, randomly chosen and stratified by country. PPARs should be undertaken only when data from the second year of operation of the project are available.
Project impact studies should cover 20 per cent of the Bank's completed projects.
A comprehensive annual performance evaluation program should be prepared to bring together the activities of different depar1ments/offices related to project performance.
Wherever relevant, PPARs should be based on the logical framework design of the project, and should cover an assessment of "process" as well; they should also evaluate capacity building efforts and the related institutional impact of the project.
A review of PEO should be undertaken to examine the desirable institutional changes, staffing levels and budget for post-evaluation in light of its workload.

97. The Task Force recognizes the impor1ant role played by the Board of Directors in providing guidance and feedback to ensure that quality projects are approved by the Bank and that the projects financed achieve significant developmental impact. Of particular importance is the feedback received during Board meetings when loan proposals are discussed and during Audit Committee meetings when PPAR findings are reviewed. Comments offered when COSS and COPPs are reviewed will be of increasing impor1ance as the Board gets more involved in these discussions. The Board also has the oppor1unity to review TA proposals as well as project implementation performance when the quar1erly PAC notes are reviewed. The signals Board members give are important to the Bank, as they reflect the direction of thinking, as well as the expectations, of shareholders from both developed and developing countries. Bank staff should continue to be sensitive to the concerns expressed and use them as feedback for preparing and processing better country programs and loan projects in the future.

Back
B. Project Implementation