Catalog

	Table of Contents
p. 32 of 95 BACK | NEXT Message from the Chairman of the Board of Directors Members, Capital Stock and Voting Power The Record Abbreviations 2004 in Review: Board of Directors' Report Special Theme: The Changing Face of the Microfinance Industry: Building Financial Systems for the Poor Part 1: Institutional Effectiveness Managing for Development Results Accountability Knowledge Management Achieving Development Impact Human Resources and Budget Administration Part 2: Poverty Reduction Part 3: Financial Statements: Management's Discussion and Analysis

Annual Report 2004 : Part 1: Institutional Effectiveness

---

Achieving Development Impact

Emphasizing Integrity, Impartiality, and Independence

The credibility of evaluation depends on the quality and integrity of analysis, the degree of transparency, and the independence of the evaluation unit. The dimensions of independence include (i) behavioral autonomy, (ii) avoidance of conflict of interest, (iii) insulation from external influence, and (iv) organizational independence. The autonomy of OED is evident from its project, sector, and country evaluation studies. OED both recognizes positive achievements and points out shortcomings. OED has maintained a high degree of transparency and openness in reporting and publicly disclosing its findings. To enhance its independence from Management, effective 1 January 2004, OED reported to the Board of Directors through the Development Effectiveness Committee (DEC). For the first time in 2004, the Board appointed the Director General of OED under the new procedures on the recommendation of DEC and the President.

The focus of OED's work shifted to cover (i) the broader developmental agenda, and (ii) project and program evaluations feeding into country assistance program evaluations (CAPEs). The shift requires closer links between the CAPE and the CSP and between sector assistance program evaluations and other evaluation studies (impact, special, and thematic). The alignment of OED's work in these areas will ensure that its products are timed to feed into the preparation of CSPs, sector policies, and development strategies.

Consistent with the redefined role of OED, the section of the Operations Manual on evaluation was reissued in September 2004 to reflect changes in OED's reporting structure and organizational arrangements and in the way in which private sector operations are evaluated. In coordination with the Private Sector Operations Department, OED is preparing new guidelines for project performance audit reports on private sector operations that incorporate the Good Practice Standards for Evaluation of Private Sector Investment Operations agreed upon among the members of the Evaluation Cooperation Group (ECG) of MDBs. The guidelines will also form the basis for preparing project completion reports (PCRs). OED also began to review existing evaluation guidelines for individual public sector operations (projects, programs, and technical assistance) to harmonize them with the good practice standards prepared by the ECG.

OED continued to participate in the ECG, including (i) developing good practices for policy-based lending, (ii) benchmarking private sector good practice standards, (iii) capacity building for evaluation, and (iv) strengthening and promoting the dissemination of lessons learned and of feedback. OED prepared a report on good practices for evaluating policy-based lending and an addendum to the ECG's good practice standards for MDB-supported public sector operations, both adopted by the ECG in 2004. To enhance in-house expertise and to improve the quality of its evaluation studies, OED introduced a seminar series, the OED Colloquium on Evaluation. Open to all ADB staff, the colloquium provides a forum for free exchange of methodological and empirical concerns on the effectiveness of development.

Country Assistance Program Evaluations

CAPEs are designed to assess the development effectiveness of ADB's strategy and assistance programs in a country over an extended period, to derive lessons learned, to put forth recommendations for more effective future assistance, and to improve coordination among aid agencies. CAPEs have generally focused on the quality of ADB's strategies and programs, their implementation and outcomes, and their perceived contribution to overall development. Eight CAPEs have been done since 1998—Bangladesh, Cambodia, PRC, Mongolia, Nepal, Papua New Guinea, Philippines, and Viet Nam—with the current rate usually two per year. CAPEs have used varied methodologies including focus group interviews, formal and informal surveys, detailed desk reviews of existing data, statistical analyses, field visits, and case studies.

CAPEs revealed problems with the former country operational strategies (COSs) and country assistance plans (CAPs) and thus contributed to the decision to switch to the CSP as COSs and CAPs tended to address the formal, internal needs of ADB rather than providing new insights for ADB operations. COSs were too vague and general; they should instead have been more operational. CAPs should have been more focused and selective. The eight CAPEs consistently pointed to the need for a standard methodology for evaluations which will be developed in 2005.

The CAPE for Nepal done in 2004 was prepared in parallel with the CSP, ADB's first results-oriented CSP. The two processes fed into each other. Overall, the Nepal CAPE concluded that the ratings of completed projects were generally within the level of success achieved by other countries in the region. The evaluation found that the government and ADB were able to improve performance and capitalize on successes. Another major strength was the willingness of the government and ADB to tackle difficult policy and institutional issues in the major productive sectors. Shortcomings included an overestimation by ADB of the ability of government agencies to resolve institutional and policy issues, which resulted in slow project/program implementation. Recommendations in the CAPE that were incorporated in the CSP include (i) development of a more focused program, (ii) flexibility in responses to in-country conditions, and (iii) sector focus that can enhance successful investments based on proven achievements in management.

Development Results of ADB’s Lending Operations

The performance of ADB's lending operations is assessed through evaluation of completed projects and programs, self-evaluation by operations departments in the PCR, and independent evaluation by OED in the project/program performance audit report (PPAR). PCRs are prepared for all public sector projects/programs 1–2 years after their completion but have contained a rating only from 1995 onward. The target 25% PCR average for private sector projects has not been achieved. PPARs are typically prepared 3 years after project/program completion. Until 1984, PPARs were prepared for all projects and programs. Thereafter there has been 100% coverage for program loans, but coverage of projects was reduced to 70% and then to 35%.

Both PCRs and PPARs use the same methodology for rating a project/program. A three-category system—generally successful, partly successful, and unsuccessful—was used prior to 2000. As part of harmonizing the evaluation methodologies of MDBs, a four-category system—highly successful, successful, partly successful, and unsuccessful—has been used since 2000. The rating is derived on the basis of five criteria: relevance, efficacy, efficiency, sustainability, and institutional development and other impacts. Work started in 2004 to develop guidelines for private sector operations evaluation that, among other things, will broaden coverage of assessing development results.

PCR and PPAR results are aggregated by using PPAR ratings in all cases where both PCR and PPAR ratings are available. To provide a “success rate" for all rated individual operations, projects/programs rated generally successful under the three-category system and either highly successful or successful underthe four-category system are added together. During 1974–2004, 1,048 public sector projects/programs were completed and rated through PCRs and/or PPARs. Of these, 654 (62%) were rated successful, 311 (30%) partly successful, and 83 (8%) unsuccessful.

The figure below shows the trend by year of completion for projects/programs rated successful. Due to the relatively small number of projects completed each year, there are wide fluctuations from year to year; however, two distinct trends can be discerned. From 1974 to 1989, there was a declining trend in the proportion of programs and projects rated successful. The downward trend was reversed in 1990, which may be explained by the combined effect of efforts to improve project quality covering both project preparation and project administration and by changes in the composition of operations. An analysis of project design and formulation carried out for 201 projects comprising all PPARs prepared from 1994 to 2003 revealed that the design of ADB-funded projects improved over the last decade. The ratio of projects with major design issues fell from 50% of approvals during 1976–1980 to 12% during 1991–1995. In terms of composition, an improvement in ratings for South Asia was reinforced by higher success rates in large, new borrowing DMCs, e.g., India. A slow improvement in ratings for agriculture and social infrastructure was strengthened by the higher success rate and higher proportion of energy and of transport operations.

Of the 1,227 completed public and private sector projects/ programs with PCRs, 689 have been independently evaluated. The PPARs rated 59% of these projects/programs as successful, 31% as partly successful, and 10% as unsuccessful. PPAR ratings by sector ( see table on page 42) show continuing high success rates for transport and communications and for energy projects/programs. These results confirm that financing infrastructure projects is one of ADB's strengths.

On the other hand, there is a significantly lower probability of success for agriculture and natural resource projects/programs, although a small improvement has been achieved for those evaluated over the past 10 years with half rated successful. Agriculture projects/programs have often been adversely affected by weaknesses in formulation and design, ineffective policy environment, institutional weaknesses, and declining commodity prices. Changes have been made by ADB in some of its approaches in the sector, but it appears that they are not working well enough. The special evaluation study (SES) on Effectiveness of Participatory Approaches in Rural Development Projects found that participatory or bottom-up approaches do not necessarily offer an effective solution to problems encountered in earlier projects. A further serious review of ADB approaches for the agriculture sector is needed to develop the right business models for the current environment as about two thirds of the region's 3 billion people live in rural areas and depend on agriculture. The SES Small-Scale Freshwater Rural Aquaculture for Poverty Reduction recommended steps to make ADB operations in aquaculture development more relevant for poverty reduction.

ADB's operations in the financial sector have also performed relatively poorly ( see table on page 42), mainly due to the poor performance of development finance institution (DFI) projects, which comprise 88% of the total evaluated in the sector. Excluding those in the Republic of Korea and Singapore, the success rate for DFI projects declined to 32%. Areas of concern in the sector include (i) the institutions through which funds are channeled since experience with state-owned institutions has been less than successful, particularly in the Pacific; (ii) poor assessment of demand for credit resulting in lower than anticipated disbursements in a number of cases; (iii) ADB lagging behind commercial banks in developing a broader range of suitable products for DFIs; and (iv) inadequate monitoring and records of key impacts (e.g., jobs created, exports) making evaluation of outcomes difficult. As with the agriculture sector, ADB's business model is not working well enough for the financial sector. Further serious review is needed to get the right business models to improve the performance of projects in the sector.

The results of PPARs done in 2004 confirm the continuing success of ADB lending operations in the energy and transport sectors as well as modest improvements in the agriculture and social sectors. All four infrastructure projects evaluated were rated successful. Among those, the trans-basin Nam Leuk Hydropower Project in the Lao People's Democratic Republic (Lao PDR) exceeded the targeted output of providing 215 gigawatt-hours per year. With diligent operation and maintenance, the plant can be sustained for more than its 40-year designed economic life. Project-related relocation issues were carried out well; some follow-up actions were recommended on a few resettlement issues. In the agriculture and social sectors, only five of nine projects/programs evaluated were rated successful. The successful rural enterprise credit project in Thailand provided small loans to farmers and operators who used them productively. Project clients included women who were full business partners in their enterprises. The successful education development project in the Cook Islands enabled schools to mobilize funding from the government, parents, and their communities to enhance the quality of education and to improve school facilities and infrastructure. Parents and communities became proactively involved in school management and became aware of the importance of educational standards.

Portfolio Performance

Annual Report on Loan and Technical Assistance

OED's third annual report on portfolio performance represented a significant move to a more evaluative format. Special investigations were conducted into projects delegated to RMs, project performance management, cancellation of surplus loan proceeds, and the country portfolio review mission.

Based on project performance report (PPR) ratings, there was little change in portfolio performance year to year. Evidence shows that the PPR system fails to identify all projects at risk. The report points out the incompatibility of the dual purposes served by the PPR and recommends that it be used only for its intended purpose of identifying the bottom quartile of projects in terms of performance. Key performance issues include the following: (i) over the last 5 years, 84% of the projects closed required extensions, and fully 65% required multiple extensions; (ii) net resource transfer from ADB to DMCs was a negative $5.3 billion in 2003, heavily influenced by $5.6 billion of voluntary prepayments from large borrowers that refinanced their loans elsewhere; (iii) many loans were approved before project readiness criteria were met; (iv) disbursements were at their lowest level in 5 years; and (v) cancellation of surplus loan proceeds remains a significant issue. The principal recommendation of the report is to pilot-test annual reporting on generating project outputs.

Project Cost Estimates

Significant positive correlations were found between the level of cost underruns and the number of scope changes, the proportion of local costs to the total cost, the proportion of the local currency component, and the proportion of costs other than civil works. In analyzing variations between expected and actual cost estimates, PCRs cited price-related reasons most frequently (71%), scope changes (35%), and inaccurate estimation of base costs (33%). Other reasons for underruns included natural calamities and events related to peace and order (38%).

Key factors for major overruns or underruns were poor cost estimates in feasibility studies, poor project design, major delays in implementation, and predetermined loan amounts. A survey of ADB staff ranked inadequate or difficult-to-collect information and inadequate resources and time for feasibility studies as the most important factors in poor project cost estimation. Recommendations of the study include (i) realistic time and resource allocations to improve project design; (ii) more rigorous consideration of inflation and depreciation factors in cost estimates; (iii) more realistic estimation of civil works and their operation and maintenance costs; (iv) consideration of two-part loans for projects, one for base costs and one for contingencies and minor overruns; (v) more attention and resources for project administration; and (vi) periodic updating of cost estimates.

Project Frameworks

As part of the focus on MfDR, OED promoted the use and improved the quality of project frameworks. Since the introduction of the project performance management system (PPMS) in the 1990s, OED has played a leading role in its development. The project framework is a core element of the PPMS and a principal tool for ensuring sound project design as it establishes the basis for monitoring and evaluating performance. OED staff presented briefing sessions attended by more than 300 headquarters staff on what to look for when assessing the quality of project frameworks. The group sessions generated a large demand for one-on-one guidance. To meet the needs of RM staff and those unable to attend the briefing sessions, a video presentation was recorded. The briefing material and a checklist have been included in the operations tool kit. OED assessed the quality of all frameworks for loans and technical assistance approved in 2004. The quality of loan frameworks showed a statistically significant improvement compared with 2002, with the average 2004 score in the satisfactory range (it was partly satisfactory in 2002). However, the quality of advisory technical assistance showed almost no improvement, and the average rating remained partly satisfactory. Regional technical assistance showed significant improvement but overall remained partly satisfactory.

Development Results of ADB’s Technical Assistance Operations

ADB must provide knowledge and innovation to complement its lending operations. Technical assistance is an important strategic tool for ADB, particularly in enhancing institutional development in DMCs.

As with lending operations, technical assistance is evaluated through a combination of self-evaluation by the responsible department and independent evaluation by OED. Independent technical assistance evaluation by OED started in 1990; self-evaluation through completion reports by responsible departments began in 1993. The two methods have yielded different results. Completion reports, which focus on outputs, have reported consistently high success rates of over 80%. Independent performance audits, which focus on achievement of results and sustainability of outcomes, have shown much lower success rates averaging 49%–62% ( see tables below).

The results of technical assistance are mixed. Successful technical assistance contributed significantly to institutional development in a number of DMCs and played a catalytic role in some sectors. For example, ADB-financed technical assistance for the development of genetically improved farmed tilapia served as a major impetus in associated research and development investments. ADB's interventions through technical assistance on nutrition and food fortification played a catalytic role in nutrition development in Asian countries.

On the other hand, ADB encountered some problems with the performance of its knowledge-based products. The SES on capacity development assistance to the Lao PDR concluded that the overall impact on the country was scattered; the report suggests that ADB systematically review its approach to capacity development. The results of technical assistance activities evaluated in 2004 showed that considerable improvement is needed so that DMCs get the full potential benefits of ADB's knowledge-based products. Only 5 of the 17 evaluated were rated successful, and 11 were partly successful. Nearly two thirds did not fully achieve their objectives and/or experienced problems with effectiveness, efficiency, sustainability, or impact. The findings of key technical assistance performance audit reports done in 2004 follow.

Fiscal Management and Tax Reform

This study evaluated six technical assistance operations in India approved during 1995–2003. Three were successful and three partly successful. Three focused on capacity building as part of the fiscal management programs in Gujarat, Madhya Pradesh, and Kerala. The report concludes that technical assistance in Gujarat was partly successful and was successful in Madhya Pradesh and Kerala. The report notes the importance of political will to reduce budget deficits and to improve governance as prerequisites for effective fiscal management. Another focus of technical assistance was building capacity to administer value-added taxes that the government decided in 1999 to introduce nationwide by 2002 but subsequently postponed until 2005. The report concludes that capacity building in a large country like India requires more work, a sustained commitment by ADB, and the involvement of high-level staff.

Institutional Strengthening in the Power Sector in Viet Nam

Five technical assistance operations aimed to strengthen power sector institutions by (i) developing financial accounting andmanagerial systems, (ii) improving the managerial and technical skills of staff, and (iii) formulating tariff policies and structures. By and large, their impact was limited. Four were rated partly successful, and one was not rated because it had been only recently completed and its full impact was not clear at the time of evaluation. Important findings of the report are that (i) key recommendations were dismissed as irrelevant because they could not be implemented readily and more practical recommendations were overlooked; (ii) the executing agencies did not demonstrate strong ownership and proactive involvement, and outputs were regarded more as a reference than as an agreed upon course of future action; and (iii) results were not adequately disseminated to relevant participants, and most stakeholders and the government agencies concerned were in fact generally unaware of them. To increase the involvement of recipients and to help ensure that the design of subsequent projects can achieve the desired outcomes and impact, the report recommends that ADB request executing agencies to evaluate the performance of technical assistance activities within 2 years of completion.

A broad lesson learned to date is that capacity building is a long-term process requiring comprehensive support, training, monitoring, and strong supervision, and cannot be achieved within the short duration of a single technical assistance activity. ADB needs a strategic focus for technical assistance operations in a given country and not just for particular sectors or themes.

Back Knowledge Management	Next Human Resources and Budget