Catalog

	Table of Contents
p. 10 of 10 BACK | NEXT Operations Loans Technical assistance Grants Equity investments Cofinancing and guarantee operations Portfolio management Private sector operations portfolio Resource transfers Performance evaluation and development impact

Annual Report 2002 : Operations

---

Performance evaluation and development impact

The Operations Evaluation Department (OED) independently evaluates ADB’s operations, including the performance of selected projects, programs, and technical assistance about 3 years after completion. It also undertakes thematic evaluation studies in the form of impact, special, and country assistance program evaluations. In 2002, OED completed 48 evaluation reports. This included 31 project or program performance audit, 6 technical assistance, 3 country assistance program, 6 special and impact evaluation, and 2 annual reports.

OED prepared for the first time a report evaluating the performance of ADB’s portfolio of ongoing projects and programs. OED was further strengthened in 2002 with an advisor post for identifying appropriate indicators for measuring development effectiveness.

Project and program performance audit reports

In 2002, 27 public sector projects and 6 programs that were completed in or around 1998 were assessed in 30 reports. Based on five evaluation criteria—relevance, efficacy, efficiency, sustainability, and institutional/other development impacts—4 projects/programs (12%) were rated highly successful, 23 successful (70%), and 6 partly successful (18%). None was rated unsuccessful. These results continue a trend toward higher success rates for projects and programs. There is little difference in success rates among country groups. Partly successful and successful projects/programs appeared in all four country groups (A, B1, B2, and C; see box on country classification on page 66), and highly successful projects in three of them. Of 15 infrastructure projects and programs—6 in energy and 9 in transport and communications—3 were highly successful, 11 successful, and 1 program partly successful. Of 10 agriculture and 5 social sector projects, 10 were successful and 5 partly successful. The other 3 projects/programs were successful or highly successful. In addition, one private sector project was evaluated and rated successful. To view evaluation reports, go to http://www.adb.org/evaluation/.

Of the six programs, four (67%) were rated successful and two (33%) were partly successful. All four successful programs, including agriculture sector programs for the Kyrgyz Republic and Viet Nam, achieved their program purpose with impacts likely to be sustained. On the other hand, the partly successful Agriculture Sector Program in Mongolia accomplished only three fourths of policy reform measures and experienced some key policy reversals. Of the 27 public sector projects, 4 (15%) were rated highly successful, 19 (70%) successful, and 4 (15%) partly successful. The sector lending approach resulted in the highly successful Third and Fourth Power Transmission (Sector) projects in Thailand and the successful Tenth and Eleventh Road (Sector) projects in Indonesia.

The highly successful National Air Navigation Development Project in Mongolia generates significant foreign exchange earnings from overflights, and has improved safety and efficiency at Ulaanbaatar Airport. The highly successful Laiwu Steel Modernization Project in the PRC has been a model for modernization in the steel industry and the reform of state-owned enterprises. On the other hand, the four partly successful agriculture projects showed some weaknesses in their design and implementation. The Employment Generation Project in Mongolia and Second Barani Area Development Project in Pakistan were not effective in reaching out to the majority of their intended poor beneficiaries. For some successful projects, outcomes were constrained by the lack of beneficiary participation and limited impact on poverty.

The performance audit reports confirmed project completion report ratings for 27 public sector projects/programs, and reclassified 6. The Bangladesh Railway Recovery Program was reclassified from generally successful to partly successful owing to a relapse in both program components and progress toward financial sustainability. Exceptionally, another five projects/programs were reclassified from partly successful to successful in recognition of sustainable achievements, and associated institutional and socioeconomic development impacts.

Lessons: The evaluations provided several lessons. For poverty reduction, the evaluations highlight the importance of beneficiary participation and stakeholder ownership, multiple synergistic interventions based on client demand, capacity building for local government, social preparation for the poor prior to physical investments, and arrangements for effective implementation with nongovernment organizations and communities. For program lending, the success of a reform process depends on the sustained commitment of all stakeholders to policy reforms, stemming from ownership built before reforms, a sharply focused program design that recognizes implementation capacity and the sensitivity of reforms, and flexible program implementation.

Special and impact evaluation studies

Beneficiary perceptions of the impact on poverty reduction of agriculture and social infrastructure projects: A special evaluation study using surveys examined beneficiary perceptions of the impact on poverty reduction of selected agriculture and social infrastructure projects in six DMCs. Depending on the type of impact, 24–50% of the beneficiaries felt they gained substantially from ADB-financed operations. The proportion was higher for income and general well-being benefits than for more specific benefits to women and the environment. Projects approved in the 1990s were more effective than those approved in the 1980s. A significant survey finding was that among the poor, only a small improvement in income was needed for them to perceive themselves as no longer poor. Major lessons included the need for a balanced program of poverty intervention projects at the macro level; consultation with beneficiaries during project preparation and implementation; more time and resources in promoting social capital for every poverty intervention project; beneficiaries’ trust in development partners; and timing in the projects’ turnover to beneficiaries prepared to handle the responsibilities.

Impact of rural road improvements on poverty reduction: A case study-based impact evaluation examined the impact on poverty reduction of ADB-financed rural road improvements in road project investments and rural development projects. Better rural roads are a necessary but not sufficient condition for graduating from poverty. The poor first need to accumulate a surplus, for example by reducing time for collecting water, allowing them time to seize new opportunities from motorized transport. The study recommends designing interventions that concentrate on removing access and mobility constraints of the poor in their existing livelihoods, through improved transport modalities and carrying capacity; providing transport services by the government as a public service obligation and later developing sustainable private transport services; establishing transparent criteria for selecting a particular road; and encouraging labor-based technology for income generation.

Investment fund operations (IFOs): An evaluation study examined the development impacts of 29 IFOs relating to infrastructure, venture capital, portfolio investment, and special purposes, representing about 19% of cumulative private sector and about 59% of cumulative equity investment operations of ADB. The IFOs had significant impact on mobilizing resources particularly for infrastructure, supporting small- and medium-sized enterprises, developing capital markets, and generating employment opportunities. However, financial performance was generally modest, affected by the Asian financial crisis; changes in regulations such as in the telecommunications sector; and the ability of fund managers to identify, appraise, and supervise projects. The study recommended follow-up actions for ADB in the selection and remuneration of fund managers, administration of IFOs, monitoring and evaluation of projects financed by IFOs, and an exit policy for ADB.

Water supply and sanitation projects: An impact evaluation study of water supply and sanitation activities reviewed 50 loans and technical assistance and undertook 6 in-depth assessments in selected DMCs. All projects substantially increased water consumption by households, with more than half the beneficiaries living below the poverty line, although satisfaction with improved water services from piped connections varied among domestic and nondomestic consumers. Sanitation received less attention than water supply systems; its impact has been limited and mixed. None of the water utilities concerned had achieved full cost recovery. Important lessons were the need to include sanitation, hygiene, and health promotion in more projects; apply further measures to manage demand and reduce nonrevenue water; assess alternative means of distributing water, such as bottled water for drinking; involve beneficiaries at all stages of rural water supply projects; and aim utility pricing at full cost recovery to ensure continued and expanded water supply and sanitation services. The recommended actions are consistent with the Water for Asian Cities Program, announced by ADB President Tadao Chino at the World Summit on Sustainable Development in Johannesburg.

Primary education projects: A special evaluation study examined government and nongovernment provision of primary education in three DMCS, to assess whether the private sector provided higher quality, better school management, and greater access to primary education. Using interviews with primary education stakeholder groups and secondary data, the evaluation concluded that private provision reduces the financial burden on government and improves access; and that while for-profit private schools also improve quality and efficiency, they may do so at a cost to poor families. The quality of education continued to be a problem: teachers lacked qualifications and the number of classroom hours was inadequate.

The study showed that a long-term sector framework should be developed on interaction between government and nongovernment, religious, and general agencies, and formal and nonformal schooling systems; more resources are required for reaching and retaining in school the children from the remote areas; school management and committees need legal powers, budget, and training; local governments, communities, and parents should be included in capacity-strengthening efforts; and financial norms should be established to maintain a minimum acceptable level of per student and nonsalary proportion of recurrent expenditure.

Country assistance program evaluations

Three country assistance program evaluations were undertaken for Bangladesh, Mongolia, and Philippines.

Bangladesh: From 1973 to 2001, Bangladesh received 141 public sector and 8 private sector loans and equity investments, and 272 technical assistance grants. Of 52 postevaluated projects, 38% were classified as generally successful, 52% partly successful, and 10% unsuccessful, showing a much higher proportion of partly successful projects than ADB’s average. ADB’s country strategies and operations recognized the need for mainstreaming governance issues, took a long-term approach to structural reform, and provided a significant impact on economic growth and poverty reduction through infrastructure projects.

Weaknesses included failure to achieve sufficient structural reform in some sectors, inability to implement governance reforms affecting local communities, inadequate support for health and education, and failure to mainstream the environment in the country program.

There is a need for

• a more proactive approach to governance, both at central and local government levels;
• more lending for small enterprise development for employment generation;
• support for increasing the recurrent budget for health and education; and
• focus on fewer sectors to enhance poverty reduction and development effectiveness. The last of these implies reduced support in sectors where sector performance is poor, the potential for catalyzing is limited, and ADB has no comparative advantage. Commitment to reforms to improve sector efficiency should be a key criterion.

Mongolia: An assessment of all operations in Mongolia since 1991 focused on four themes: policy reform, economic diversification, capacity building, and financial system reform. Policy reform assistance was all-encompassing in the context of transition and development, and led to a major change in ownership patterns. However, policy matrixes were overloaded, and policy-based lending to agriculture and industry was only partly successful. Changes in price and trade policies with investment in infrastructure sectors laid the foundation for unbundling industries and increasing private sector involvement. Capacity building was supported in all operations, but mostly focused on inputs and was equated with training and the provision of consulting services. The fragmented approach did not promote overall public sector resource management. The sector strategy became clearer in the mid-1990s partly through assistance to the financial sector, which was starting to have a positive impact in the late 1990s. All three country operational strategies recognized key transition issues, but did not convert them sufficiently into sector strategies and projects. Overall, ADB’s country assistance program made significant contributions to transition, with an acceptable portfolio performance. Its impact on poverty is difficult to assess because of external and natural influences.

The evaluation underscored the importance of

• providing strategic guidance to operations;
• undertaking necessary background work to, and using a logical framework approach in, formulating country strategies and programs (CSPs); and
• setting specific monitorable targets for country operations.

Philippines: ADB’s operational strategy for the Philippines tracked the national strategy and lent support to the country’s overall development objectives of macroeconomic stabilization, poverty reduction, and social development. The outcome of investment projects was disappointing: 33% of all completed projects from 1986 to 2001 were rated unsuccessful, although there was some improvement through the 1990s. Issues of sustainability arose in all lending and nonlending interventions, stemming from lack of institutional capacity, inadequate budget allocations, and lack of political will.

There were four main lessons.

• Political and macroeconomic stability is a critical factor in the success of development assistance.
• Project success requires careful project preparation and relatively simple project design.
• Sustaining development impact requires careful monitoring during implementation and beyond.
• The success of the assistance program depends on the degree of institutional development.

Technical assistance performance audit reports

Six technical assistance performance audits were undertaken in 2002. Thirty-three technical assistance activities were evaluated: 6 were rated highly successful, 18 successful, 8 partly successful, and 1 unsuccessful.

Capital market development: Six technical assistance supported reforms and institutions for capital market development in Pakistan. Expected results were largely realized with a stronger regulatory framework in the capital market; effective functioning of the Securities and Exchange Commission; updating of regulations on mutual funds; privatization of public sector mutual funds; and enactment of a new insurance law and issuance of rules. Two public sector insurance entities were restructured and interest rates on the national savings were rationalized and are being adjusted periodically based on market signals.

The main lessons are that strong government ownership is critical to the success of policy and institutional reform, and sharply focused technical assistance projects have a better chance of success.

One technical assistance was rated highly successful, three successful, and two partly successful.

Road sector institutional development: Five technical assistance projects for institutional development and policy support in the road sector were attached to projects in Kazakhstan, Kyrgyz Republic, and Mongolia; and a regional technical assistance for road design and construction standards in Kazakhstan, Kyrgyz Republic, Mongolia, and Uzbekistan. Initial assistance supported organizational change. Progress was made in developing road departments to manage the road network and become the client for maintenance and construction contractors. Follow-on technical assistance had less impact; the governments failed to implement some politically sensitive recommendations such as establishing a road board in Mongolia, and giving control of the road fund to the road department in the Kyrgyz Republic. The revised design and construction standards from the regional technical assistance are not in use in Kazakhstan and the Kyrgyz Republic; in Mongolia, some are in use.

The main lessons are that workshops and training on procurement processes may be less appropriate than clarifying job classifications and descriptions; a road fund by itself is not a guarantee that sufficient funds will be provided to meet sector needs; greater clarity is required on the issue of road funds among development partners; and regional activities should take note of government priorities and relevant ongoing activities.

The three initial technical assistance were rated successful, the two follow-on technical assistance partly successful, and the regional technical assistance unsuccessful.

Education sector reform: Six technical assistance for reform of the education sector in Central Asian countries were evaluated. For Kazakhstan, a comprehensive sector review led to planning and establishing the rationalization process, developing curriculum reforms, strengthening textbook development capabilities, and training teachers. One technical assistance for the Kyrgyz Republic produced a blueprint of policy reforms; a second suffered from uncooperative consultant team members and frequent changes of senior counterpart staff. In Uzbekistan, a functional reform monitoring system was developed, as were strategies for improving the cost-effectiveness of the education system.

Lessons included the following: inception missions should familiarize consultants with the goals and objectives of the technical assistance, and establish the critical achievement factors; technical assistance designs should allow for flexibility; continuous dialogue on education policies is needed to respond to changing needs and aspirations of the beneficiaries; and evaluation capacity development should include training in data management.

The two technical assistance in Kazakhstan were rated successful, one each in the Kyrgyz Republic successful and partly successful, and the two in Uzbekistan highly successful.

Pacific audit capability: Six technical assistance for strengthening audit capability in 12 Pacific DMCs included four regional technical assistance cofinanced by supreme audit institutions. The design and purpose were appropriate. The expected benefits were improved audit knowledge and skills of the Office of the Auditor General staff, improved internal audit work and work processes, increased number of trained audit staff, and enhanced sharing of audit experiences and solutions. Weaknesses still need to be overcome in legislation, reporting processes, accounts preparation, and management of financial accounts and computer systems.

Lessons for the 12 Pacific DMCs include the need to ensure that legislation and government preparation of accounts and computer systems are consistent with the purpose of the technical assistance; amend legislation to expand the scope of auditing; and ensure follow-up after auditing to increase the overall effectiveness of the process.

Three technical assistance were assessed highly successful and three successful.

Road sector management: Four technical assistance for improving road sector management included two technical assistance attached to road improvement projects in the Lao PDR and the Philippines for promoting privatization and management of road sector institutions, respectively; and two stand-alone technical assistance to Papua New Guinea (PNG) for supporting the establishment of a road asset management system (RAMS) for the central and provincial governments. The privatization of some transport sector functions in the Lao PDR, and restructuring in the Philippines, were carried out. The first PNG technical assistance established the RAMS in the central Government; the second has not yet installed the RAMS in the provinces.

The lessons were that a politically sensitive technical assistance attached to a loan may be a more effective mechanism than a stand-alone technical assistance; long-term involvement of external funding agencies is necessary to maintain momentum; where staff retrenchment is considerable, transition arrangements need to be designed to minimize social and political costs; and management buyouts may be considered an appropriate form of privatization for transition economies like the Lao PDR.

Three technical assistance were rated successful and the technical assistance aimed at the provinces in PNG, partly successful.

Agricultural planning in Nepal: Five technical assistance supporting agricultural planning, statistics, and institutional strengthening in Nepal were designed to assist the Government in setting a new direction for the agriculture sector through a 20-year Agriculture Perspective Plan. The policy directions of the first technical assistance were relevant. Two subsequent technical assistance, in the context of an agriculture program loan, assisted in shifting the strategy toward less reliance on public investment and subsidies, and more on deregulation and market forces.

The lessons were as follows: difficulties can arise with a change in ADB staff; participation needs to be properly managed to create the opportunity to introduce innovative approaches; and without extensive local participation the plan produced was overambitious.

Discontinuing the institutional structure established for plan implementation, and the Government’s reluctance to publish the updated agricultural statistics, limited the impact of the other two technical assistance. Overall, three were rated successful, and two partly successful.

Portfolio performance

An annual evaluation report on portfolio performance showed a reduction in the percentage of problem projects, from 31% in 2000 to 25% in 2001; a substantial gap between projected and actual contract awards with only 69% achievement; and declining compliance on the submission of audited project accounts.

The report concluded that to improve portfolio performance and reduce the number of projects at risk, high-level support is needed for ADB’s project performance management system by highlighting the results of project performance report assessments in management decision making; assigning adequate resources for project performance report preparation and validation; and providing adequate resources for internal skills development. For more on annual evaluation reports, see http://www.adb.org/Evaluation/annualreports.asp.

Other activities

OED provides direct support to operations through its review of new projects and project completion reports. In 2002, it also assessed the new loan classification system, and took the lead in reviewing the process for performance-based allocation of ADF resources.

The ADB Board of Directors’ Development Effectiveness Committee’s discussion of the annual report of evaluation activities for 2001 resulted in a study in 2003 on project cost estimation, recognizing the high number and extent of cost underruns in recent years. For more on the Development Effectiveness Committee, see the 2002 in Review: Board of Directors’ Report chapter.

The Evaluation Cooperation Group of the multilateral development banks (MDBs), having completed good practice papers for individual public and private sector operations, turned its attention to higher-level evaluations, including country assistance programs and evaluation of policy-based lending, a study for which OED has prepared a framework and terms of reference. OED and the other evaluation departments are responding to the MDB presidents’ statements in 2002 on the need for results-based monitoring and management.

Back Resource transfers	Annual Report 2002