Catalog

	Table of Contents
p. 4 of 7 BACK | NEXT Maximizing Institutional Effectiveness Streamlining Organizational and Business Processes Mobilizing Human Resources and Budget Assuring Development Impact Strengthening Accountability Managing Knowledge Fostering Partnerships

Annual Report 2003 : Maximizing Institutional Effectiveness

---

Assuring Development Impact

ADB has made significant progress in integrating performance considerations into its operations. Improved use of project evaluation results and project performance management systems is helping to create better designed and better targeted interventions.

Reorganization of OED

The Operations Evaluation

The Operations Evaluation Department (OED) assesses projects, policies, and procedures to account for the use of resources, to provide lessons for future operations, and to sustain activities already completed. Evaluation criteria are relevance, efficacy, efficiency, sustainability, and impact. Evidence presented in 2003 showed that since 1989 the success rate of individual operations and of most sectors and countries has been increasing (see the Annual Review of Evaluation Activities in 2002 at http://www.adb.org/Documents/ PERs/RPE_OTH_2003_12.pdf). During 2003, OED completed 38 reports including 22 project and program performance audits, 6 technical assistance reports, 2 country assistance program evaluations, 6 special and impact evaluations, and 2 annual reports.

In December 2003, the Board approved a set of recommendations to further enhance the independence and effectiveness of OED. The following changes that came into effect on 1 January 2004 brought OED to a comparable position with evaluation units in other multilateral development finance institutions.

1. OED will report directly to the Board through the Development Effectiveness Committee (Committee).
2. The Director General of OED (Director) will be appointed by the Board upon the joint recommendation via the Committee of the Board and the President.
3. OED will have fuller autonomy in selecting its staff; the final selection of OED personnel will be made by the Director in consultation with the Budget, Personnel and Management Systems Department in accordance with ADB human resource policies and procedures.
4. The annual work program of OED prepared in consultation with the Committee and the President will be approved by the Board.
5. Within the overall ADB-wide budgetary framework, the Director in consultation with the Committee and the Budget, Personnel and Management Systems Department will prepare an annual budget proposal for OED which will be presented by the President for consideration and approval of the Board in a separate resource envelope.
6. Completed evaluation reports will be circulated by the Director simultaneously to Management and to the Board for their response. Disclosure of reports, which will be authorized by the Director, may include as attachments Management’s response, OED’s comments on it, and a summary of discussions by the Committee.

Corresponding changes in practices relating to work program and budget and to circulation and disclosure of reports will be instituted during 2004 based on continued engagement with operations and other departments. OED’s role will be guided by the principles of usefulness and timeliness and of disengagement. The first principle requires OED to use lessons identified in evaluations in reviewing the design of future projects and to provide timely and useful inputs to current operations. The second principle requires that OED maintain an arm’s-length relationship from day-to-day operational decisions. OED will continue to have unfettered access to all relevant ADB records and information.

The results of evaluations undertaken in 2003 and in recent years, with summaries by sector and country, and all evaluation reports can be viewed at http://www.adb.org/Evaluation.

Project and Program Performance Audit Reports

Most public sector projects and programs evaluated in 2003 were completed in 1999. Of the 22, 16 were rated successful or higher. None was rated unsuccessful. Seventeen confirmed the ratings given in completion reports, and five were reclassified. The economic infrastructure sectors continued to record the highest success rates. Five (62%) of eight programs were successful compared with an average of 30% over the last 5 years.

Twenty-four technical assistance projects for capacity building and advisory studies attached to these projects/programs were also evaluated. Of the 21 that were rated, 11 were successful or better, 8 were partly successful, and 2 were unsuccessful.

Five programs involved financial sector reform. Three were successful, including in particular a strong commitment by the Government of Viet Nam to sustain positive impacts of financial sector reform. The Agriculture Sector Program in Cambodia, the first for the country, removed some key constraints including the passage of a land law. The Public Sector Reform Program in the Federated States of Micronesia was successful and helped restore fiscal balance to the country.

Of the 14 projects evaluated, 2 (14%) were rated highly successful, 9 (64%) successful, and 3 (22%) partly successful. The highly successful Dalian Water Supply Project in the People’s Republic of China (PRC) confirmed that commitment at the local government level is an essential factor for success.

Special and Impact Evaluation Studies

A special study was undertaken on financial cost recovery in the power sector in 14 DMCs during the 1990s. ADB’s contribution was assessed in part through the record of compliance with financial covenants by 42 power utilities that received 69 loans totaling $9.2 billion during 1990–2001.

The principle of financial cost recovery in the power sector is that revenues from electricity sales should fully recover operational expenses and depreciation and should generate a reasonable return on the capital invested. The three key criteria for evaluating cost recovery were (i) cost minimization, (ii) tariff setting, and (iii) revenue collection efficiency. Performance in three countries was satisfactory; in five countries was unsatisfactory; and in six countries was tolerable. The review of compliance with covenants showed an overall average compliance rate of 61%. ADB is (i) moving away from further lending to utilities that are in chronic default of covenants; (ii) assisting DMCs to create new institutions with a more businesslike approach to utility management; and (iii) shifting away from project lending to utilities to program lending to governments for improving the sector policy environment.

Many countries are moving in the direction of sector reforms involving independent regulatory bodies, unbundling the sector by function, and establishing a competitive wholesale market for electricity. ADB should assist integrated utilities to unbundle their utility tariffs and to organize their business units by function. Commercialization should ensure that unbundled tariffs are at full cost-recovery levels.

Two studies in particular yielded valuable lessons for future ADB assistance in the power sector. Assistance to Bangladesh over the last 30 years comprised 16 public sector loans and 1 private sector investment for a total of $1.1 billion as well as 19 technical assistance grants totaling $8.8 million. Lessons learned include the need to (i) minimize extensive delays in implementation; (ii) improve local capacities in project preparation, implementation, and operation and selection of appropriate technology; (iii) improve ownership of technical assistance; (iv) improve financial management capacities; and (v) avoid use of project phasing to finance a change such as improving financial systems. At the strategic level, a key lesson was that ADB’s approach to sector reform, which is based on piloting change and learning by doing, facilitated change where it took place through the success of individual projects and newly created entities.

A study of the impact of power assistance in Indonesia since 1980 provided further guidance for future operations. The conclusion was that conventional lending operations had significant, positive impacts on the development of power systems and on socioeconomic and institutional development but that sector reform assistance had not been as successful.

Evaluations in the water and forestry sectors proved equally valuable for guiding future operations. A special evaluation analyzed participatory processes in six projects in three DMCs. The conclusion was that participatory approaches had added value to natural resource management project design, implementation, and operation and if applied effectively could increase ownership and sustainability. Environmental outcomes were almost universally positive. Comanagement was optimal for forests to ensure benefits to local communities. Participation offered major benefits in rural income and poverty reduction through improved resource planning and management and through promoting interaction of the poor with the community, enhancing their sense of worth and self-esteem.

ADB provided $73 million of assistance in loans and grants to build capacity in several water utilities in Pacific DMCs. The projects provided valuable lessons.

1. Setting up boards of directors with the right composition was a good first step but needed to be complemented with training.
   
2. Introducing performance-based principles did not ensure that the data were actually used in decision making and that actions were taken when performance fell short of targets.
   
3. Efforts to change operational procedures were effective only if the utility, its management, and its staff took full ownership of necessary changes.
   
4. All stakeholders had to support plans to incorporate to build politically acceptable solutions, to increase transparency, and to ensure sustained implementation of institutional changes.

In general the conclusion was that ADB’s capacity-building approach needed to change. Systematic institutional diagnostics should be undertaken jointly before a decision is made on the areas and direction of change. A medium-term strategy for capacity building should be developed, providing a framework for continuous assistance with interim milestones. Terms of reference of consultants need to be changed to give greater emphasis to capacity-building roles.

Country Assistance Program Evaluations

Papua New Guinea

The evaluation focused on three central issues: reforming the machinery of government, managing the dual economy, and delivering public services. Projects, the main assistance tool, were generally successful in producing outputs but did not generate significant or sustained impacts. Sector performance did not improve and contributions to promoting economic growth, particularly in rural areas, were overshadowed by declining international commodity prices, the Asian financial crisis, economic downturns, deteriorating law and order, and periodic droughts. Improvements to service delivery were lacking largely due to weaknesses in resource transfers and the absence of qualified staff. Projects tended to work in isolation rather than form an integrated country program with a clear strategic development goal.

There is a need to focus on a development agenda that creates jobs and addresses law and order, on managing debt and changing public resource management to avoid perpetual financial crises, and on building capacity in public administration.

Cambodia

ADB resumed operations in Cambodia in 1992 and has played a leading role in its recent development. Assistance was guided by three country strategy documents that emphasized poverty reduction; however, Cambodia remains poor with only limited economic opportunities in tourism and garment exports.

The strategies were generally responsive to the needs of the country but could have placed greater emphasis on improved governance and on more diverse private sector development. The program has adapted over time to incorporate operations with a greater impact on the poor.

Portfolio performance was good (above the ADB average) though overall there were some weaknesses. The substantial resources contributed for capacity building were focused on training specific individuals rather than on the effective functioning of institutions as a whole. Coordination of external assistance proved difficult in light of the complex government structure and inadequate harmonization of programs across agencies.

Implications for the future are: (i) ways need to be found to induce investment and to generate financial resources for the Government;(ii) some reduction in sector coverage would be beneficial with continued focus on governance issues;(iii) coordination of Government and external resources needs to be improved; and (iv)a broader approach to capacity building that will reduce dependence on external assistance should be adopted.

Technical Assistance Performance Audit Reports

In the 6 audit reports, 30 technical assistance activities were evaluated: 2 were rated highly successful, 17 successful, 7 partly successful, and 1 unsuccessful. Three were ongoing and therefore not rated.

Environmental Issues in the PRC

Seven technical assistance projects, both completed and ongoing, addressed needs in all key areas of environmental management and capacity building at the central, provincial, and municipal levels. Technical assistance to establish the Center for the Transfer of Environmentally Sound Technology, to improve environmental management in Shaanxi Province, to formulate provincial legislation on environmental protection and natural resources conservation, and to provide leadership training on urban environmental management in key cities achieved their objectives.

The evaluation identified some important lessons for each stage of a technical assistance cycle. Their essence is how best to meet the needs of the Government rather than perceived needs or the needs of ADB. Several projects were more supply driven than client oriented. One technical assistance project was rated highly successful, two successful, and one partly successful.

Pension Reform

Five technical assistance projects totaling $4.4 million were approved between November 1997 and December 1999 to promote pension reform in Bangladesh, India, Indonesia, Thailand, and Uzbekistan. Two—those in Bangladesh and Thailand—were attached to capital market development programs. A common feature was that pension reform was seen primarily through development of domestic capital markets rather than as a means of providing income for the vulnerable. The assumption that pension reform will promote long-term domestic capital formation sufficient for good social policies, particularly for the most vulnerable groups, should be revisited. The skills and knowledge of pension staff and coordination between finance and human development staff during pension reform should be improved. Distribution issues, particularly those involving women and disadvantaged groups, must be explicitly addressed in the design of any future pension reform.

Development Planning and Statistics in Cambodia

Technical assistance started in 1992 with the approval of a large grant to strengthen capacity in macroeconomic management. From 1992 to 1997, three other projects focusing on development planning and national statistics totaling $6.8 million were approved. The capacity built up in national statistics was impressive; from virtually zero, the National Institute of Statistics is now producing a large amount of widely used data and has sufficient staff to continue to do so. In development planning, the outcomes were somewhat less convincing owing mainly to the changed institutional role of the ministry concerned.

The main lessons learned included the following: (i) it may be appropriate to have smaller projects with fewer components; (ii) in transition economies where local capacity is limited, it may be useful to separate funds to substitute local capacity with external capacity from those for training to produce local capacity; (iii) if a ministry undergoes major organizational change, it may be useful to support a thorough institutional assessment to review the need for ongoing or new support.

Funding salaries and operational expenses through technical assistance should be replaced by regular financial support to priority mission groups. Institutional assessment of the ministry should be conducted and its role expanded through village, commune, district, and provincial databases. The public investment program should become an integral part of macrobudget planning rather than a means of mobilizing aid. Three projects were assessed successful and one partly successful.

Gender and Development in Cambodia

Three projects provided assistance over 8 years. Two emphasized data collection and policy formulation while the third was for project preparation and also provided assistance for capacity building and promoting women’s employment. The main lessons learned were (i) without mainstreaming, gender issues integral to poverty reduction will continue to be marginalized; (ii) revising technical assistance design could be successful if there were early identification of key constraints, early objective analysis of achievable goals, and efficient, participatory management; and (iii) achieving objectives must be sensitive to the political and institutional environment if failure is to be averted.

The ministry should be reorganized to reinforce consideration of gender concerns. The policy on women should be reviewed, updated, and published, and there should be greater focus on gender training and mainstreaming. All technical assistance projects achieved planned output in terms of reports and had considerable success in achieving gender outcomes. One project was partly successful because it failed to take account of the capacity of the executing agency, and two were successful.

Institutional Strengthening and Policy Support to the Power Sector in Bhutan

The three projects had two design weaknesses: the time given for consulting input was inadequate and sequencing of activities was not taken into account. Lessons from the evaluation included (i) before developing complex new accounting software, the Government should agree to replace the existing system; (ii) more attention needs to be paid to domestic characteristics and conditions when technical assistance is formulated; and (iii) the time allowed for counterpart feedback must be adequate.

The Government and ADB should agree to an acceptable level of increase in domestic prices for electricity, and indicators other than the standard financial covenants should be developed to gauge the effectiveness of sector reforms. One project was rated successful and two partly successful.

Water Supply and Sanitation

Eight projects for improvements in water supply and sanitation included three advisory technical assistance projects in the Lao People’s Democratic Republic (Lao PDR) and five in Viet Nam. Lessons learned included the following: (i) effects could be cumulative and positive in institutions that receive continued advice over time even if individual projects were unsuccessful; (ii) health and hygiene programs associated with water supply and sanitation are more effective when implemented by an experienced professional organization; (iii) where an advisory technical assistance project has several interrelated components, consultants’ inputs should be organized so that outputs can be properly coordinated; and (iv) coordination and timing between associated projects and advisory technical assistance is important. One project was assessed highly successful, six successful, and one partly successful.

Annual Report on Loan and Technical Assistance

The report assesses performance in the previous year. In 2002, the public sector portfolio improved markedly over 2001. The proportion of projects at risk of performing below expectations dropped from 26% in 2001 to 16% in 2002. Success in moving projects out of the “at risk” group is measured by a proactivity index; this improved from 39% in 2000, to 59% in 2001 and 67% in 2002. Though overall performance improved, problems were identified such as the time taken for loans to become effective, the high proportion of loans requiring extensions, and the proportion of loan cancellations in some DMCs. The proportion of technical assistance and project completion reports actually carried out remained disappointing. The performance of the private sector portfolio was little changed in 2002. Delegating project administration to resident missions and doing so earlier in implementation will improve performance.

Back Mobilizing Human Resources and Budget	Next Strengthening Accountability