Regional : Developing Impact Evaluation Methodologies, Approaches, and Capacities in Selected Developing Member Countries (Cluster TA)

Public sector management / Economic affairs management

Impact evaluation (IE) is a cross-cutting theme that is relevant in all countries, sectors, and regions. Conducting impact evaluation for strategically selected interventions will help improve relevance and effectiveness of country and regional partnership strategies. Properly implemented impact evaluations will help in the proper selection of a strategic mix of effective interventions. By implementing impact evaluations of development interventions in the Asia-Pacific region, the proposed cluster TA will contribute to the creation of knowledge on what works in development, how it works, and why.

The proposed cluster TA will comprise three sub-projects with each sub-project covering 4 to 5 impact evaluation studies, or a total of 12 to 15 studies distributed over the five regional departments. By the end of the proposed TA, impact evaluation will have been mainstreamed in ADB operations, i.e., it is envisioned that impact evaluation is incorporated in the design of ADB's investment projects, especially in project preparatory technical assistance (PPTA) projects.

Sub-projects

Each subproject will include 4 to 5 impact evaluation studies and will comprise two main components, namely, (a) the conduct of IE studies applying various methodologies and survey instruments, and (b) the development of impact evaluation methodologies, approaches, and capacities in both ADB and the DMCs.

(a) Conduct of IE studies

The IE studies component will cover support to conduct of a number of IE studies while ensuring that at least one study will be supported in each of the five regions (Central and West Asia, East Asia, the Pacific, South Asia, and Southeast Asia). Activities under the individual studies may include design of the study methodology and survey instruments; conduct of baseline, mid-line and end-line surveys; and production of the study reports. The proposed TA will endeavor to support the conduct of full IE studies from beginning to end within the timeframe of the proposed TA. However, as learned from the experience of TA 7680, the TA may support key phases of individual studies; for example, as to be approved by the IE Committee, only the design and conduct of the baseline survey may be implemented under the proposed TA, while the mid-line and end-line surveys can be financed through separate initiatives. Through this component, the TA will provide direct technical support by individual and firm consultants, and ERD specialists to individual studies by advising on study design, methodology, and application in all phases of the study. This technical support will also include partnerships with renowned IE research institutes (e.g., International Initiative for Impact Evaluation or the Jameel Lateef Poverty Action Lab) to provide expertise, as well as ground-level support for DMCs in the design of IE frameworks and methodologies (e.g., visits to project sites, pre-testing of survey instruments, and other fieldwork activities).

(b) Development of impact evaluation methodologies, approaches, and capacities

The second component on development of IE-methodologies, approaches and capacities will entail a number of activities including (i) review of IE methodologies and adoption of available techniques and tools for sector and theme specific studies; (ii) conduct of pilot studies to test sector- or theme-specific techniques and tools; (iii) IE awareness raising and capacity building for DMC officials and project staff; (iv) establishment of a central IE data source containing information on project baseline, mid-line and end-line surveys (e.g., questionnaires, sampling methodology, interviewer manuals, field data, etc.); and (v) conduct of knowledge-sharing activities (i.e., regional conferences, where IE studies conducted will be presented; workshops and seminars, where experts will be invited to share methods, practices and lessons on IE) and dissemination of IE studies (publishing IE studies for distribution to ADB and government counterparts, and uploading studies in the ADB website).

Bilateral and multilateral development agencies spend more than $30 billion each year on social development programs, with developing countries spending another $300 billion annually. However, the development effectiveness of this spending is not always clear, and evidence on their impact, if they are available, is scant. In particular, it is not clearly known how much of Asia's recent improvements in poverty reduction and human capital outcomes have been due to development spending and how much are trickle-down effects of the region's rapid economic growth. This lack of information limits the ability of policy makers and development partners to further improve effectiveness of the development spending through proper design of programs and projects.

ADB has supported its developing member countries (DMCs) in designing, financing, and implementing project and policy interventions aimed at fostering economic development and reducing poverty in the region. ADB's economic sector work, cost benefit analysis and past effort at benefit monitoring and evaluation, as well as design and monitoring frameworks and social safeguard requirements, collect information that provide a basis for identifying project performance ex-ante and ex-post. While these are useful exercises in designing projects and monitoring their performance, they do not necessarily identify the actual impacts of ADB's development support. Although it may not be possible to identify specific impacts of development spending at the macro level, it is possible to do so at the project level through the use of rigorous impact evaluation (IE) studies.

Thus, mainstreaming IE in ADB operations is highly beneficial not only for the Bank but also, and more importantly, for DMCs. Gaining accuracy in the evaluation of interventions will permit policymakers in DMCs and ADB to anticipate the expected consequences of these interventions. In this way, IE will allow DMCs and ADB to allocate limited resources more efficiently, guide the design of more effective interventions, and enhance evidence-based policy dialogue.

Improved development effectiveness of ADB-designed projects responsive to the development needs of DMCs

Impact evaluation is mainstreamed in ADB operations

1. Impact evaluation studies adopting appropriate methodology with effective technical support

2. Impact evaluation awareness-raising seminars and capacity building workshops for ADB staff and DMC officials

3. Reports and dissemination workshops on lessons learned from review of methodologies and conduct of pilot studies

1) Seventeen impact evaluation studies are being conducted, covering 8 sectors (agriculture and natural resources, education, energy, employment, finance, health and social protection, transport and ICT, water supply and other municipal infrastructure and services) across 10 DMCs (Bangladesh, India, Mongolia, Nepal, Pakistan, PRC, Philippines, Papua New Guinea, Sri Lanka and Viet Nam). Of the 17 studies, 1 was completed, 5 with completed baseline, and the rest are ongoing at various stages of implementation.

2.a) Under Subproject 1, four conferences and workshops on impact evaluation methodologies and practices and their applications on various sectors were conducted from 2012 to 2015. One seminar was conducted where the results and methodologies of the TA-funded and RD-implemented IE study (SARD-IND: Enhancing Energy-based Livelihoods for Women Micro-entrepreneurs) were presented.

2.b) Under Subproject 2, eight seminars were conducted where methodologies and results of impact evaluation studies in various sectors (e.g., energy, finance, poverty reduction and social development, social protection, transport, climate change and disaster risk management, and rural development and food security) were presented. Resource persons include reputable experts in their respective field.

2.c) An impact evaluation guidebook, intended as a practical resource for those interested in planning impact evaluation studies, was published and released in December 2017.

3.a) The Impact Evaluation (IE) portal, a repository of IE information, materials, and activities within ADB, was launched in 2017 and continuously updated and improved since then. It is intended for bankwide dissemination of IE materials (approved proposals, status of implementation, financial status, evaluation reports, survey instruments, etc.) and was made possible by a cross-departmental undertaking led by ERCD.

3.b) Policy brief was disseminated for one IE study (CWRD-PAK: Transport for Employment in Lahore, Pakistan) titled "Overcoming barriers to women's mobility".

3.c) Final report for a completed IE study (SARD-IND: Enhancing Energy-based Livelihoods for Women Micro-entrepreneurs) was published and disseminated.

3.d) Two pilot studies that analyze the impact of: (i) introduction of IT on customer acquisition and retention under off-grid roof-top solar power business in rural India; and (ii) modernized irrigation infrastructure, volumetric water pricing and water saving technology on water use efficiency; are being implemented through close coordination between ERCD and PSOD and SARD, respectively.

Each study team may work in close collaboration with the concerned government agencies, both oversight agencies and project/intervention implementation units.

This technical support will also include partnerships with renowned IE research institutes (e.g., International Initiative for Impact Evaluation or the Jameel Lateef Poverty Action Lab) to provide expertise, as well as ground-level support for DMCs in the design of IE frameworks and methodologies.