Implementing Impact Evaluation at ADB | Asian Development Bank

Implementing Impact Evaluation at ADB

Impact evaluation provides empirical evidence regarding the outcomes or results attributable to development projects. Evidence on what works and does not work is important in improving accountability and learning from the development interventions. Before–after comparisons or comparisons between areas with and without projects often lead to false conclusions about project effects, because the differences observed are due to many other factors than the project alone. Impact evaluation offers a set of tools to separate the consequences of projects from other drivers of change, in order to understand what changes are truly attributable to projects. Impact evaluation has a variety of applications: from understanding which approach to an intervention is most effective, to ascertaining causal relationships that will effectively reduce poverty. All impact evaluation techniques are premised on the concept of a “counterfactual,” which is either identified as a randomly selected “control group” and is approximated econometrically.

Implementing impact evaluation at ADB

Although impact evaluation is a growing field, there remains little evidence about the contribution of individual infrastructure projects and programs, which receive a majority of ADB’s finance, to development goals. Moreover, there are many choices and assumptions that underpin the designs of projects in all sectors, of which few are empirically tested. More systematic measurement of project/program outcomes through impact evaluation supports evidence-based policymaking and helps development partners design and implement projects more effectively. ADB recognizes that impact evaluation will help ADB and its developing member countries (DMCs) allocate limited resources more efficiently, guide the design of more effective interventions and enable evidence-based policy.

In line with its priority on managing for development results and mainstreaming impact evaluation in the project cycle, ADB approved the following technical assistance projects—TA 7680: Implementing Impact Evaluation at ADB (approved on 7 December 2010) and Cluster TA 0012: Developing Impact Evaluation Methodologies, Approaches, and Capacities in Selected Developing Member Countries (DMCs) on 7 December 2010 and 14 January 2013, respectively. Cluster TA 0012 was an offshoot of TA 7680 to support completion of IE studies and respond to the need to sustain and continuously mainstream impact evaluation in ADB operations. Cluster TA 0012 finances the conduct of IE studies by the regional departments, and the development of IE methodologies as well as the conduct of IE awareness-raising activities through seminars, learning sessions, and capacity building workshops for ADB staff and DMC officials, and maintenance and regular updating of an IE Portal.

Completed impact evaluation studies

India: Enhancing Energy-Based Livelihood for Women Micro-Entrepreneurs

The evaluation study highlights the impacts of the trainings given to women entrepreneurs and women members of self-help groups (SHGs) in the fields of gender and energy, effective use of electricity based livelihoods, livelihood skills. Economic and social impacts (impacts on: income of women, on work participation, loans and saving habits, time-use, mobility, decision making, perception of safety, health, children’s education) were estimated by comparing trained women (treatment group) with nontrained women (control group) using propensity score matching and regression.

Mongolia: Medicard and Food Stamp Programs

The evaluation study identified and quantified the impact of two innovative projects for the extreme poor in Mongolia, the Health Care Subsidy program (Medicard), and Food Stamp Program. Both programs are proposed to be temporary but are addressing two major gaps in the Mongolian social protection system, which lacks adequate safety nets for dealing with shocks (economic, food price increases, other).

Completed baseline

China, People's Republic of: Ningxia Liupanshan Poverty Reduction Rural Road Development Project

The evaluation study developed methodology for measuring the impact of the Loan Project on poverty reduction in the Liupanshan target area. It involved the conduct of a baseline survey to collect comparative data before the road projects are constructed and an endline survey to analyse the project outcome and impacts, as well as necessary follow up surveys to evaluate longer-term impacts.

Georgia: Sustainable Urban Transport Investment Program Tranche 1

The evaluation study focused on the Tbilisi metro extension and the benefits of urban infrastructure development to households and businesses, as well as for women and vulnerable groups to help improve project selection and design so that they can benefit as much as possible from future investments.

Mongolia: Pilot Models to Improve School Dormitory Environment and Services

The evaluation study will establish a baseline to measure impacts of the project interventions which will take place from October 2016 to October 2018. The interventions include reform of the school dormitory system in Mongolia to ensure access to quality education for all students, particularly those from herder families who live a nomadic life.

Nepal: Raising Incomes of Small and Medium Farmers

The evaluation study developed and identified optimal strategies that can help farmers increase their incomes and reduce vulnerability to poverty through diversification to crops into high value commodities and creating market linkage through farmer cooperatives.

Papua New Guinea: PNG Financial Literacy Program

The evaluation study provided a snapshot of the existing level of financial literacy, financial awareness, use of financial products, among others; and validated that on the average there is no significant difference between treatment and control wards and that any differences in the end line survey will be due to the Financial Literacy Program.

Papua New Guinea: Road Access on Poverty and Other Social Welfare Indicators in Rural Highlands Communities

The evaluation study measured the impact of road development on household welfare in rural PNG over the period between 1996 and 2010, using two cross-sectional household surveys and corresponding road maps. Assessment covered the entire highlands region with focus on rural areas where approximately 85% of the population live.

Philippines: Poverty Graduation Program

The evaluation study assessed whether the existing convergence efforts, specifically of the Conditional Cash Transfer (CCT) program and Sustainable Livelihood Program (SLP) wll more effectively increase the well-being of the extreme poor and make them more economically active.

Sri Lanka: Eastern and North Central Provincial Road Project

The evaluation study assessed the impact of road projects on poverty, economic activities, small and medium enterprises, access to education and educational outcomes, and mechanism by which road projects contribute to improved living conditions.

Ongoing studies

Bangladesh: Effects of Modernized Irrigation Infrastructure, Volumetric Water Pricing and Water Saving Technology on Water Use Efficiency

The evaluation study will assess the effects of modernized irrigation infrastructure, volumetric water pricing and water saving technology on water use efficiency. Outcomes to be tested include reduced periods of water deficit, improved yield and land productivity; increased water use efficiency, and enhanced equity in agricultural incomes.

Bangladesh: Short-Term Training Programs by Public and Private Institutions (Baseline, Midline and Endline Survey)

The impact evaluation will study the effectiveness of the short-term training interventions provided by public and private training providers in major manufacturing industries in Bangladesh (e.g., readymade garment) and their impact on employment and earnings. The IE will also shed lights on varying effects of Skills for Employment Investment Program (SEIP) on different demographic groups, especially on women and socially disadvantaged groups.

Bangladesh: Child Nutrition Education in Selected Slums of Dhaka and Gazipur (Baseline)

The impact evaluation will study the impact of the service delivery package to support child nutrition under the urban primary health care service delivery project (UPHCSDP). It will assess the treatment effect of project community-based satellite workers carrying out monthly home visits to counsel and promote infant and young child feeding practices and basic hygiene.

China, People's Republic of: TVET Sector Reforms in the Guangzxi Zhuang Autonomous Region

The evaluation study will set a baseline for an impact evaluation and will provide information on the current state of learning in vocational secondary schools in Guangxi through a ready-to-use database. It will focus on secondary vocational schools, and will assess how interventions aimed at improving the learning environment affect graduate outcomes.

Fiji: Secured Transactions Registry (Baseline survey)

The impact evaluation will test the impacts of introducing Secured Transactions Registry (STF) in the use of credits by businesses whether this will lead them to expand their operations, and increase their productivity and profitability.

India: Rural Road Investment Program in the States of Chhattisgarh and Madhya Pradesh

The evaluation study involves the conduct of a baseline survey before the implementation of the loan project based on the established evaluation methodology. It will focus on rural roads covered by Second Rural Connectivity Investment Program (RCIP2) where existing roads are upgraded through paving road surface to “all weather” condition.

India: Enhancing Roof-top Solar Power Business Using Digital Technology

The evaluation study will assess if using digital technology would enhance the productivity and profitability of Simpa Network Inc. business by analyzing if the introduction of IT (i.e., smart phones) would improve customer acquisition and retention under off-grid roof-top solar power business in rural India.

Nepal: Support to Collective Action for Agricultural Value Chain Development in Nepal

The evaluation study will measure the effects of projects—Raising Incomes of Small and Medium Farmers Project (RISMFP), World Bank's Agricultural Commercialization and Trade (PACT), IFAD's High Value Agriculture Project (HVAP)—on a range of outcomes among farmer groups/cooperatives such as agricultural net revenues and household incomes/wages; participation in off-farm activities; and agricultural production practices.

Pakistan: Transport for Employment in Lahore, Pakistan

The evaluation study will quantify the effect of urban transport on access to employment for women and men in Lahore, Pakistan. Results of the study will be useful to policy makers and national urban transport planners in addressing challenges affecting the labor market.

Pakistan: Warehousing and Transport Services for Rice Farmers in Hafizabad district, Punjab

The evaluation study will measure the size and distribution of benefits from providing small- and medium-scale rice farmers in Hafizabad district, Punjab access to warehousing and transportation services. The interventions will provide them opportunity to engage in a more organized value chain that would lead to increased competition in the local economies through the elimination of middlemen or the reliance on ad-hoc markets, and better prices for farmers.

Papua New Guinea: PNG Financial Literacy Program (Phase 2)

The evaluation study will assess the effectiveness of financial literacy programs in supporting financial inclusions in remote parts of PNG; how the knowledge gained has influenced financial decision making, including its effects on savings and budgeting behaviors which are likely to impact upon both personal and business activities.

Philippines: Employment Facilitation for Inclusive Growth

The evaluation study will measure the impact of MyFirstJob program on young persons’ integration into the labor market.  Using randomized IE, control and treatment groups are selected through a lottery of eligible applicants to the MyFirstJob program.

Events

2015 Conference on Impact Evaluation for Development Effectiveness, 10–11 November 2015

The event featured presentation and discussion of use of impact evaluation (IE) in policy making; sharing of IE studies of programs and projects in social sector, rural and agriculture, infrastructure; and training on impact evaluation. The training focused on key concepts, methods and approaches, and managing IE in terms of time, budget, and governance.

2014 International Workshop and Conference: Making Impact Evaluation Matter, 1–5 September 2014

A five-day back to back event of workshop and conference organized by ADB and the International Initiative for Impact Evaluation (3ie). It featured parallel and plenary sessions on various topics on impact evaluation, focusing on the theme, "Making Impact Evaluation Matter: Better Evidence for Effective Policies and Programmes. The event gathered researchers, policymakers and practitioners for exchanging ideas on impact evaluations of development programmes in particular sectors like education, health, infrastructure, climate change, agriculture, social protection and community-driven development.

2012 Conference on Impact Evaluation: Methods, Practices, and Lessons, July 2012

The conference brought together impact evaluation experts from the region and around the globe to discuss best practices in conducting impact evaluation, how methodologies are applied in sector studies, and the lessons learned from impact evaluation studies. The event featured discussions on evaluating impacts on health and education as well as presentation from regional departments of their impact evaluation studies.