This research and development technical assistance (TA) for Selected Evaluation Studies for 2014 supports the 2014-2016 program cycle, focusing on studies due in 2015 and 2016. It will fund select evaluations and related outreach activities from both the derived operational (i.e., country and sector program evaluations) and knowledge (i.e., thematic, corporate and impact evaluations, topical working papers) programs of the Independent Evaluation Department (IED). These include, but are not limited to:
1. Country assistance program evaluations (CAPEs) and validations of country partnership strategy final reviews. The TA will fund at least two CAPEs (second-generation CAPEs for Papua New Guinea and the Peoples Republic of China), and validations of at least three country partnership strategy (CPS) final reviews (Bangladesh, Lao Peoples Democratic Republic, and Timor Leste). Each CAPE will evaluate ADB support to the country and cover ADB strategy, policy dialogue, completed and ongoing projects and TA, and ADB performance. It will provide ADB and the developing member country (DMC) with an evaluative basis of use for designing the future CPS. When appropriate, country sector assessments will provide inputs during the preparation of a CAPE. Each sector assessment will evaluate ADB performance within a given sector in the country, focusing again on strategy, policy dialogue, ongoing and completed projects and TA, and performance.
2. Review of Asian Development Fund (ADF) Operations. The thematic evaluation study (TES) will review the commitments made under the ADF X and XI replenishments and assess the relevance, responsiveness, and results of ADF operations over the said period. It will comprise a desk review of selected project documents, evaluation reports, and pertinent literature. DMC consultations may be undertaken in a range of countries, a perception survey held, along with country portfolio reviews covering all five regions.
3. Environmentally Sustainable Growth. The TES will review the design and implementation of ADB's 2002 Environment Policy and subsequent directional documents. It will assess the responsiveness, results, and relevance of the policy and environmental elements of Strategy 2020 (S2020), to draw out lessons for future operations. The study will initially undertake a desk review of project documents, evaluation reports, and pertinent literature. It will also take stock of prior environmental evaluations to learn from best practice of development partners. Perception surveys, country case studies of important environmental projects, and key informant interviews of regional DMC stakeholders and other development partners undertaking environment-related work will also be conducted to elicit views about ADB's achievements.
4. Effectiveness of ADB's Partnerships. The TES will evaluate the effectiveness of ADB partnerships through a review of such agreements, co-financing partnership arrangements, and related documents. It will analyze projects and TAs, and conduct in-depth interviews and a stakeholders survey. The study will cover at least 2 countries/regions (tentatively the Greater Mekong and Central Asian regions). The study will first examine the type of partnerships that ADB signed, and assess their relevance, efficiency and effectiveness. Second, it will look into various financing partnership agreements ADB is handling, administering, or participating in, on behalf of development partners (e.g., project-specific co-financing, framework arrangements, trust funds, financing partnership facilities) in terms of relevance, effectiveness, and efficiency of procedures. Third, it will look at progress made in co-financing operations as part of partnerships with other agencies, e.g., international financial institutions, commercial and private sector partners such as banks. Lastly, it will administer a survey questionnaire to resident missions and headquarters staff (country economists).
5. ADB Risk Management Practices for Non-sovereign Operations (NSOs). The TES will assess the relevance, efficiency, and effectiveness of ADB's risk management policies (e.g., exposure limits), systems and processes for NSO to identify any required enhancements that can facilitate the development effectiveness and sustainability of NSO. It is currently envisaged that the TES will have a special focus on how ADB manages the triangle of risk, reward, and development impact, in particular, how it prioritizes these objectives and handles potential trade-offs and whether it achieves an optimal mix. The review will look at ex ante strategies and policies, their application, as well as results based on a review of completed transactions. ADB risk management standards and practices will be compared to good international practices, especially those utilized by comparable multilateral financial institutions. Relevant findings of a planned internal audit of ADB's risk management and NSO pricing by the Office of Auditor General will be utilized as appropriate.
6. Impact evaluations. The TA will support at least one impact evaluation (most likely related to energy). The study will use quasi-experimental designs and emphasize validity and reliability of evaluation results. It will involve: (i) literature review, (ii) secondary data analysis and identification of counterfactuals, (iii) primary data collection (usually a large questionnaire survey), and (iv) data analyses and report preparation. The evaluation will build on data collected during the project's implementation and identify relevant comparison groups (counterfactuals) based on questionnaires, key informant interviews, and secondary data.
7. Other sector and thematic assessments, and topical working papers. The TA will support selected country, sector, and/or thematic assessments needed to feed into high-level evaluations in the IED work program. This will also include selective evaluative research on high impact topics of specific interest and relevance to DMCs.
8. Evaluation outreach, Knowledge sharing, and Partnership initiatives. The TA will support interactive knowledge-sharing events (i.e., trainings, seminars and conferences) on its completed major evaluations. These will be a venue for learning and showcasing of evaluation products; informing stakeholders of formative, summative, and prospective evaluation approaches, and findings and lessons. The learning events are to be held in selected DMCs in partnership with national think tanks, evaluation societies, public and/or nonprofit research institutions, and other partners. From each evaluation report or learning event, IED may also develop derivative communication and outreach products (such as podcasts, multimedia products, and synopses) to promote further learning. Other dissemination efforts may include preparation and publishing of syntheses, briefs, and occasional papers tailored to widen readership, and the use of new mediums for knowledge sharing (e.g., online readers, tablets) to extend the reach of IED products.
|Project Rationale and Linkage to Country/Regional Strategy
The yearly work program of IED focuses on higher-level evaluations covering multiple countries, sectors, and themes to complement project evaluations. These evaluations give a more strategic perspective to ensure that ADB's directions and operations are captured and supported by more intense knowledge and outreach activities to maximize usefulness and impact. The Board of Directors of the Asian Development Bank (ADB) approved the IED's 2014-2016 work program, with effect from 11 December 2013.
ADB has made significant efforts to track and achieve the goals of Strategy 2020 (S2020). This TA will help IED fulfill its mandate, role, and functions (as guided by ADB's evaluation policy) in promoting learning and accountability by: (i) supplying evaluative feedback on the effectiveness of ADB operations to gauge the relevance and effectiveness of S2020 in several important thematic and operational areas, and (ii) facilitating the sharing and use of evaluation knowledge.
1. Results of ADF operations. IED will conduct real time evaluation on both recently completed and ongoing replenishment of the ADF, as an input to its next replenishment. Management asked IED to conduct an evaluation of ADF X and XI operations in 2015, to feed into the ADF XII replenishment process. It succeeds a 2011 thematic evaluation of ADF VIII and ADF IX (2001-2008) and their development effectiveness. This round of evaluation is of special interest given rapid development in Asia, and the likelihood of further graduations of countries out of ADF. Other areas which may be looked into include (i) use of ADF by relatively new modalities such as the multitranche financing facility and results-based lending; and (ii) the importance of addressing inequality in poor and middle income countries, and the potential need to continue ADF operations in for particular groups and areas in those countries.
2. Supporting environmentally sustainable growth. This evaluation will come more than a decade after the approval of ADB's environment policy in 2002, 7 years after S2020's first formulation of environmentally sustainable growth as a key strategic agenda, 6 years after the energy policy of 2009, 3-5 years after several infrastructure sector operational plans were approved with environmental guidance, and 2 years after the issuance of ADB's new Environment Operational Directions (2013-2020). ADB's Development Effectiveness Review series over the past five years reports a gradual increase in the number of operations with environmental themes beyond original targets, and their good performance. But still, issues of environmental degradation, waste, urbanization, deforestation, flooding, environmental protection, and climate change have become starker since the approval of S2020 in 2008.
3. Pursuing effective partnerships. S2020 identified this as a key objective in order to mobilize resources, leverage knowledge, and meet the need for highly specialized development projects, with special attention for non-core areas of operations. In past evaluations, independent evaluation has argued that ADB should ensure that its work is complementary to that of its development partners, realizes synergies, and achieves well rounded development results. All of this calls for an increasing reliance on effective partnerships. This evaluation will assess how the S2020 aspirations in this area have been operationalized, how effective they have been, and how much scope there is for untapped partnerships. Good practices in existing partnerships in both ADB and other aid agencies will be identified. Lessons and recommendations may also lead to confirmations or proposals to adjust ADB's overall programmatic approach and level of specialization in operational areas, with a view to improve development effectiveness. The review will also aim to provide insights on the costs and benefits of such arrangements and suggestions for improvement.
4. Strengthening risk management practices for NSOs. During its midterm review of S2020, ADB management stated that Private Sector Operations Department should seek investment opportunities that carry higher development returns while managing higher risks. Despite recent improvements in the risk management of NSOs, Management continues to identify the need for further strengthening ADB's risk management systems and reviewing existing country and sector exposure limits. Also, without taking equity risk (and specifically, early development equity risk), ADB will remain limited to financing already bankable projects rather than contributing to taking many more projects from concept stage to the bankable stage. This implies that to add value, ADB will have to take higher risks than private sector parties. To maintain its financial viability, ADB needs to be adequately compensated for its risks. ADB's current pricing of NSOs is directly linked to credit risk ratings, which some stakeholders consider to not always be in line with market conditions.
5. Improving evaluation utilization. Like other evaluation units of multilateral and bilateral development institutions, IED is continually challenged to optimize the use of evaluation. A well-targeted outreach to relevant decision makers can make evaluations more influential. Likewise, if evaluations are shared proactively, they will benefit from enhanced credibility and provide better incentives to act upon them. IED has made steady improvements in identifying and generating evaluation knowledge and has moved to strengthen sharing and use. If there is a good outreach plan, and resources are available, IED can efficiently and effectively communicate results to multiple audiences. Development agencies value the role of participatory communication channels for learning and new communication approaches are needed (for instance, interactive knowledge sharing and learning events that communicate specific aspects of evaluations with emphasis on interpretation, knowledge exchange, and potential application).