Building Better Evaluations by Capitalizing on Lessons Learned: Evaluation HEAD-lines

Video | 22 June 2020

In this episode of Evaluation HEAD-lines we discuss how evaluators are building better evaluations by capitalizing on lessons learned from past crises. Different multilateral development banks work in different country contexts, have different challenges, thus different learnings. Saleha Waseem of Independent Evaluations at the Asian Development Bank speaks to Roland Michelitsch, Evaluator General at the African Development Bank (AfDB) Group and Anasse Aissami, Director, Operations Evaluation Department, Islamic Development Bank (IsDB) to discuss how their experiences from the past are helping shape evaluations today and possibly preparing them for the future.

Evaluation HEAD-lines is an interview series where we speak with heads of independent evaluation offices to understand how they are adapting their evaluations to respond effectively to the COVID 19 crisis. The COVID-19 pandemic will have a profound impact on the methods and key indicators used for evaluation. Evaluators will need to rethink their evaluation plans, designs, and methods.

Transcript

Saleha Waseem: Hello and welcome to Evaluation HEAD-lines our new series where we speak with heads of independent evaluation offices to understand how they are adapting their evaluations to respond to the COVID 19 crisis.

Different multilateral development banks work in different country contexts, have different challenges, thus different learnings. Today we will discuss how experiences and lessons learnt are helping shape what could be a long road ahead with COVID-19.

Joining me today is Roland Michelitsch, Evaluator General at the African Development Bank (AfDB) Group and Anasse Aissami, Director, Operations Evaluation Department, Islamic Development Bank (IsDB)

Roland, if we speak about MDB response to the COVID 19 crisis, AfDB is in a unique position with immense learning from responses to past crisis and upheavals, not least the crisis response to the Ebola epidemic in 2014. Can you put the current crisis in perspective to the past crisis’s that AfDB has responded to?

Roland Michelitsch: Thank you Saleha. Africa unfortunately has been affected by many crisis…. when you put things in perspective…so far from COVID 19 about 5 thousand people have died. 380 thousand people die each year from Malaria…. a preventable disease…and die of hunger. But COVID 19 is not just a health crisis…it is a financial and social crisis too. (Elaborates)

Saleha Waseem: How is AfDB using its experience to respond effectively to the present crisis and how is your office of independent evaluation supporting AfDB respond effectively?

First order of business: Ensure our colleagues and their families are safe and that they can work.

Huge impact, affecting all our 54 member countries. Not just health effects – so far fortunately relatively minor in Africa – but also huge economic effects (commodity prices, fiscal balances, debt), social effects (lost income – most vulnerable can’t “work from home”) and the risk that existing problems (e.g., food crises, health problems – malaria, TB, HIV-AIDS) will be exacerbated.

AfDB Operations changed: $3B social bond; $10B crisis response facility – expected to be some 70% of this year’s program; support for WHO in Africa.

Evaluation and IDEV must follow operations. AfDB is in a unique position to learn from responses to past crisis and upheavals, not least the crisis response to the Ebola epidemic in 2014. Therefore we must: (1) Focus on lessons learned – from past crisis responses (a) Ebola crisis; (b) Crisis-response budged support; (2) Focus on results measurement framework for crisis response – so that eventually will be able to evaluate it; (3) Coordination with others – on lessons, results frameworks and potential future evaluations.

Focus on how to best measure the results of the Bank’s response to the crisis – its overall support, and its individual operations in specific countries, to ensure we can later evaluate the crisis response. Careful to distinguish role of independent evaluation (we can help with building a results framework, but mgmt. needs to implement it).

Saleha Waseem: Anasse, IsDB works in a very diversified constituency across 4 different Continents. What have been your specific challenges in responding to the COVID 19 crisis?

Anasse Aissami: As you know IsDB has a very diversified constituency of 57 MCs in 4 different Continents. This is by itself a challenge as the context of our MCs is very different.

A key characteristic of development evaluation is that its results should be evidence based. It entails seeing results of interventions on the ground (and up to outcomes level). In this regard, engaging with end-beneficiaries is essential.

The main challenge we have faced because of the COVID-19 pandemic is the inability to conduct site and field visits to collect evidence. This challenge is of paramount importance as it disrupts the vital direct interaction with our stakeholders in the evaluation process. These interactions provide a source of opinions and insights on issues and lessons, as well as the sustainability of IsDB interventions. 

Lessons Learned & Input to Project/Program Design

Amid the few positive outcomes of this crisis is that the Islamic Development Bank Group leadership has increased its demand for evidence and lessons from our previous interventions. This, also, provided ’Evaluation’ the opportunity to share with the operations what has worked, what did not, and what we have learned, and more importantly how we can do better moving forward.

Saleha Waseem: Roland, are there some early lessons that you can draw on the role of independent evaluation in time of crisis? What does it mean for the future?

Roland Michelitsch:    Need to be ready with “lessons from crises” – the next one will almost certainly come (and we will be better prepared). Lessons include both (1) organizational lessons (how to work better in a crisis) and (2) strategic / operational (where should an institution focus during a crisis; how can it best implement its response). But we also need to remember that the next crisis will be somewhat different, and we leave flexibility.

-Preparations for crisis response are important: (1) organizational (drills, procedures for WFH); and (2) simplified processing for CRF projects. Ideally, they should be prepared during good times.

-Ideally, a results framework should already be part of the approval of the CRF – not follow later. Responding quickly in a crisis is crucial, and establishing such a  framework very quickly speeds up later processing.

-We are still learning lessons on how better to conduct evaluations. Collaboration with other institutions and learning from them will be important, as is donor coordination, particularly at the country level. A key issue is going to be attribution – that’s extremely difficult during the current crisis. But let us first focus on an effective, coherent, and coordinated – and successful – response. That is right now more important than who gets the credit, even though of course we will want to know who made what contributions.

Conclusion: We hope that many lessons will be learned during this crisis, which requires good M&E being built into the projects and CRFs … and that they will be applied in the next crisis.

Saleha Waseem: Anasse, we spoke about the challenges faced by your office and yes you did touch upon some positive outcomes too. How are you addressing these challenges and harnessing the opportunities?

Anasse Aissami: Data Collection

Saleha, we have put on hold all evaluation efforts requiring primary data until travel restrictions are lifted. In its stead, the Evaluation in IsDB is focusing more on desk-based reviews. However, we have considered using virtual data collection tools for few cases. Lesson Learned and Input to Project/Program Design.

To meet the increasing demand for evidence and lessons for COVID-19 initiatives, we have prepared a synthesis of Lessons Learned on the IsDB’s past experiences over the last five years. These lessons provided astute guidance to operations in the design of impactful interventions in MCs. One such example is from the Bank’s emergency response to the Ebola Crisis in West Africa. In addition, we took the initiative to compile lessons learned from other sister institutions.

We have intensified the use of online tools to disseminate evaluation knowledge. We have, also, conducted virtual dissemination forums for high-level evaluations.

Beyond the immediate crisis response, the COVID-19 pandemic should be the impetus to sustain the gains and accelerate implementation of long-overdue results-based measures to set the world on a more sustainable development path and make the global economy more resilient to future shocks. That’s why the IsDB Response Package has been designed under the “3Rs” Framework: ‘Respond, Restore, and Restart’. These 3 Tracks include the short, medium and long term.

Knowledge Dissemination and Capacity Building

The current crisis might be a rude awakening that the ambition expressed in the Global Goals is unprecedented and exceedingly ambitious; but the opportunity to ‘Transforming Our World’ is still there. The international development community needs to work more closely and more in synergy than ever to change its approach in global governance, to help resolve the pressing challenges of economic development, poverty eradication, social justice, and global environmental protection.

Saleha Waseem: Thank you both for joining us today. That was indeed informative. We will see you again soon with another topic and newer panelists etc.