Opening message by Marvin Taylor-Dormond, Director-General, Independent Evaluation Department, Asian Development Bank at the 2019 Asian Evaluation Week on 2 September 2019 in Kunming, People's Republic of China
Mr. Hao Lei, Deputy Director General of the Budget Department, Ministry of Finance,
Dr. Wencai Zhang, Director General of the International Financial Cooperation Department, Ministry of Finance and former Vice President of ADB,
Mr. Zhang Yansong, Director General of the Yunnan Provincial Department of Finance,
Mr. Shixen Chen, Vice President of ADB,
Dr. Li Kouqing, President of the Asia-Pacific Finance and Development Institute and Shanghai National Accounting Institute,
Distinguished officials from the Ministry of Finance and Yunnan Provincial Government of the People’s Republic of China,
Friends, colleagues and partners in evaluation and in development,
A pleasant good morning to you! Zǎoshang hǎo!
Welcome to the 4th Asian Evaluation Week here in the City of Kunming, a place rich in history spanning nearly 2,400 years. Already at his time, the famous explorer Marco Polo described Kunming as a “large and splendid city” upon seeing its great prosperity, the diversity of its people and the abundance of its land.
As the southwest gateway to the famed Silk Road, Kunming’s thriving trade facilitated the convergence of people, goods and knowledge thereby supporting the link between the East and the West. We could not have chosen a more appropriate place to hold our annual gathering of evaluators from across Asia and from around the world for stimulating meaningful partnerships in advancing our evaluation discipline.
Before I delve into our central theme for this year’s AEW, let me pause for moment to pay special tribute to Farzana Ahmed, our colleague and friend, whose determination, never ending energy and intellectual leadership were critical for the launching of AEW four years ago. Farzana will be retiring from ADB soon and this will be her last meeting as an ADB/IED staff. Thank you Farzana for everything, we will miss you and hope to see you here again as part of your new mission.
This year’s AEW theme is titled “Quality Evaluation for Better Results: Local, National, Regional Perspectives”. I cannot emphasize enough the demands that the evaluation field is now facing to help achieve the global development results outlined in the 2030 Agenda. Concerted efforts at the local, country and regional levels are essential to realize these goals and evaluation is called to play a central role in this endeavor.
Quality evaluation
The expectations about evaluation contributing to advancing today’s development agenda hinge crucially on the quality of our assessments. This is one of these cases in which as the Roman philosopher Seneca succinctly puts it, “It is quality rather than quantity that matters.”
But what do we really mean by quality evaluation? Do we have a shared understanding within the evaluation community on this matter? In answering these questions, let me indicate in the first place that evaluation in itself is an exercise in the search for quality. In this search, there are two dimensions of quality that we must be all clear about: 1) quality of the evaluation itself; and 2) quality of the evaluand or the object of evaluation.
Quality of the evaluation itself
Let me address the first dimension, or the quality of evaluation itself. When speaking of this subject, this dimension is the one that receives most of the attention. Clearly, views on quality of evaluation are diverse, arising from individual work experiences, and the many standards and principles that have been adopted.
However, distilling through the various evaluation policies, principles and standards across institutions and organizations, we find a common thread on three areas defining quality of the evaluation work: a) expertise and methodology; b) enabling environment; and c) utilization and fit for purpose. These three areas complement each other to determine the quality of an evaluation.
Concerning expertise and methodology, these refer to the evaluators’ competencies (i.e. proficiency, analytical and technical skills) and the necessary moral compass that should guide them in undertaking evaluation work, as well as to methodologies underpinned by a high degree of rigor, particularly related to validity, reliability and replicability.
With respect to the second area, an enabling environment works hand-in-hand with strong expertise and rigorous methodology. By enabling environment, I refer to the institutional underpinnings of the evaluation function with a focus on independence, that is, on structural and functional independence.
Lastly on the area of utilization and fit for purpose, quality is seen from the perspective of stakeholders and end-users of evaluation. What I mean is that evaluation should meet the expectations and knowledge needs of users for them to be able to do the right things and do them right.
Quality of the evaluand or the object of evaluation
Let me now turn to the second dimension of the search for quality in evaluation, that is the quality of the evaluand or the object of evaluation. We should not lose sight of the importance of this dimension, for our work is essentially about finding and judging quality in everything we assess. As Robert Stake has stated, two distinct aspect that are necessary for defining the quality of an evaluand: 1) criteria, which refer to the attributes to judge the evaluand; and 2) standards, which denote the amount of these attributes needed for arriving at a specific judgment.
On criteria, in the case of ADB and several other organizations here present, for instance, the most used criteria are relevance, effectiveness, efficiency, and sustainability. On the other hand, standards are the bars within each criterium to define, for example, if interventions are highly relevant, relevant or less than relevant.
The distinction between criteria and standards is often confused by professionals in the field of evaluation. This confusion must be avoided to appropriately identify quality, merit, and worth in the object of evaluation.
Excellence and influence of evaluation
Within the context of quality evaluation work I have just described, let me pause for a moment and pay special attention to two important aspects of this concept: excellence and influence.
These aspects are exemplified by Zhang Qian, China’s famous and great envoy of the Han Dynasty, who despite being held in captivity for 12 years on his mission to explore the West, proficiently delivered on this mission and brought back influential knowledge that paved the way for the development of the Silk Road and opened the communication of China with the rest of the world.
And so, on excellence in evaluation, good evaluators should be guided by the personal and intrinsic urgency of delivering on their mission, meeting the highest standards, while being aware of the consequences that our assessment can have. We should always keep in mind, as I have said in previous years, that evaluation has real consequences.
Concerning influence of evaluation, the mere submission of good quality reports cannot be the only standard that defines our work. The final test lies on the influence of our contribution. I refer to the direct or instrumental impact, the conceptual or contextual influence, and the advocacy or persuasion, attributed to evaluation.
As Mark and Henry have pointed out, a comprehensive view of the mechanisms underlying evaluation’s influence can help ensure that our work fulfills its objective and role in the service of social betterment. Certainly, in assessing the global progress on the SDGs, a comprehensive view of influence mechanisms should involve local, country and regional perspectives, which explains the intellectual and strategic focus of this meeting.
Concluding remarks
Colleagues and friends, Kunming is also known as the City of Eternal Spring – favorable climate, passionate people, and plants and flowers that bloom all year long. Let us all be inspired by this wonderful framework to stimulate our exchanges during the course of this week to contribute to quality evaluation with an emphasis on excellence and influence of our practice at all levels – local, national and regional. And let us all be guided throughout this week by the wise precept of Confucius: “Learn avidly, question repeatedly, analyze carefully, and put what you have learned into practice, intelligently.”
Thank you all for joining us! Xiè xiè!