The CAPI Effect_Boosting Survey Data through Mobile Technology:

This special supplement to Key Indicators for Asia and the Pacific 2019 discusses the role computer-assisted personal interviewing (CAPI) can play in transforming survey data collection to allow better monitoring of the Sustainable Development Goals. The report provides quantitative evidence on why CAPI is a sound and viable alternative to the traditional pen and paper interviewing method, particularly in the context of nationally representative surveys. It also discusses the benefits of delivering CAPI training to statisticians using the popular massive online open course format. Finally, the report provides a summary of existing CAPI platforms and offers some preliminary advice for NSOs to consider when selecting a CAPI platform for their institution.


Foreword
Economies across the world are working towards achieving the 17 Sustainable Development Goals (SDGs), which are monitored using a global framework comprising 169 targets and 232 indicators. Several of these vital SDG indicators rely on censuses or nationally representative surveys, which are currently implemented using the pen and paper interviewing (PAPI) technique in many developing economies. The PAPI method involves enumerators completing paper questionnaires in the field, with the questionnaires then physically transported back to a central location and subsequently digitized through data entry and validation processes. PAPIbased surveys are therefore generally expensive to conduct, take a long time to process, and may also suffer from data quality issues. As a result, there is a need to adopt innovative technologies to reduce survey duration and costs, without compromising data quality.
One such innovative tool that is gaining traction among survey practitioners is computer-assisted personal interviewing (CAPI), which refers to the collection of survey data using handheld digital devices such as tablets or smartphones. A switch from PAPI to CAPI is expected to improve data quality, as error checks can be built into the system and triggered during the interview process. Similarly, interview duration may be reduced as enumerators spend less time manually navigating through the questionnaire. CAPI is likely to be cost-efficient as it eliminates the need to print paper questionnaires and encode data once the survey has been completed. Easy data transfer and the ability to collect ancillary survey information such as global positioning system (GPS) coordinates, photos, and videos are other advantages of CAPI.
There is, however, an informational gap when it comes to rigorously quantifying the benefits of CAPI, particularly in the context of its use in Asia and the Pacific. Such analyses would be extremely useful in building a case for the adoption of CAPI across the region. Similarly, broadening the use of CAPI calls for staff from national statistics offices (NSOs) to become experts in this technology. Finally, available CAPI platforms must be clearly differentiated based on their capabilities and functionalities, to help NSOs assess the relative advantages of each platform.
In 2015, the Asian Development Bank (ADB) launched a technical assistance project to promote the use of CAPI in the nationally representative surveys of three developing member economies: the Lao People's Democratic Republic, Sri Lanka, and Viet Nam. Technical assistance resources were used to train NSO staff in each country on all aspects of CAPI survey implementation; from questionnaire programming and field data collection, to data transfer and the creation of final datasets. To quantify the benefits of CAPI, pilot studies were conducted in Sri Lanka and Viet Nam to compare CAPI with PAPI across four domains: time efficiencies, data quality, cost efficiencies, and respondents' perceptions to technology. To extend the impact of the technical assistance to statisticians around the world, the project team also created free online courses on two popular CAPI platforms: CSPro and Survey Solutions. Finally, through desk research and consultation with various stakeholders, a shortlist of existing CAPI platforms was prepared for inclusion in a forthcoming detailed ADB comparative study, which will greatly assist NSOs in selecting a CAPI platform appropriate to their needs.
This year's special supplement to Key Indicators for Asia and the Pacific presents the case for widescale use of CAPI by summarizing the results of the aforementioned technical assistance project. The report has been produced by the Statistics and Data Innovation Unit within the Economic Research and Regional Cooperation Department at ADB, under the overall direction of Kaushal Joshi. The project and report teams were led by Lakshman Nagraj Rao, with valuable research and technical support from Pamela Lapitan and Dianne Robillos. Lakshman Nagraj Rao, Jude David Roque, Dave Pipon, Lachlan Bruce and Guido Pieraccini served as coauthors of the report. Elisabetta Gentile provided valuable input to this report, working alongside the project team on a range of analysis activities. Guido Pieraccini developed the CAPI software that was used to implement the comparison of CAPI with PAPI in Sri Lanka and Viet Nam. Thomas John Ballatore and Lachlan Bruce developed the online courses on CAPI. Rea Jean Tabaco and Anna Christine Durante provided excellent research assistance throughout the implementation of the project. Dave Pipon and Jude David Roque assisted with data cleaning and compilation activities. Ma. Roselia Babalo and Aileen Gatson provided able operational support through the course of the technical assistance project. Technical advice on data analysis was provided by Joseph E. Zveglich Jr., Rana Hasan, Valerie Mercer-Blackman, and Kathleen M. Farrin, as well as participants at the Asian and Australasian Society of Labour Economics 2018 Conference and the International Statistics Institute World Statistics Congress 2019. The cover of this publication was designed by Marjorie G. Ofaga. Manuscript editing was performed by Paul Dent, while the publication's layout, page design, and typesetting were carried out by Rhommell Rico and Principe Nicdao.
We are extremely grateful to the participating NSOs for their active involvement in every stage of the project. Their contribution in terms of data collection, cleaning, and processing as well as expert input, local knowledge, and practical advice, was vital for the successful completion of both the technical assistance project and this year's special supplement.
We hope that this publication will be instrumental in championing the use of CAPI across Asia and the Pacific, as a means of delivering highquality, timely, and cost-effective data for SDG monitoring. • Timely, high-quality, consistent, and comparable data serve as the backbone to measuring the progress made towards achieving the targets embedded within the Sustainable Development Goals (SDGs). National statistics offices (NSOs) play a crucial role in collecting, validating, and reporting the data for the indicators that are linked to the SDGs. They do so in a manner that is aligned with the methodologies set forth by international custodian agencies.

Yasuyuki Sawada
• Meeting the data requirements of the SDGs in the developing member economies of the Asian Development Bank (ADB) is challenging because several SDG indicators rely on census and survey data that are expensive to collect and take a long time to process. This necessitates targeted technical assistance from international agencies, coupled with a strong commitment of resources to the production of quality statistics.
• The quality of survey data is another important factor to consider within the SDG framework since the reliability of an SDG indicator is directly impacted by it. In many instances, data quantity might be at odds with data quality because increasing sample size or adding more questions to a survey entails a more elaborate field survey protocol and, consequently, increases the potential for data errors. This calls for the need to identify innovative data collection practices that are not only cost-effective in delivering data quantity, but also minimize the inherent conflicts with data quality to provide accurate statistics for SDG monitoring.
• Information technology has revolutionized field data collection methods with computer-assisted personal interviewing (CAPI) gaining traction amongst survey practitioners. CAPI refers to the collection of survey data using handheld digital devices, as opposed to the typical pen and paper interviewing (PAPI) method. With inbuilt data checks, navigation tools, easy data transfer options over the internet or via Bluetooth technology, and the ability to capture ancillary information such as global positioning system (GPS) coordinates and photos, CAPI not only reduces the overall time to produce a clean dataset, but is also expected to improve data quality.
• While developed economies have been using CAPI for several years, NSOs in developing economies have yet to take full advantage of this methodology. Factors such as high startup costs and lack of technical capacity to develop and implement CAPI surveys are contributors to the slow adoption of CAPI.
• To promote the use of CAPI in the Asia and Pacific region, ADB implemented a technical assistance project with three pilot economies: the Lao People's Democratic Republic (Lao PDR), Sri Lanka, and Viet Nam. The project had four broad aims to: (i) assist the pilot economies to implement CAPI in nationally representative surveys, (ii) compare the benefits of CAPI with PAPI, (iii) extend the benefits of CAPI to a wider audience through e-learning courses, and (iv) identify criteria most relevant to NSOs when selecting a CAPI platform that meets their institutional needs.
• An important concern raised by NSOs in Sri Lanka and Viet Nam was the need to quantify the benefits of CAPI relative to PAPI, in order to build a case for transitioning to CAPI for all future sample surveys. Consequently, the technical assistance project conducted randomized experiments to compare CAPI with PAPI across four important issues: time efficiencies, data quality, cost efficiencies, and the perceptions of respondents. The experiments were conducted in 2017, during Viet Nam's Labor Force Survey and Sri Lanka's Agricultural Household Survey, with a sample of households randomly assigned to either a CAPI survey group or a PAPI survey group.
• In the case of Viet Nam, the study found that a switch from CAPI to PAPI reduces interview duration by an average of 9.42 minutes, translating to 27.3% less time per household. For Sri Lanka, assessing interview duration was more problematic due to the inability of the software to accurately account for time lapses when callbacks were required. This calls for more research and innovation by CAPI platform developers to enhance their time-capture mechanisms.
• In terms of data quality, results from both economies showed that a switch from PAPI to CAPI reduced the total number of errors.
In particular, CAPI reduced the number of errors by 0.8 per survey interview in the case of Viet Nam, and by 6.2 per survey interview in the case of Sri Lanka. To put this into context, for a survey of 10,000 respondents, the findings would translate into a reduction in total errors of 8,000 for Viet Nam and 62,000 for Sri Lanka.
• The studies in the two economies also found that CAPI becomes cost-effective for large-scale surveys. CAPI's fixed costs-generally associated with programming the software-might initially be higher relative to PAPI. However, since CAPI virtually eliminates PAPI's variable costs such as printing questionnaires and data entry, it becomes cheaper to implement as sample sizes increase. The breakeven pointthe sample size at which it becomes cheaper to implement CAPI over PAPI-was calculated to be 1,769 households for Viet Nam and 1,467 households for Sri Lanka.
• In terms of respondents' perceptions of CAPI relative to PAPI, the two studies delivered mixed results. This may be related to how familiar the survey respondents were with technology in general. Households in Viet Nam did not seem to exhibit any attitudinal differences across the two methods of data collection, which may be attributed to the familiarity of the general public with technology, particularly in urban areas. However, some differences were observed for Sri Lanka, with the agricultural survey being conducted in more rural localities, where a lower familiarity with technology might be expected. This, however, does suggest that an outreach and education campaign on CAPI may be necessary prior to the launch of any large-scale survey.
• To maximize the impact of technical assistance resources beyond the three pilot economies, ADB partnered with the Food and Agriculture Organization of the United Nations to develop two freely accessible online courses on CAPI. The objective of the courses was to provide practical training on CAPI to statisticians and NSO employees around the world, guiding them on how to implement a computer-assisted survey using two popular platforms: CSPro and Survey Solutions.
• With 655 individuals enrolled across the two courses, to the best of our knowledge, this represents the largest-ever online training initiative by any international organization seeking to build statistical capacity. A completion rate of almost 40% across the two courses was nearly seven times the average for comparable globally accessible online courses. Feedback provided by the students, as well as an analysis of completion and performance data, generated several useful recommendations for future iterations of the courses. These included the need for incentive mechanisms in the early modules of the courses and the provision of training in multiple languages.
• The rise in CAPI-based data collection in recent years has led to a growing number of software platforms being developed. At the time of this report, there were over 40 CAPI platforms available for general use, with platforms differing greatly in several aspects such as ease of programming, interface, specialization, pricing, and data transfer options, to name a few.
• While a diversity of options in CAPI platforms is a positive, selecting the appropriate platform can be a challenge for NSOs wanting to transition to this new technology. There is an abundance of product information when it comes to CAPI platforms, but such information is not standardized and no platform developer offers useful advice on whether, and to what extent, a platform might be appropriate for complex and/or large-scale household surveys.
• ADB is conducting a study to provide a critical comparison of freely available CAPI platforms suitable for NSO use. The comparative study will be based on the following key areas: availability of general features; functionality of the programming interface; and usability for interviewers, supervisors, and survey managers. The application of some key criteria for platforms to be included in the study led to the shortlisting of six platforms. These will be reviewed in depth, with the aim of crafting suitable recommendations for NSOs.

Introduction
The Sustainable Development Goals (SDGs) were born out of a universal call for action to "end poverty, ensure shared prosperity, and protect the planet to achieve a sustainable future." 1 Spread across 17 goals, 169 targets, and 232 indicators that are cross-cutting in nature, the SDGs build on a global vision to "leave no one behind" by committing to address issues such as health, education, environment, climate change, and jobs, in tandem with economic growth.
Timely, high-quality, consistent, and comparable data serve as the backbone to monitor the SDGs, and to measure the progress towards the targets embedded within them. National statistics offices (NSOs) play a critical role in collecting, compiling, validating, and reporting data for the indicators linked to the SDGs. They are also at the forefront of implementing the statistically sound methodologies that are set out by international custodian agencies to measure relevant SDG indicators at the national level.
However, generating data that meet all the criteria stated above is an arduous task, and one that requires targeted technical assistance complemented by resource and financial commitments to produce quality statistics. A 2018 report by the United Nations Statistics Division noted that, of 98 economies for which relevant information was available, only 71 had national statistical legislation that was compliant with the Fundamental Principles of Official Statistics (United Nations 2018). More pertinently, of the 71 economies with compliant legislation, only 37 were developing economies. The report also highlighted that only around 0.3% of official development assistance (ODA) has been allocated to statistics annually between 2010 to 2015. For 2015, this equates to $541 million, a figure that is dramatically lower than independent estimates that suggest SDG monitoring over a 15-year period to 2030 will require a total of $254 billion, or $16.94 billion annually (Jerven 2014).
Several SDG indicators rely heavily on census and survey data, which form a significant portion of an NSO's total budget. This is primarily because censuses and surveys in developing economies are mostly conducted using paper questionnaires and need to be digitized before they can be used for analysis. This process of digitization can get expensive very quickly. After accounting for resources from existing international survey programs, a conservative estimate of global cost gaps to meet the monitoring targets of those SDGs relying on survey data is $300 million per year (Demombynes and Sandefur 2014). Similarly, Kilic et al. (2017), estimate that the real unit costs for household surveys across developing Asia could vary anywhere from $48 to $130.
In addition to the cost factors associated with implementing sample surveys, data quality and data quantity are often at odds with each other. The SDG framework requires indicators to be disaggregated by location, sex, age, income, and other relevant dimensions. As the granularity requirements for SDG indicators increase, so does the associated sample size required, and the complexity of survey instruments needed to collect such data. However, larger sample sizes also require hiring more enumerators and implementing a more complicated field survey protocol, which are factors associated with higher volumes of sampling and non-sampling errors. This need to balance the inherent conflict between data quantity and data quality has driven calls to identify survey practices that are innovative and cost-effective and can provide reliable data for SDG monitoring.
Technological innovation is constantly transforming the way our societies operate and how our economies prosper, with information technology having a profound influence on even the most traditional of industries. The rate of technological advancement has been remarkable: from floppy disks in the 1990s that could store only 1.44 megabytes of data, to today's half-inch memory cards that can store 512 gigabytes of data. The more recent trend towards miniaturization and wireless networking, along with the proliferation of communication satellites, has translated into rapid innovation in handheld devices and delivered a significant extension of computing capacity, mobility, and information-sharing abilities. A typical smartphone can now record videos, capture images, open documents, search the internet, help with trip navigation, and provide access to e-mail and other electronic communications. This technology has dramatically changed how we interact with the world.
The benefits of technological advancement have extended into the survey world, with digitized data collection gaining traction amongst survey practitioners. One notable advancement is computer-assisted personal interviewing (CAPI), which refers to the collection of survey data using handheld devices such as tablets or smartphones, as opposed to the traditional pen and paper interviewing (PAPI) method. The potential advantages of the switch from PAPI to CAPI include reduced survey cost and duration, easy data transfer through wireless or Bluetooth technology, capturing of images, use of global positioning systems (GPS), and other ancillary information about survey implementation. Importantly for data quality, CAPI also permits the inclusion of several types of data checks, which can be initiated during the interview process. These checks enhance questionnaire flow and are likely to reduce erroneous data. However, the developing economies are yet to fully tap into the transformative role that CAPI can have on survey data collection and management processes. This is largely due to a perception among respondents about lack of data safety when being surveyed using handheld devices.
With the objective of promoting CAPI to NSOs within Asia and the Pacific, the Asian Development Bank (ADB) implemented a technical assistance project in three pilot economies: the Lao People's Democratic Republic (Lao PDR), Sri Lanka, and Viet Nam (ADB 2015). The project design comprised three major components: (i) assisting the three pilot economies with the implementation of CAPI in nationally representative surveys; (ii) conducting rigorous analysis comparing the benefits of CAPI to PAPI; and (iii) extending the benefits of the training materials developed in the technical assistance project to a wider audience through e-learning courses on CAPI. In addition, ADB is currently undertaking a comparison study of existing CAPI platforms, with the objective of providing guidance to NSOs on the features and advantages of each available platform. This study is expected to help NSOs make informed decisions on which platform might best suit their institutional needs.
This special supplement to Key Indicators for Asia and the Pacific 2019 is intended to showcase the results of the aforementioned technical assistance project. The report is organized around three major themes. The first theme focuses on the use of a rigorous evaluation methodology, to systematically present the benefits of CAPI to NSOs. Next, the report shares ADB's experience in partnering with the Food and Agriculture Organization of the United Nations to expand on the traditional modes of statistics capacity building using the increasingly popular massive open online course (MOOC) format to train statisticians on CAPI. Finally, the report provides a list of existing platforms and presents a set of broad criteria that have emerged as important factors for NSOs when selecting an appropriate CAPI platform.
ADB's Strategy 2030 sets the course for the bank's efforts to achieve a prosperous, inclusive, resilient, and sustainable Asia and the Pacific, while sustaining efforts to eradicate extreme poverty, and respond to the changing needs of the region (ADB 2018). This special supplement reflects ADB's commitment to promote the use of innovative technology across its operations and in the provision of capacity building support to its developing member economies. Efforts to support NSOs in modernizing their statistical survey operations are encouraging digital transformation, a key cross-cutting theme throughout Strategy 2030.

Pen and Paper Interviewing
In the developing world, data collection by NSOs relies almost exclusively on PAPI. Under the PAPI methodology, feedback or answers provided by survey respondents are recorded onto paper forms or questionnaires by field enumerators. The responses are then verified by field supervisors, before being sent on to the next level, usually the survey manager located at the headquarters of an NSO. The survey manager conducts his or her own set of checks, before sending the questionnaire for data entry, a process in which the questionnaire is encoded to prepare digitized datasets. The data entry process of NSOs typically involves an independent double entry system, where the first data entry is compared with the second data entry. Unmatched values from the two data entry processes are validated by pulling out the questionnaires and recoding the correct information. These data are subsequently handed over to the analyst for batch editing and further data filtering to produce a clean dataset.
Several challenges have been linked to the PAPI method. The first and most obvious challenge relates to the amount of paper to be dealt with under PAPI. Enumerators are required to carry large and sometimes weighty volumes of paper questionnaires into the field, and the completed questionnaires then have to be safely transported back to headquarters for encoding. Any census or complex survey could easily accumulate tens of thousands of sheets of paper, which subsequently also take up significant physical space in terms of document storage. Correspondingly, a lot of time and labor is needed to compile and encode all the data gathered.
The second challenge with PAPI relates to growing data needs. Surveys questionnaires are becoming increasingly complex, requiring additional effort and scrutiny by enumerators to properly navigate through a multitude of skips and record valid answers in the appropriate fields. The likelihood of human error increases with the number of skips in a questionnaire and the number of interviews required. With PAPI, enumerator fatigue is more likely to occur and there are limited mechanisms for supervisors and data managers to ensure the quality of responses at each stage of intervention. This may lead to a compromise in data quality and subsequent analysis.
The third challenge with PAPI is the timeliness of data availability, an issue of tremendous significance for survey practitioners. The time between data collection and data analysis is heavily contingent upon the logistics of transferring the paper questionnaires from one place to another. For surveys conducted in remote or inaccessible areas, as is often the case in developing economies, transportation of questionnaires is bound to take significant time. Further, once the paper questionnaires do arrive at the NSO headquarters, the responses still have to be entered individually and converted to a clean dataset before any analysis can begin. In fact, this step of data entry and cleaning is the most time-consuming aspect of the survey process. All these factors may lead to a substantial delay in the release of survey estimates, upon which several time-sensitive policy decisions might depend.
The final challenge relates to survey cost. With PAPI, not only is there the need to print the paper questionnaires, they then need to be transported back and forth from the NSO headquarters, double-entered for consistency, and later stored. Survey manuals and all ancillary documentation relevant to the survey process also need to be printed for all enumerators. For nationally representative surveys in developing economies with large populations, the cost implications of printing, transporting, digitizing, and storing paper questionnaires are very high.

Computer-Assisted Personal Interviewing
While PAPI is a tried and tested technique, its limitations in the context of contemporary data requirements have driven survey practitioners to ask the question: "Is there a better alternative?" Given the growing need to expedite and improve the quality of data collection and management, CAPI has become the popular alternative and is often heralded for its benefits over the traditional PAPI method.
CAPI allows for the recording of interview responses on handheld devices, such as tablets or smartphones, rather than on paper questionnaires. The obvious benefit of this method is that, instead of lugging volumes of paper questionnaires into the field, enumerators carry just a small, lightweight digital device. This eliminates the time and expense associated with printing, transporting, and distributing the reams of paper typically associated with PAPI surveys. Consequently, the upfront digitization of interview responses is inherent in the CAPI methodology, eliminating the need for time-consuming and costly back-end data entry, and generating digital or electronic datasets that are almost instantaneously available for review and analysis. As such, CAPI is regarded as a substantially more time-efficient and affordable option compared to PAPI. Figure 3 presents a flowchart comparing the steps required to obtain a final dataset using both the PAPI and CAPI methods.
CAPI may also reduce the likelihood of enumerators making errors during interviews. The accuracy of PAPI interviews depends on how the enumerator conducts the interview, navigates through the questionnaire skips, and determines whether or not specific responses are valid. In other words, the traditional PAPI method is prone to human error. A CAPI survey, on the other hand, can be equipped with automated routing or skipping mechanisms as well as logic checks to validate answers, thereby minimizing the errors that can be made through erroneous human judgment. In this respect, CAPI has the potential to vastly improve the quality of the data gathered. CAPI also possesses the additional functionality of modern technology, such as capturing still images and video, recording audio, and even allowing GPS plots to be embedded into questionnaire responses. These are not tasks that can be easily integrated into PAPI surveys.
In time, web-based surveys (also known as computer-assisted web interviewing) and interactive voice response technology may be viable alternatives to CAPI. Box 1 describes ADB's experience with implementing interactive voice response technology in the Philippines.

Box 1: Next Generation Data Collection in the Philippines
Interactive voice response (IVR) refers to a technology that uses computer-generated voices to interact with humans in an automated fashion. IVR is commonly used by the banking sector to provide customers with access to bank accounts over the phone in an automated fashion, based on voice responses or keypad inputs that match the bank's database. Advanced IVR technology also incorporates machine learning and speech recognition to match specific voice patterns to individual bank accounts. IVR systems can work with both prerecorded and dynamically generated audio.
Another technology closely related to IVR is short message service (SMS) text blasts, which refers to sending text messages to a large number of recipients simultaneously. Often, SMS text blasts can be combined with a short survey, whereby automated followup questions are sent to the recipient depending on their responses to the original message and subsequent messages. This tool can be useful if follow-up data from individuals needs to be collected immediately and in an inexpensive manner, assuming the people in question own cell phones.
A novel application of IVR and SMS text blasts was implemented by the Asian Development Bank in partnership with the Department of Education in the Philippines. A study was being conducted to identify the best way to contact parents of senior high school students to discuss student performance and career options. The Youth Education Investment and Labor Market Outcomes Survey (YEILMOS), conducted by the bank and the department, had been completed in 2017. It covered four survey areas: Davao del Sur Eastern Samar, Ilocos Sur, and the National Capital Region. The YEILMOS collected the names of the parents interviewed and their phone numbers, which made it possible to use IVR and SMS text blasts to send follow-up messages as part of the current study.
The study compared different outreach strategies to invite parents to guidance meetings organized by the schools and attended by their children. For one set of randomly selected parents, formal letters were sent as invitations (control group). For a second group of randomly selected parents, the formal letters were followed by IVR messages and SMS text blasts to serve as reminders. These were sent using the EngageSPARK platform. The figure in this box provides a snapshot of how the text blasts were set up.
The first step involved setting up the reminder text to be sent out. The figure shows just a portion of the text used. Note that certain fields can be automated to be personalized, such as names, times, dates, and locations. The respondents could respond to this notification with a "Yes, will attend". The second step, as seen in the second screenshot in the figure, was to upload the contact list for the text blast. The implementation of IVR messages was similar, except that, instead of setting up text as the content, a voice recording was uploaded. The IVR messages were received by the YEILMOS respondents as phone calls.
The measure of invitation effectiveness was gauged by the actual attendance of the parents at the guidance meetings. The study found that the group of parents that received both the letter and follow-up IVR and SMS messages had a 12% greater attendance rate than the group that received only the letter, with this difference being statistically significant. The IVR and SMS technology also dramatically reduced the time and cost required to assess the follow-up option, as compared to assigning school staff to make individual phone calls and record responses. These data were easily collected using the options embedded in the EngageSPARK platform.
continued on next page The CAPI Effect: Boosting Survey Data through Mobile Technology

Setting Up A Text Blast on EngageSpark
Source: Asian Development Bank estimates using EngageSpark.

The Need to Clearly Communicate CAPI's Benefits
The capacity to program different types of data checks (missing, validation, skip, logical) into a CAPI questionnaire along with enhanced features such as GPS mapping, voice recording, and a plethora of data transfer methods has been a game-changer in the survey world. NSOs are exploring CAPI systems with more interest than ever before.
However, there continues to be an information gap when it comes to quantifying the specific benefits of CAPI, particularly in terms of data quality, time efficiencies, cost reductions, and survey respondent experience. Evidence is mostly anecdotal, with very few rigorously conducted studies comparing CAPI to PAPI. In addition, most of the literature focuses on developed economies, and the few studies that have looked at the implementation of CAPI from the perspective of a developing economy have focused on those in Africa. Independently validating the benefits of CAPI in the context of developing economies across Asia and the Pacific merits its own analysis. Such an analysis must assess differences in how respondents relate to technological innovation in the collection of survey data, taking into consideration cultural norms and/or familiarity with technology.
This section discusses the existing evidence on the benefits of CAPI and assesses four key areas: (i) time efficiencies, (ii) data quality, (iii) cost efficiencies, and (iv) respondent perceptions.

Time efficiencies
The ideal scenario for any survey is to have as many questionnaires as possible completed in the shortest amount of time, without compromising on data quality. Interview length is therefore important because it corresponds to the rate at which questionnaires are completed. One of the perceived benefits of CAPI is that it reduces the duration of interviews because of easier navigation through the questionnaire by the enumerator.
In a PAPI survey, an enumerator has to fill out the response to a question by hand and follow the skipping logic to proceed to the next question. This is particularly time consuming when the survey is initially conducted, since the enumerator is still familiarizing him or herself with the questionnaire. Further, if an enumerator does not fully understand the meaning of one or more questions or the skipping logic, he or she will need to temporarily pause the interview, pull out the training manual, and review the section pertaining to the question, before returning to the paper questionnaire. All of these factors can lead to increased interview time in a PAPI survey.
When the CAPI method is first introduced, the process of enumerators adjusting to new technology could lead to a short-term increase in interview duration. However, with the current penetration of smartphones and other handheld devices, even in remote areas, it can be argued that this may be of limited concern in the context of Asia and the Pacific.
Early studies such as Lynn (1998) and Baker et al. (1995) argued that CAPI surveys can reduce interview times by 16% to 20%. Caeyers et al. (2012) found that CAPI reduced interview duration by nearly 10% relative to PAPI, and attributed this to the automated skipping features unique to CAPI, as observed in a study conducted in Tanzania. A few other studies also observed reductions in interview duration when using CAPI compared to PAPI, but found these differences to not be statistically significant (King et al. 2013, Zhang et al. 2012. However, some studies have suggested that interviews using CAPI might, in fact, take longer than those using PAPI (Martin et al. 1993, Fuchs et al. 2000, Watson and Wilkins 2011. This could be attributed to the process of enumerators adjusting to new technology as well as the intentional allocation of more proficient enumerators to administering PAPI. That said, Leisher (2014) in a comparison of household surveys in Africa observed that the mean duration of interviews using CAPI decreased after 3 days of tablet use. This corroborates the hypothesis that there is an enumerator adjustment period before the perceived time efficiencies associated with CAPI become evident.
While interview duration is an important factor in comparing CAPI to PAPI, another significant contributor to time efficiencies in survey management is the ability of CAPI systems to deliver data quickly and efficiently using wireless or Bluetooth technology. The movement of data from the field to an NSO's data processing center is almost instantaneous in CAPI, making the dataset readily available for analysis, which is important within the NSO context, where strict deadlines must often be observed to publish key economic and social statistics. De Leeuw (2008) and King et al. (2013) have highlighted the benefit that CAPI's faster back-end processing, allowing data cleaning and processing to be completed immediately after interviews have been conducted.

Data quality
Another perceived benefit of CAPI is its impact on data quality. Research has acknowledged CAPI's ability to improve data quality, noting its automated routing and skipping features as well as its built-in logic checks (De Leeuw 2008, Sebestik et al. 1988, Olsen 1992. Nicholls et al. (1997) state that CAPI would eliminate virtually all respondent omissions in survey questionnaires. However, the advantage has been presented mostly through anecdotal evidence, and very few systematic studies relying on rigorous methodologies have been conducted, particularly in the context of Asia and the Pacific.
A study by Caeyers et al. (2012) is one of the few attempts to systematically examine CAPI's impact on data quality. A randomized experiment was conducted in 1,840 households on Pemba Island, Tanzania, using three different study groups to compare and isolate the effects of CAPI versus PAPI. The first study group was composed of households that were interviewed for a consumption survey via the traditional pen and paper method. The second group was interviewed for the same consumption survey using CAPI, but the questionnaire was equipped only with automated skipping patterns. The third group was also interviewed for the consumption survey using CAPI software, but this time the questionnaire contained both automated skipping patterns and other logic checks.
The study found that 94% of the invalid responses attributable to skipping errors in the PAPI interviews were addressed with the CAPI software. In addition, the results from the third study group showed that incorporating logic and consistency checks further reduced inconsistencies and errors in responses. Correspondingly, the study found that the share of missing or illogical values was lowest within the group that was interviewed using CAPI with built-in logic and consistency checks (only 2% of responses) compared to the PAPI group (83% of responses).
Another attempt to quantify CAPI's edge in terms of data quality was made by Fafchamps et al. (2012), with the researchers conducting a study on microenterprise sales and profits in Ghana. Because sales and profit data based on recall are prone to mismeasurement and are often "noisy", the study collected the information using two CAPI consistency checks. The first check cross-sectionally compared sales and profits for firms, while the second flagged large differences between sales and profits in the previous check. The study found that such checks reduced the variation in responses, yet noted that only 3% to 13% of the errors were corrected. The authors of the study suggested that CAPI's modest effect on data quality may have been attributable to the volatility of the microenterprise sector, thereby contributing to the noisiness of the data. This finding does not discount CAPI's ability to improve data quality; it merely reveals its limitations in the context of sales and profits, as opposed to surveys such as those on consumption expenditure, where the benefits of CAPI seem to be more apparent.

Cost efficiencies
A good deal of the literature on data collection has acknowledged that there are high fixed costs associated with switching to CAPI. For example, Schrapler et al. (2010) stated that "CAPI requires a sizable investment in hardware and more front-end design and development work than PAPI." CAPI's use of technology also means that more resources and time need to be spent training field enumerators and supervisors on how to use both the software interfaces and the devices.
Yet, De Leeuw (2008) argued that there can be significant cost savings from the back-end efficiencies achieved with CAPI. These include vastly reduced printing costs, reduced logistics costs associated with transporting paper questionnaires to and from the field, and elimination of data entry and data cleaning costs.
According to Leisher (2014), the cost of CAPI per completed interview is 74% lower than for paper-based surveys, with most of the savings attributable to the elimination of data entry and cleaning as well as lower enumerator fees. Other recent studies also corroborate CAPI's cost efficiencies, finding that the back-end savings are significantly larger than the front-end expenses such as hardware and software development (King et al. 2013, Zhang et al. 2012).
To gain a complete understanding of the potential cost efficiencies that might be achieved by implementing CAPI surveys, it is necessary to juxtapose the increase in front-end costs against the reduction in, or elimination of, back-end costs. In line with this, there is evidence that points toward the increased efficiency and affordability of CAPI, particularly when it comes to larger samples. Caeyers et al. (2012) found that CAPI becomes most cost-effective for sample sizes of 3,600 or more, as this is the breakeven point at which the reduction in the variable costs typically associated with PAPI would exceed the fixed costs incurred by switching to CAPI. This finding is supported by an earlier study from Weeks (1992), which stated that "because back-end costs are variable and the front-end costs are largely fixed, CAPI [becomes] economically attractive for large surveys." There is a strong case for research into an updated cost comparison between PAPI and CAPI, as many forms of digital technology have become omnipresent and therefore less expensive. Increased competition in markets for tablets and computers has led to substantial price reductions, shifting the cost-benefit analysis further in favor of CAPI (Rahija 2016). In addition, the development of low-cost and easy-to-learn CAPI software in recent times, has vastly reduced the expected high programming and design costs associated with CAPI.

Respondent perceptions
As with any form of change or a new approach to doing something, it is not unusual for there to be a reluctance to accept new technology because of unfamiliarity. It is therefore important to consider the extent to which respondents might accept or respond to changes in data collection modes, and to gauge their attitudes to the relatively new methodology of CAPI in comparison to the traditional approach of PAPI.
Previous literature has investigated metrics on nonresponse and willingness to respond in different kinds of data collection interviews. Couper (2005) reiterated the concern that new technology in the form of CAPI might reduce the willingness to respond in an interview, due to hesitation among respondents to share information with a computer. Nonetheless, most studies have found no significant differences in nonresponse or refusal rates between PAPI and CAPI. For instance, Nicholls et al. (1997) stated that CAPI had very little impact on reducing the number of refusals for an interview. Meanwhile, Baker et al. (1995) also found no significant difference in refusal rates between PAPI and CAPI, while Tourangeau et al. (1997) reported only slightly higher nonresponse rates with CAPI.
Some studies have also looked at the emotional sentiments of respondents regarding CAPI. For example, Baker (1992) found that CAPI was perceived by most respondents as "interesting and professional". De Leeuw et al. (1995) found that most reactions to CAPI were either positive or neutral, with only around 5% of survey respondents preferring the traditional pen and paper method. Nearly half of the respondents in this study seemed to trust the process of collecting data via computer (De Leeuw et al. 1995). The trust factor is also supported by Baker et al. (1995), who found a greater willingness to share more sensitive information with CAPI compared to PAPI.
There are, however, some contrary study results regarding respondent perceptions to CAPI. Schrapler et al. (2010), using a multivariate logistic regression, determined the factors surrounding nonresponse in a socio-economic survey. The estimates of their model revealed that the first three survey waves exhibit significantly higher probability of refusals with CAPI than with PAPI, particularly on questions dealing with gross and net income. While confidentiality concerns related to new technology may be an explanation for this, the authors suggested that it was possibly the interviewer's lack of confidence in CAPI, given its novelty at the time, and that this lack of confidence was, in turn, perceived by the respondents. In any case, the findings suggest that perception can impact data quality because the reliability of the information respondents provide will always be reflected by their level of trust in the interviewer coupled with the mode of data collection. Caeyers et al. (2012) went beyond looking at nonresponse or willingness to respond, and looked at the perceptions of respondents in terms of aspects such as degree of comfort, enjoyment, confidentiality, and difficulty, among others. While this analysis did not find any significant differences between PAPI and CAPI across these different aspects, it would have been worthwhile to examine whether there exists any respondent, or even interviewer, characteristics that influence perception.
Ultimately, the extent to which the perceptions of respondents can affect the accuracy and reliability of the answers they provide-in turn impacting the survey results and analysis-is an area that should be further explored. Similarly, it is also worth exploring the extent to which the perceptions themselves reflect, or do not reflect, reality.

ADB's Efforts toward Mainstreaming CAPI
Monitoring the Sustainable Development Goals (SDGs) entails the availability of timely, disaggregated, and high-quality data. Since most of ADB's developing member economies rely on PAPI as their primary data collection method for field surveys, there can be long periods before data are readily available for analytical purposes. Given the dynamic nature of the economies in Asia and the Pacific, outdated data can lead to suboptimal policymaking.
While several developing economies in the region have made early strides in integrating CAPI into their respective censuses or surveys, this has generally been done on an ad hoc basis. The shift to CAPI has not been sustained over time or scaled up due to a host of issues, including the technical capacity of the NSOs, high costs in training enumerators and supervisors, and other operational challenges. There is also the significant front-end investment in hardware and software. Another barrier to introducing CAPI might be certain logistical and organizational changes required, as the roles and responsibilities needed for its implementation differ from that of the traditional PAPI method.
In support of the increased data requirements of the SDGs, and to quantitatively demonstrate the numerous benefits of transitioning to CAPI, ADB conceptualized a technical assistance project to build statistical capacity within NSOs and promote the use of CAPI in nationally representative surveys. The desired impact of the project included the improvement of the coverage, quality, and timeliness of statistics produced by national statistical systems via the proposed technological improvements in data collection methods. To achieve this goal, technical assistance was provided to undertake three major activities: (i) assist three pilot economies with the implementation of CAPI in nationally representative surveys, (ii) conduct rigorous analysis comparing the benefits of CAPI to PAPI, and (iii) extend the benefits of the training materials developed for the project to a wider audience through e-learning courses on CAPI. In addition, ADB is currently undertaking a comparison study of existing CAPI platforms, with the objective of providing guidance to NSOs on the features and advantages of each available platform. This study is expected to help NSOs make informed decisions on which platform might best suit their institutional needs.
Three economies-the Lao People's Democratic Republic (Lao PDR), Sri Lanka, and Viet Nam-were selected to be part of this project, and NSO staff from these economies were trained in all aspects of CAPI implementation. This included questionnaire programming; the training of supervisors, enumerators, and survey management teams; data transfer; and conversion of raw data into readily available datasets. In the Lao PDR and Viet Nam, ADB worked with the NSOs to support labor force surveys. Meanwhile, in Sri Lanka, support was extended to implement a complex and multitopic Agricultural Household Survey (AHS).
An important concern raised in Sri Lanka and Viet Nam was the need to quantify the benefits of CAPI relative to PAPI. Such an analysis was needed to provide concrete and convincing reasons that would build a case with NSO management to scale up CAPI for all future sample surveys. To respond to this concern, the project implemented randomized experiments to compare CAPI with PAPI across four important domains: data quality, cost efficiencies, time efficiencies, and perceptions of respondents. A description of why randomized experiments can be useful in comparing CAPI with PAPI is provided in Box 2.

BOX 2: Randomized Experiment: CAPI versus PAPI
Experiments are studies wherein an intervention is introduced by the researcher with the objective of studying its effects. The alternative to an experiment is an observational study, wherein a treatment or an intervention is observed for its effect, without any attempts to change who will or will not be exposed to it. Cohort studies and case control studies are two types of observational studies. Freedman (2006) argues that "experiments offer more reliable evidence on causation than observational studies." A randomized experiment is a commonly used statistical methodology to obtain causal inference on the impact of a particular intervention. Such experiments have been widely used in medical trials to assess the effectiveness of certain drugs. The field of economics has also used this method to look at the impact of specific socioeconomic interventions such as conditional cash transfers (Skoufias et al. 2001) or bednets (Tarozzi et al. 2014). However, the application of randomized experiments to the field of survey management is relatively unexplored.
The simplest kind of a randomized experiment includes two groups: a treatment group, which receives the intervention; and a control group, which does not receive the intervention. So long as the choice as to which is the treatment group and which is the control group is purely random and does not depend on observable or unobservable characteristics of the population, statistical theory shows that the two groups are, on continued on next page average, comparable. This is because selection bias between the treatment and control groups has been eliminated. Thus, any intervention that is randomly assigned to one group versus another will lead to unbiased estimates of the impact of the specific intervention. That said, special care needs to be taken in terms of power calculations to ensure that the sample size is sufficient to determine the impact of the intervention.
In order to rigorously compare computer-assisted personal interviewing (CAPI) with pen and paper interviewing (PAPI), and ensure unbiased results, one could implement a randomized experiment technique. The treatment group in this case would be exposed to CAPI, as this is the novelty that is being introduced to surveying, while the control group would be exposed to PAPI, the conventional method in the field. So long as the allocation of CAPI and PAPI is random and not a function of predetermined characteristics, the two groups are comparable and any differences in outcome can be attributed to the treatment, i.e., being interviewed using CAPI. The figure in this box provides an illustration of how this would be implemented in practice.

Box 2 continued
Research Field Implementation: CAPI versus PAPI CAPI = computer-assisted personal interviewing, PAPI = pen and paper interviewing. Source: Asian Development Bank estimates.

Enumeration Area
Within each enumeration area, a random sample of households are allocated to the treatment and control group. Both groups are given the same survey questionnaire.

Control (PAPI)
Households selected as treatment are interviewed using the computer-assisted personal interviewing method.
Households selected as control are interviewed using the pen and paper interview method.

Viet Nam's Labor Force Survey and associated experimental design
One of the major surveys implemented by Viet Nam's General Statistics Office is the nationally representative Labor Force Survey (LFS), which is conducted every quarter and serves to provide a snapshot of the country's labor market. The survey instrument comprises 71 questions that are grouped into three main sections: (i) household and resident information, (ii) respondent characteristics, and (iii) questions classifying economic status. The sampling frame of the LFS is based on the 2014 Intercensal Population and Housing Survey and is drawn using a two-stage stratified technique. Under this technique, the enumeration areas are selected proportional to the size of the two independent subsample frames (urban and rural) in the first stage, while the second stage involves a random selection of households. The sample is nationally representative for 63 provinces or cities, and can be disaggregated quarterly down to 6 major economic regions, the cities of Hanoi and Ho Chi Minh City, and rural and urban areas.
The primary mode of data collection for the LFS in Viet Nam has long been PAPI. Accordingly, ADB's technical assistance project was implemented to credibly determine the merits of transitioning to CAPI, particularly in the administration of the LFS. To achieve this, a randomized experiment was devised and implemented during Quarter 3 of LFS 2017 in Ho Chi Minh City. Specifically, 30 households in each enumeration area were randomly assigned to one of two groups: control or treatment. The control group households were administered with paper surveys, while the treatment group had interviews conducted using CAPI. The findings of the pilot experiment allowed the project team to empirically identify CAPI's relative merits in terms of data quality, cost, timeliness, and respondent perceptions. Table 1 provides data on sampling characteristics of the survey.
The experiment enlisted separate enumerators for CAPI and PAPI. The main respondent for the survey was the household head, although some household members were asked for feedback on certain questions. The CAPI arm of the experiment was implemented using CSPro for Android, which is a free platform developed by the United States Census Bureau, and widely used among national statistical systems in Asia and the Pacific.
A total of 174 enumerators were recruited, with 73 interviewing via CAPI and 101 assigned to PAPI. To the extent possible, the enumerators were randomly assigned to each group, which is reflected in the similarity of enumerator characteristics in Table 2.
Across both modes of interviewing, over 60% of the enumerators were males. The average ages of enumerators for the two groups were also very similar. Moreover, more than 75% of the enumerators across both methods had reported completing a university degree. The average number of years of enumerator experience was 5.6 years.

Sri Lanka's Agricultural Household Survey and associated experimental design
In Sri Lanka, the country's first Agricultural Household Survey (AHS) was used as the vehicle to conduct the pilot experiment under the ADB technical assistance project. The overall objectives of the experiment remained the same, i.e., to compare data quality, cost, timeliness, and respondent perceptions between CAPI and PAPI.
The AHS compiles information on agricultural production and private food grain stocks for agricultural households across Sri Lanka, simultaneously collecting relevant indicators such as cost of production, waste, farm gate prices, etc. The survey is nationally representative and provides estimates for key variables at the district level. A two-stage stratified sampling method is implemented using the census blocks identified in the 2011 Census of Population and Housing as primary sampling units (PSUs). The selection of PSUs at the first sampling stage is done using the "probability proportional to size" sampling method, wherein the selection probability given to each selection domain is proportionate to the number of housing units available within the census block. A relisting activity is undertaken prior to the conduct of the final survey to update agricultural holding information of the PSUs selected. For the second sampling stage, 10 housing units are selected systematically from each census block, resulting in a total sample size of 25,000 housing units. Within the housing units, sample households are interviewed. These households are visited twice to obtain data for the two major cropping seasons: Maha (September to March) and Yala (April to August).
The randomized experiment under the technical assistance project was restricted to Anuradhapura district only for the Maha season of 2017, wherein 10 households were randomly assigned to the treatment group (CAPI) and 10 additional households were randomly assigned to PAPI within each PSU. Table 3 provides the sampling characteristics of the Sri Lanka pilot experiment.
A total of 61 field enumerators were engaged for data collection, with 22 interviewing via CAPI and 39 assigned to PAPI for the said experiment. It is important to note that, unlike the experiment in Viet Nam, the assignment of enumerators to the two groups in Sri Lanka was not random, due to logistical challenges in the field. The PAPI enumerators had, on average, fewer years of experience in collecting survey data than their PAPI counterparts. They were also younger than their CAPI counterparts. Table 4 shows the enumerator characteristics for the CAPI and PAPI enumerators in Sri Lanka.  The relative age and inexperience of the PAPI enumerators may lead to an upward bias in the differences presented in the next section, so the results comparing CAPI and PAPI in Sri Lanka must be assessed keeping this caveat in mind. That said, both sets of enumerators in the Sri Lanka pilot experiment attended a series of training courses to get familiarized with the survey's requirements, with the only difference being that the CAPI enumerators were also trained on how to use tablets to collect data.
Two other critical distinctions are important to note between Viet Nam's LFS and Sri Lanka's AHS.
First, the 2017 LFS was fully implemented using PAPI, whereas the 2017 AHS was fully implemented using CAPI. That is, CAPI was technically the treatment in the case of Viet Nam, while PAPI was technically the treatment in Sri Lanka. Consequently, it can be argued that the PAPI sample for Viet Nam and the CAPI sample for Sri Lanka might have been prioritized by the NSOs. Given that the project management team specifically set up in each context closely monitored the field staff in the pilot areas, this possibility should not be viewed as a problem.
Second, in the case of Viet Nam, a comparison is made between clean PAPI data, which already underwent single validation by field personnel, and the CAPI data, as provided to the project team by the Viet Nam NSO. On the contrary, in the case of Sri Lanka, the PAPI data provided was raw data and had not gone through any extensive data cleaning. In both Viet Nam and Sri Lanka, the CAPI data had a range of checks programmed into the software to minimize field errors, but had not been cleaned any further. This would be akin to a good enumerator reviewing the paper questionnaire after the field data collection is completed. This distinction is important as it gives readers of this report an upper and lower bound on the extent of errors possible.

Interview duration
A weighted least squares regression was implemented to look at the impact of mode of data collection on the time taken to conduct the LFS interview in Viet Nam. Household head and other household characteristics were used as control variables, as were enumerator characteristics. The results are shown in Figure 4.
The experiment found that a switch from PAPI to CAPI reduced the interview duration by an average of 9.42 minutes. Given that the average time on PAPI for Viet Nam was 34.43 minutes, this represents a savings of 27.3% in terms of time. This provides rigorous proof on the impact of CAPI on interview duration. The data also show that household head and household characteristics had a statistically significant impact on interview duration. For example, an increase in one adult household member within the household increased the interview duration by almost 3 minutes. Age and

Figure 4: Factors Affecting Survey Duration, Viet Nam
CAPI = computer-assisted personal interviewing. Note: All variables are significant at p < 0.05 except for enumerator age that is significant at p < 0.10. Source: Asian Development Bank estimates using Viet Nam's Labor Force Survey data of Quarter 3, 2017. -. -. . -. -. -.
-. -household income were also statistically significant, but their magnitudes were significantly lower. Some enumerator characteristics, such as the sex of the enumerator, were also important. A female enumerator was likely to take 5.27 minutes less to conduct each questionnaire than was her male counterpart. The education level of the enumerator also mattered, and the results show that those with at least a college level degree were likely to be nearly 5.18 minutes quicker per questionnaire than enumerators whose highest educational attainment was from a level below college.
In Viet Nam, the LFS was fairly straightforward and targeted, which led to interviews being conducted mostly in one sitting. This made it easy to capture the interview durations as the enumerators (for both CAPI and PAPI) only needed to record the start and end times and the date. However, this was not the case in Sri Lanka. Given the complex nature of the AHS questionnaire, respondent fatigue was regularly observed and this resulted in several callbacks. In these instances, calculating the precise interview duration was very complicated. Box 3 provides a detailed account of this problem, which is why results for Sri Lanka are not provided in this report.

Box 3: The Difficulty of Measuring Time Using CAPI for Complex Surveys
Interview duration is a critical variable in survey datasets because it relates to the overall cost of implementing the survey. It also has implications for the willingness of respondents to not only participate in surveys, but also to give accurate answers.
Intuitively, long and complex questionnaires with various skips and layers contribute to respondent fatigue or burden. This is supported in the research literature, with Bradburn (1979) stating that interview duration was one of the four influential factors for respondent burden. Sharp and Frankel (1983) also found duration was the only significant variable associated with respondent burden.
Respondent burden can have a significant impact on data quality. Fricker et al. (2014) found that, when respondents believed a survey was difficult, they tended to give suboptimal answers, which in turn influenced data quality. Similarly, a study by Krosnick (1991) highlighted the possibility of respondents compromising the quality of their responses when faced with increased questionnaire burden.
The Agricultural Household Survey questionnaire in Sri Lanka was complex in nature and lengthy in design as it contained several types of rosters and sub-rosters, along with numerous modules and intricate skipping patterns. The survey was split into a household questionnaire and operator questionnaires, which would depend on the number of agricultural operators within a household. This led to respondent fatigue, particularly for households with more individuals engaged in agriculture. As such, multiple visits and callbacks to a household had to be organized.
In a situation where callbacks and multiple visits come into play, precisely identifying the total interview duration is not easy using current computer-assisted personal interviewing (CAPI) platforms. The interview information that is typically collected and transferred to headquarters for analysis on a CAPI survey includes interview start and end times, which are automatically generated by the system upon beginning and completing the interview. As a result, the total survey duration would be calculated as the difference between the start time and the end time. This is problematic because the start and end times posted by the system do not consider the stops or breaks that occur in between. This may result in erroneous interview duration figures.
In the case of Sri Lanka's Agricultural Household Survey, interview durations of more than 24 hours were recorded in some cases. The standard deviations were very high across households, irrespective of the number of agricultural operators, suggesting large discrepancies. The significant variation in time, coupled with unrealistic numbers, rendered the information on interview duration unhelpful when it came to analysis.
An indirect way to solve this issue would be to program the software to collect the time taken for each respondent to answer each question, and create a new time variable. However, under this solution, the question for which the interview was temporarily paused and then restarted would have an overstated duration. To complicate matters in areas with poor internet connectivity, as the survey instrument becomes longer, the dataset becomes larger, so the additional time information per question only makes the data file more difficult to transfer. Getting time stamps at the beginning and end of each module, and creating automated algorithms to calculate total survey duration, is another solution. However, even this might not work if a respondent requests a callback halfway through continued on next page a module. The ideal solution is to embed a timer which goes into pause mode when a questionnaire is partially saved when a survey is temporarily stopped, and is activated when the survey again picks up.
The difficulty in gathering accurate information on interview duration should be considered for future advances of CAPI platforms. The importance of reliable information on this measure cannot be overstressed as it provides critical inputs to the planning of survey operations in statistics offices, especially given its associations with respondent burden and survey costs. Some CAPI platforms have started investing in paradata, which is background information about a survey gathered while the interview is being conducted. If there were ways to transmit this background data from the field to survey managers in real time, by optimizing file sizes so as not to require high-speed internet connections, errors regarding interview durations may become a thing of the past.

Survey errors
In this report, "total survey errors" refers to the sum of four types of errors: logic, skip, missing, and validation. This cumulative variable is easy to calculate for both CAPI and PAPI surveys. Table 5 provides a definition for each type of error. Skip error This refers to errors committed when the enumerator fails to skip a subsequent question or questions, under certain conditions based on the interviewee's response. An example of a skip error would be when an enumerator did not skip questions 5, 6, and 7 when he or she was supposed to skip these questions if the respondent answered "Yes" to question 4. Missing data before the skip is not included in this classification.

Data validation error
This refers to errors committed when an input to a question is prohibited from a range or list of choices. An example of this error is when the response is restricted to 100, but the answer is encoded by the enumerator as 1,000. This does not include missing data.
Logic error This refers to errors committed based on an implemented cross-check or logic check.
For example, the enumerator coded a response as "Currently Pregnant" for a "Male" respondent.
Missing error This refers to an unanswered item that is required to be filled based on a condition from the previous question.
Source: Asian Development Bank estimates.
For CAPI, error messages were programmed into the questionnaires. If an enumerator made a mistake, a warning or error screen would be displayed. The enumerators had to address this immediately and could only override the error screen on an exceptional basis after consulting with their supervisor. In the case of PAPI, a field supervisor would go through the paper questionnaire and investigate for the same set of errors. Wherever an error was found, the enumerator would have to correct it by recontacting the particular household in each instance. Figure 5 and 6 show results that estimate the impact of the data collection method (CAPI versus PAPI) on total number of errors. There were statistically significant results in both Viet Nam and Sri Lanka. CAPI reduced the number of errors by 0.80 per interview in the case of Viet Nam and 6.2 per interview in the case of Sri Lanka. While these may look like small values, if one were to compound it to a nationally representative survey in the two economies, the total number of errors increases to 40,000 for Viet Nam and 154,000 for Sri Lanka. 2 While most household The CAPI Effect: Boosting Survey Data through Mobile Technology and enumerator characteristics did not seem to affect error patterns, an interesting result was that an increase in household size, i.e., the number of household members (in the case of Viet Nam) or agricultural workers (in the case of Sri Lanka), led to a statistically significant increase in the number of errors per interview.
In summary, the results in Figure 5 and 6 provide quantitative evidence that a switch from PAPI to CAPI reduces the total number of errors, and this is a key justification for such a switch within NSOs.

Survey costs
Implementing either CAPI or PAPI involves a combination of fixed and variable costs. The cost analysis in this report involves assessing fixed and variables costs to identify the breakeven point at which CAPI becomes more cost-effective than PAPI.
Fixed costs generally include questionnaire construction, software development, and programming of data management systems. In the context of the ADB technical assistance project, the software costs were free for both PAPI and CAPI, since CSPro (an open source platform that can be used for developing both PAPI data entry and CAPI applications) was utilized. On the other hand, programming costs were significantly higher for CAPI due to the need to set up the tablets, server, and data transfer network.
Variable costs generally include such things as purchase of tablets, data entry, printing and distribution, and miscellaneous costs (e.g., storage, bags, internet access, electricity, etc.). The most prominent variable cost under CAPI is the purchase of a sufficient number of tablets to successfully complete the entire survey. To calculate the actual expense of each tablet to the survey, the cost needs to be divided by the number of days the tablet is used in the field and the number of interviews conducted per day. Meanwhile, under PAPI, there are printing and distribution costs and data entry expenses that do not accrue for CAPI. where FC is the fixed cost and VC is the variable cost to conduct the survey. While different for the two economies (1,769 interviews for Viet Nam and 1,467 interviews for Sri Lanka), the breakeven point where CAPI becomes more feasible was found to be less than half of the estimate in the only previous study where such a systematic effort was made (Caeyers et al. 2012).

Respondent perceptions
For CAPI and PAPI surveys in both Viet Nam and Sri Lanka, households were asked a few questions about their perceptions of different aspects of the interview. The objective of these questions was to identify if the use of technology made any difference for recording survey responses with CAPI respondents compared to PAPI respondents. Table 7 presents the seven questions asked and their associated answer options. CAPI = computer-assisted personal interviewing, PAPI = paper and pen interviewing. Note: For Q6 in Sri Lanka, "A little comfortable," "Comfortable," and "Very comfortable" were merged to one category called Comfortable. Source: Asian Development Bank estimates using data from CAPI online course assessment.
In Viet Nam, no major differences 3 were found in most cases, which implies that the general population is comfortable with being interviewed using CAPI. This could be attributed to several bold technology reforms implemented in Viet Nam between 2006 and 2008, leading to its status as the developing country with the highest penetration rates for information and communication technology (Chun and Tang 2018). However, in the case of Sri Lanka, there were some differences in how CAPI respondents rated the interview process, compared to PAPI respondents. This suggests that it may be necessary to run a familiarization campaign on CAPI, prior to conducting the survey, so that households do not regard the methodology in a suspicious manner.

Summary of results
Based on the results of the experiments in Viet Nam and Sri Lanka, it is found that CAPI has a significant impact on interview times, with the caveat that this variable is not accurately captured on existing platforms when there are outliers due to callbacks and/or interviews being conducted over more than one sitting. In terms of errors, there is rigorous evidence that CAPI reduces the number of errors made relative to PAPI surveys. In terms of costs, there are also definitive breakeven points at which CAPI becomes more cost-effective than PAPI. Although the breakeven point might vary from survey to survey, it is low enough that most surveys in developing Asia would benefit from a switch to CAPI. Respondents in Viet Nam did not show any major resistance to the use of technology with CAPI, which may be attributed to the familiarity of the country's population with information technology. In Sri Lanka, however, there appears to be a need to educate the population about the use of CAPI prior to its implementation.

Training Statisticians on CAPI through Online Courses
Traditional approaches to knowledge dissemination and capacity building in statistics often entail complex logistics, the need for significant human resources, and high costs. While person-to-person training has its definite advantages, it is also important to identify ways to use technology to expand the reach of the training materials developed in any statistics capacity building project.
In the context of the ADB technical assistance project, the bank partnered with the Food and Agriculture Organization of the United Nations to maximize the impact of the project's resources beyond the three pilot economies. The result was the development of two freely accessible massive open online courses (MOOCs) on CAPI. The objective of the MOOCs was to provide hands-on training to statisticians and NSO employees around the world on how to implement a CAPI survey using two popular platforms: Survey Solutions and CSPro for Android. 4 With the MOOC approach, the reach of knowledge dissemination is limitless as the primary distribution channel is the worldwide web. Since the courses are administered remotely, students across the globe can access the content at any time within the period that the course is offered. This allows for a more flexible schedule and environment for NSO staff, who may not be able to take the course during the day, due to their work commitments. Figure 7 shows the timeline of activities for the two MOOCs. The overall development of the two courses took nearly 8 months and involved multiple steps, including: (i) drafting the course outline, (ii) creating the draft modules, (iii) assessing the course materials to determine the level of rigor and content, (iv) filming the videos for the different modules, and (v) assessing the course modules on the online platform. The enrollment of students began a few weeks before the launch of the online course. Students could enroll in one or both courses and were awarded individual certificates upon completion of each course. The two courses ran over an 8-week period from 28 January 2019 to 22 March 2019.
Each course comprised eight modules, with one new module released every week for each course. Each module comprised lecture videos, short exercises between the videos, and quizzes at the end of every module. The exercises and quizzes formed the basis for the students' assessments and grades for the courses.
The first module was identical for both courses. It introduced the concept of CAPI and discussed topics such as workflows, hardware requirements, and advantages of CAPI observed during field experience. The subsequent modules covered ways to program questionnaires, adapting survey instruments to handheld devices, integrating program flow using logic programming, and ensuring data quality through integrated data checks. Both courses also included modules to build tools for survey management, tracking, and data transfer. Table 8 shows the course structure for both CSPro and Survey Solutions.    ( ) Enrolled in both courses .

Demographics of enrollees
The gender breakdown of the enrolled students reveals some interesting patterns. More men registered for the courses than did women, although it is encouraging that almost 43% of the student body were women (Table 9). The gender breakdown was also remarkably consistent across the three cohorts (CSPro, Survey Solutions, and both courses).
Nearly 70% of all enrolled students were below the age of 40 years (Table 10). Again, this pattern was remarkably consistent across all cohorts. Nearly 8% of the sample did not reveal their age.
A majority of the enrolled students were holders of either a bachelor's degree or a master's degree (Table 11). Together, they comprised of nearly 83% of the total enrolled students. There were no significant differences across the three cohorts in terms of educational attainment of enrolled students.   More than a quarter of the enrolled students were from NSOs, while other line ministries and agencies accounted for 18.5% of the sample (Table 12). Academics comprised of over 25% of the total number of enrolled students.
Nearly 73% of the enrolled students were from Asia and the Pacific, followed by Africa, which represented nearly 18% of all enrollments (Table 13).  Less than 1% of the enrolled students were from South America, but this was somewhat expected since the courses did not offer subtitles and/or ancillary learning materials in Spanish or Portuguese.

Completion rates
Given the structure and relatively short duration of the courses, it was easy to monitor their completion rates over time. Figure 9 shows the completion rates for the eight modules across both the CSPro and Survey Solutions courses. In both, the course completion rate was nearly 40%. This may seem like a low proportion of graduates, but it is seven times the average for MOOCs globally (Christensen et al. 2013).
There was a 100% completion rate for Module 1 across both cohorts, although this is unsurprising given that motivation levels were likely higher, the module was merely an introduction to CAPI and did not require any hands-on programming or work with tablets, and the time commitment for the module was lower than for subsequent modules.
The largest drop in completion rate was observed between the first and second modules for both courses. In the case of Survey Solutions, nearly 36% of students did not complete Module 2 and beyond, while the figure was just over 38% for CSPro. This is potentially due to the increasing complexity of Module 2 in both courses. The second largest drop in completion was between Module 2 and Module 3, which can again be attributed to increasing relative difficulty.

Di erence
For Survey Solutions, those who made it through to Module 4 were highly likely to complete all the subsequent modules. A similar result was found for CSPro, although slightly more students (almost 39%) completed all eight modules in Survey Solutions compared to CSPro (37.5%).

Student perceptions
An assessment was done for both courses to obtain feedback from students about three issues: (i) clarity and organization of online lectures, (ii) usefulness of video demonstrations in helping understand the material, and (iii) opportunities for sufficient participation. All questions could be responded to on a Likert scale of 1 to 5, where 1 indicated complete disagreement with the statement and 5 indicated complete agreement. Table 14 shows the results of the responses received from the course assessment survey. More than 90% of the respondents to the assessment stated that the lectures for both courses were clear and organized. For the statement assessing the video demonstrations used in the CSPro and Survey Solutions courses, more than 90% of the respondents in both cases either completely agreed or agreed that the demonstrations helped them understand the lecture material. For the assessment of sufficient opportunity to actively participate, both courses achieved agreement ratings of around 80%. It is important to note that there are no statistically significant differences between the two platforms when it comes to assessment.

Cost assessment
The total cost of developing and conducting the two online courses was $50,000. With a total enrollment of 655 people, the cost per student was around $76. This reflects a significant cost advantage compared to classroom training, where fewer participants can be accommodated due to resource constraints. Future rollout of online CAPI training would entail only updating some course content to match the latest versions of the two platforms, and again hiring a course instructor or instructors. The course instructor could be hired to run the course exactly the same way it was conducted the first time around, or the instructor could be hired only to respond to individual questions via email and grade the final assignments (at the lowest rate). The latter option would further reduce per-capita costs, assuming that the enrollment rates remain similar to the first course intake. It might also be a more feasible option if the instructor is unavailable for a longer commitment, or if seeking extra funds to run such a course again is challenging.

Course recommendations
Several useful recommendations for the online CAPI courses were generated by analyzing the feedback provided by the enrolled students as well as by implementing algorithms to scan the student completion and performance data.
• The percentage of enrolled students completing the course remains virtually constant from Module 4 onward. The fall in completion rates from Module 1 to Module 3 is likely due to increased difficulty and the need to be able to code. Incentive mechanisms could be designed to get more students through the earlier modules of the courses. • The courses could be offered in multiple languages. This would lead to higher enrollment rates, particularly in South America. This can be facilitated through partnerships with other international organizations.
• Students generally requested more time to finish the courses due to other commitments. The courses could therefore be conducted under two structures: one with a fixed timeline (where completion of skills training is a priority) and another that is open-ended (where skills training is not time-critical). • The course discussion forum, while active, could be enhanced through features such as instant messaging and video calls. Smaller cohorts of enrolled students could also be created to facilitate more targeted discussions. These measures may assist those students who begin lagging behind in the early modules of the courses. • All modules could be available to study from the start date, so that fast learners can complete the courses earlier. This, however, runs into the risk that students who preview later modules might be intimidated by the level of programming required, and may choose not to continue their chosen course. • The cost of running updated versions of the courses is marginal and could be achieved through strategic partnerships with other agencies. • Further analysis of course graduates versus nongraduates is required to enhance the overall effectiveness of the two courses, with the obvious caveat that only data on observable characteristics can be collected. The impact of latent ability and self-motivation cannot be assessed if such an analysis were to be conducted.

Existing CAPI Platforms: A Plethora of Choice
With the increasing use of CAPI data collection, the number of software platforms designed for implementing CAPI surveys has grown. Table 15 shows the range of CAPI platforms available in 2018 and accessible globally. This range reflects the growing variety of surveys in terms of size, implementation, and complexity. It also follows a general trend by NSOs and the private sector to move from paper-based data collection to digital device data collection.
The available platforms differ in many aspects. With some, there is almost unlimited programming power and customizations. With others, which utilize a visual programming system, there is no (or limited) need to code, given their integrated front-end and back-end frameworks. Some platforms are oriented toward the Water and Sanitation Hygiene projects, while others are focused on projects where GPS mapping is of paramount importance. Even the pricing structures of the platforms varies significantly: from free platforms, to those with monthly charges, to those with charges per device, user, case collected, and/or project.
The diversification of product and growth of competition in the CAPI market is a good thing for statistics practitioners generally. The availability of a large number of platforms means that the most appropriate solution can be found for each implementing agency or project, resulting in the collection of data of the highest possible quality. Competition also breeds innovation, so it can be expected that developers will continue to incorporate more advanced features into their platforms. However, for individuals or organizations wanting to transition to CAPI data collection, choosing a platform in this crowded marketplace can be intimidating. Significant time and effort must be spent weighing up the strengths and weaknesses of the various offerings, particularly in relation to budget constraints for specific projects, to find an affordable platform that will meet their survey needs and deliver the highest possible data quality.

Helping statistics offices make the right choice
For NSOs around the world, and particularly across Asia and the Pacific, choosing the appropriate CAPI platform is of great significance. Data collected by NSOs is important for policymaking at national and international levels. The size and scope of survey projects undertaken by NSOs dwarf those of the projects undertaken by most other organizations.
The publicly available information on CAPI platforms may be of limited use to statistics practitioners and/or procurement staff within NSOs. System specification sheets vary greatly in terms of the volume of type of details they provide. Some platform developers give a comprehensive overview of the functionalities offered, while others leave it to the end-user to see what the platform can do. None offers advice on whether, and to what extent, a given package is flexible enough to support complex household surveys or more significant operations such as population and housing censuses. This complicates any attempt at a comparison of features across platforms by NSO representatives.
Another critical issue in CAPI platform assessment is the ability to capture paradata, which is increasingly becoming useful for evaluating the quality of data collected. One error source that has been studied using paradata is measurement error, or the deviation of a response from a "true" value. Paradata can be collected at a variety of levels, resulting in a complex, hierarchical data structure that can vary by the mode of data collection and the platform adopted. Not all CAPI platforms currently available have the ability to capture paradata. The issue is further complicated by how the paradata can be recorded, ranging from text files to ready-to-analyze variables.
Given the time and funding constraints of many NSOs, along with a lack of expertise to choose the most appropriate CAPI platform for their needs, ADB is conducting a comparative study of CAPI platforms, focusing on the requirements of NSOs. While a comprehensive report on the full comparison is forthcoming, this special supplement provides a snapshot of the multiple factors being considered in the comparative study. It must also be noted that the ADB study is not the first of its kind.
In 2011, the Institutional Reform and the Informal Sector (IRIS) Center at the University of Maryland, with support from the World Bank, published a report comparing software options for CAPI (Shaw et al. 2011). At the time of its release, this assessment was an excellent resource. However, the market for CAPI platforms has since expanded rapidly and developed laterally. In 2011, the norm for CAPI systems was the use of laptop computers, which suggests that most platforms were designed to run on Microsoft Windows. Only two platforms included in the IRIS study operated on Android devices. Today, due to cost and availability, the use of Android devices is the norm for CAPI data collection.
In 2016, another study was conducted to evaluate mobile survey tools for field-level monitoring and data collection for rural water and sanitation monitoring (Fisher 2016). Similarly, in 2017, the United Nations High Commissioner for Refugees published a study to benchmark mobile data collection solutions (CartOng et al. 2017). These studies, although more recent, are focused broadly on mobile data collection for rural water and sanitation monitoring and for humanitarian response operations, resulting in different considerations and platform selection criteria than those most relevant to NSOs. 5 The study being conducted by the ADB aims to provide a critical comparison of the most suitable CAPI platforms for NSO use. This comparison considers data collection via general household interviewing, and assesses platforms based on three broad areas: • availability of general features; • functionality of programming interface; and • usability for system users (interviewers, supervisors, and survey managers) To best assess these three areas, a mixed method approachinvolving a combination of desk research, experiential data from NSOs, and a usability workshop-is being employed in accordance with the steps outlined in Figure 10. The CAPI Effect: Boosting Survey Data through Mobile Technology

Essential criteria for statistics offices
An initial broad minimum criteria list was determined to decide which CAPI platforms would be considered eligible for ADB's comparative study.
Android operating system. In previous generations of CAPI data collection, platforms were used across, and supported by, a wide variety of devices. These included laptop computers, desktop computers, personal digital assistants, and early-generation tablets and smartphones. However, in recent years across developing economies, the Android operating system has achieved market domination for personal use, and is supported by a host of mobile phone and tablet manufacturers. This means that, in all developing economies, Android devices can be purchased (and repaired) easily and cost-effectively. The other major platform, Apple iOS, is not being considered for the comparative study because the sheer cost of purchasing Apple devices would be prohibitive when compared with Android devices. It should be also noted that Apple iOS has less than 14% of the total market for handheld devices in Asia. 6 Support for complex skip and validation patterns. Questionnaires conducted by NSOs generally contain a medium to high level of complexity in terms of skip patterns. For example, a questionnaire might require a question to be skipped based on the answers to several previous questions, rather than just the answer to one question. Therefore, it was determined that the CAPI platforms to be compared would need to support this level of skip complexity. Validation and logic checks, which are built to track erroneous or unexpected answers, can be flagged as they are entered into the handheld device. Being able to program complex validation or logic conditions is one of the main advantages of CAPI data collection compared with paper-based collection and thus are a key criterion for the study. 7 Adequate case management. Case management capacity is considered an essential feature for NSOs using a CAPI platform. The ability for interviewers and their supervisors to see a listing of completed cases on their tablets, before and after synchronizing, allows field teams to ensure completeness of survey cases. Support for partially completed cases is also required because, in many cases, particularly in developing economies, the respondent will be interrupted during the survey and will ask the interviewer to return at a later point in time to resume the interview. Quality control features, such as supervisor-level case checks and summary statistics across all cases, are also considered in case management.
Pricing. Platform pricing is an important factor in determining suitability for NSO use. Some platforms are offered free of charge as "global goods", meaning they are supported by one or more multilateral organizations, with the intention of improving quality of data collection worldwide. From desk research, it was identified that paid platforms varied not only in level of cost but also on how this cost is calculated, e.g. per case, per month, per user, per device, etc. NSOs in developing economies often operate on thin budgets, and large projects are usually completed with support from multilateral organizations with project-specific budgets. From the initial analysis, it was determined that the CAPI platforms requiring some form of payment offered no key benefits above and beyond the free platforms. It was therefore decided to limit the scope of the comparative study to free platforms.
Server and data security. Data confidentiality is of the utmost importance to NSOs. Any platform to be considered for comparison should ensure that data is secure, both during its transfer from tablets to the server and on the server itself. Some NSOs prefer to use their own managed physical or cloud server for data transfer and storage. This is in line with statistics laws in economies that do not permit data that has not been anonymized to leave the physical boundaries of the country. 8

Platforms selected for comparison
Following the minimum criteria, and using a short survey distributed to CAPI developers, six platforms were shortlisted for inclusion in the comparative study (Table 16).

CSPro. Developed by the United States Census Bureau and ICF
Macro, CSPro has long been the standard in data-entry software for paperbased systems. In recent years, CSPro added CAPI data collection to their suite of products. CSPro offers power and flexibility in programming as well as a strong track record in NSO work, including censuses and largescale surveys.
Open Data Kit (ODK). ODK began as a www.google.org project and is now managed by the University of Washington in the United States. It is now the standard open source data collection suite, and has laid the foundation for many other widely used platforms. In addition to being a base to create other CAPI platforms, ODK offers out-of-the-box free apps to set up a CAPI project. Kobo Toolbox. Developed by the Harvard Humanitarian Initiative, Kobo Toolbox is an open source system and is fully compatible with ODK. It was designed to assist data collection during humanitarian crisis events, which implies that it is easy to use and fast to deploy. Its backend dashboard offers customized summary reports and graphs. mWater (Solstice). Initially developed with a focus on collecting data for, and mapping of, mobile water-quality test kits, mWater has moved into CAPI data collection for surveys with the release of its sister platform, Solstice. Solstice offers flexibility as it works with Apple iOS in addition to Android and web browsers.
Smap. Developed by Smap Consulting and based on ODK, Smap offers support for Android, Apple iOS, and web browsers. It contains some features not commonly seen in CAPI platforms, such as open-end backcoding into database and task management for field teams. Survey Solutions. Starting as a collaboration between the World Bank and the Bill & Melinda Gates Foundation, Survey Solutions is now fully managed by the Development Data Group at the World Bank. It was designed to mimic the fieldwork processes of paper-based data collection, leading to flexible checking and editing of forms in the field by supervisors. The developers have recently added support for online surveys, with web browser support and email invitation management.

Next steps
Since the ideal CAPI platform should be effective at every stage of survey implementation, each platform should be evaluated taking into consideration different types of users. There is a significant knowledge gap between those with secondhand and those with firsthand understanding of all aspects of CAPI. For this reason, three different groups of CAPI users will be involved in the comparative study: CAPI developers, survey interviewers, and survey managers.
The six platforms selected will first be reviewed in depth by the comparison study authors. Insights will be gathered from hands-on use of the platforms, experiences from NSO staff, and system usability testing by trained field interviewers. A CAPI system will be designed to gain experience with how the systems perform 'under the hood'. This will include the power and flexibility of the programming environment, capacity for compilation and error testing, ease of use, and quality of support for programming queries. Once the CAPI assessment system has been developed, the platforms will be tested in a usability workshop at ADB headquarters in Manila. This testing will be performed by enumerators and supervisors with field experience working on surveys implemented by NSOs. The enumerators will use each of the six platforms to complete interviews with one another, synchronize data, and produce a quality control summary. Finally, additional evaluations will be gathered through a web survey of NSO staff who have experience using any of the six CAPI platforms at their organizations across Asia and the Pacific.
Combining data from these multiple sources will lead to a robust comparison of the CAPI platforms and will offer users a detailed analysis of the strengths and features of each. This is likely to empower NSOs to choose the optimal platform for their surveys, within the framework of their operational requirements.

Conclusion
The monitoring of SDG targets requires accurate, timely, consistent, and comparable data. NSOs play a crucial role in providing estimates for SDG indicators at the national level, and these estimates are subsequently passed on to custodian international agencies to develop global comparisons. Further, the SDGs also require disaggregation, wherever possible, across numerous statistical domains, such as location, sex, age, income, and disability (to name a few). Given the significant resource requirements of delivering on such data, which predominantly emanate from sample surveys using PAPI, there is a need to identify innovative tools and techniques that minimize data collection and management costs without compromising on data quality.
Through its use of tablets or handheld digital devices, CAPI is a tool that has the ability to transform survey data collection and management. Using inbuilt skip, missing, logic, and validation checks in the CAPI system, there is a higher likelihood of collecting more accurate data. Further, CAPI eliminates the need to encode data after the survey has ended, which significantly reduces the time and costs associated with producing a final clean dataset. Other features such as the ability to capture GPS coordinates, pictures, audio, and video make CAPI even more useful for surveying. However, there is an anticipated learning curve associated with implementing CAPI surveys, and the necessary investments should be made.
ADB implemented a technical assistance project to build the capacity of statistics practitioners on CAPI. The projected design included three major components: (i) assist three pilot countries-the Lao PDR, Sri Lanka, and Viet Nam-to implement CAPI in nationally representative surveys; (ii) conduct rigorous analysis comparing the benefits of CAPI to PAPI; and (iii) extend the benefits of the training materials developed in the technical assistance project to a wider audience through e-learning courses on CAPI. In addition, ADB is currently undertaking a comparison study of existing CAPI platforms, with the objective of providing guidance to NSOs on the features and advantages of each available platform. This study is expected to help NSOs make informed decisions on which platform might best suit their institutional needs.
The comparison of CAPI to PAPI was conducted in Sri Lanka and Viet Nam across four key themes: interview duration, survey errors, cost efficiencies, and respondent perceptions. The results build a case to scale up CAPI for all sample surveys and censuses across Asia and the Pacific. However, a widescale transition to CAPI should be complemented by sufficient training of NSO officials on all aspects of CAPI survey implementation.
ADB's online courses, delivered in the popular MOOC format and covering two widely used CAPI platforms (CSPro and Survey Solutions), provided an optimal starting point for NSOs interested in scaling up CAPI within the survey practices of their respective institutions.
Further, ADB is also conducting a comparison study of CAPI platforms to help NSOs in selecting an appropriate platform according to their needs. It is imperative that NSOs have comparable information on a number of key criteria for CAPI platforms: (i) Android operating system, (ii) support for complex skip and validation patterns, (iii) adequate case management, (iv) pricing, and (v) server and data security. Using these criteria, six CAPI platforms were selected for ADB's comprehensive comparison study: CSPro, ODK, Kobo Toolbox, mWater (Solstice), SMap, and Survey Solutions.
International organizations such as ADB play a critical role in the mainstreaming of CAPI in NSOs. Such organizations should strongly consider using CAPI for all surveys that they sponsor. The choice of platform is an important factor and, rather than each organization enforcing their own preferred platform, there should be a systematic attempt to focus on the platform already being used by the particular NSO. In addition, South-South and triangular cooperation mechanisms can be explored to implement capacity-building projects on CAPI, with organizations such as ADB playing a central role in facilitating these arrangements. The rapid evolution of technology and its associated lower costs will ultimately render CAPI as indispensable to NSOs, but the pace of adoption and the upgrading of skills depends critically on the willingness and proactiveness of developing countries to embrace such technologies.