Assessing the Impact of the Graduation Approach in the Philippines (ADB Brief 169)

1 K. Schelzig and P. Rawal. 2020. Testing the Graduation Approach in the Philippines. ADB Briefs. No. 132. Manila: Asian Development Bank. 2 According to the World Bank’s State of Economic Inclusion Report 2021, graduation (also known as economic inclusion) programs are underway in over 75 countries, reaching approximately 20 million households and benefiting nearly 92 million individuals. 3 Innovations for Poverty Action (IPA). 2020. Philippines RECOVR Survey Analysis. Unpublished. This survey covered a nationally representative sample of 1,389 phone numbers in one mobile network. In an ADB Brief produced in early 2020,1 we introduced compelling global evidence from an innovation in social protection: the graduation approach. In that brief we also outlined a pilot project in the Philippines to test different variations of this approach. We wanted to know if we could effectively build on existing government social assistance programs to reduce multidimensional poverty, diversify income sources, and build household resilience, all while reducing program costs.2 The pilot project ended in September 2020, in the midst of an unanticipated global pandemic with far-reaching impacts, especially on poor and vulnerable families. As in many other countries, the socioeconomic impacts of the coronavirus disease (COVID-19) pandemic in the Philippines were swift and deep, with reports indicating high rates of joblessness, reduction in working hours and earnings, and a rise in risky coping behaviors. In a nationwide mobile phone survey, 70% of the respondents indicated depleting their savings to pay for food, more than 20% reduced the number of meals eaten in a day, and 35% reduced the kinds of food eaten in a day.3

In an ADB Brief produced in early 2020, 1 we introduced compelling global evidence from an innovation in social protection: the graduation approach. In that brief we also outlined a pilot project in the Philippines to test different variations of this approach. We wanted to know if we could effectively build on existing government social assistance programs to reduce multidimensional poverty, diversify income sources, and build household resilience, all while reducing program costs. 2 The pilot project ended in September 2020, in the midst of an unanticipated global pandemic with far-reaching impacts, especially on poor and vulnerable families. As in many other countries, the socioeconomic impacts of the coronavirus disease  pandemic in the Philippines were swift and deep, with reports indicating high rates of joblessness, reduction in working hours and earnings, and a rise in risky coping behaviors. In a nationwide mobile phone survey, 70% of the respondents indicated depleting their savings to pay for food, more than 20% reduced the number of meals eaten in a day, and 35% reduced the kinds of food eaten in a day. 3 These adverse impacts have accelerated the search and need for evidence-driven policies and programs to protect household incomes and welfare. In this brief, we highlight the findings and results at the end of the pilot's implementation period, including promising new evidence from a September 2020 mobile phone survey of 1,243 households. The survey assessed resilience during the pandemic by comparing the experiences of pilot project households with households that only received government cash transfers.

Karin Schelzig
Principal Social Sector Specialist East Asia Department Asian Development Bank

Amir Jilani
Social Sector Economist (Young Professional) Southeast Asia Department Asian Development Bank

KEY POINTS
• From 2018 to 2020, ADB implemented a pilot project to test the graduation approach, working with 2,400 households in Negros Occidental. One objective was to find cost-effective ways to sustainably reduce poverty and build resilience to socioeconomic shocks.
• Despite severe impacts of the coronavirus disease (COVID-19) pandemic especially on poor and vulnerable households, an average of 71% of the participating households met each of the graduation criteria in September 2020.
• The graduation approach strengthened household resilience to the COVID-19 shock across a range of dimensions including financial security, food security, and mental health.
• Households that received the graduation program interventions on top of government cash transfers fared significantly better than those who received only the government cash transfers.
• Individual livelihoods coupled with group coaching proved to be an impactful variation of the traditional graduation program design. This finding could have significant implications for reducing costs.

OVERVIEW OF THE PILOT AND IMPACT EVALUATION
Based on preparatory work that began in late 2015, the graduation pilot in the Philippines ran from June 2018 to September 2020. It was implemented by the Department of Labor and Employment through its Kabuhayan (livelihood) program, which provides households with productive assets and training. The pilot targeted 1,800 beneficiaries of the Department of Social Welfare and Development's Pantawid Pamilyang Pilipino conditional cash transfer program-selected into the program in the preceding 2 years based on the national poverty targeting system. 4 The pilot participants were in 29 barangays in five municipalities of the province of Negros Occidental. The project aimed to provide participating households with a comprehensive, time-bound, and well-sequenced set of interventions tailored to individual needs to give them a big push toward sustainable and resilient livelihoods. The Asian Development Bank (ADB) engaged BRAC-an international nongovernment organization-to provide technical assistance through its Ultra-Poor Graduation Initiative. ADB underpinned the pilot with a rigorous research agenda and engaged Innovations for Poverty Action (IPA) to design and carry out a randomized control trial impact evaluation.
The interventions included (i) a one-time asset transfer for each participant, selected from a list of options identified through a rigorous market assessment in each area; (ii) coaching and life skills training covering social and health issues; (iii) core training in business and financial management; (iv) training on specific livelihoods; and (v) general mentorship by the coaches based on but in many cases extending beyond individual household development plans. Each coach supported an average of 110 households. The graduation interventions were layered on top of the government's cash transfer program benefits, paid every 2 months based on meeting health and education conditionalities. The graduation pilot thus built upon existing social assistance and livelihood programs implemented by two government agencies.
Although global evidence confirms that the graduation approach delivers robust, positive, and sustained results even long after program interventions have ended, 5 several questions remain about some of its key elements. For example, a core component of traditional graduation programs is the individual coaching and hand-holding offered to beneficiaries by trained facilitators. This is relatively costly as it is both time and human resource intensive. Similarly, though proven to be effective, the transfer of individual assets to households requires large, up-front costs that governments may not be prepared to incur if such programs are scaled up nationwide.
The ADB pilot in the Philippines was a first attempt to unravel whether group-based coaching and/or group-based asset delivery could bring costs down without reducing the impact. Participants were randomly divided into three different treatment arms of 600 households each. The first treatment arm (T1) received group livelihood assets with group coaching. The second treatment arm (T2) received individual livelihood assets with group coaching. The third treatment arm (T3) received individual livelihood assets with individual coaching. 6 A control group of 600 households received only the regular government social assistance through the cash transfer program. IPA carried out a detailed baseline survey in June 2018. 7 BRAC's innovative monitoring and evaluation system used digital technology to frequently monitor households' progress toward the graduation criteria. 8 To complement these data, an end-line survey was originally scheduled for late 2020, shortly after the completion of project activities, but pandemic lockdowns and quarantine measures made it impossible to deploy enumerators and field staff for in-person household interviews. However, ADB commissioned a rapid mobile phone survey to (i) gather real-time data on the socioeconomic impact of the pandemic on the well-being of families enrolled in the graduation pilot, and (ii) assess whether the graduation approach can help build household resilience in the event of a major socioeconomic shock. 4 After a pilot test in 2007, the Philippines' cash transfer program was launched in 2008 with 380,000 households. In a rapid, nationwide scale-up, beneficiaries were increased annually in "sets" to reach a peak of 4.4 million households (about 20% of the population) by 2014. For impact evaluation considerations, and as control for the varying length of exposure to the cash transfers, the graduation pilot participants were selected from among those who were enrolled in the program from set 8, which was not earlier than 2015. The most popular livelihood selected by participants was swine fattening (58% of households), followed by food carts (15%) and meat processing (12%). The others were chicken egg production, cosmetology and massage, carabao raising, and salted egg production. The value of assets was about $300 per household. Unpublished. This final report elaborates the four pillars and nine criteria (with indicators and rates) as follows: (i) social protection: food security (dietary diversity in the last 7 days, 86%), health behavior (use of treated drinking water, 83%), and health knowledge (retainment of correct sanitation and hygiene practices, 99%); (ii) financial inclusion: savings (able to save or use savings in the last 30 days, 73%) and access to finance (active account at a financial institution, 67%); (iii) livelihoods promotion: monthly income (at least above food threshold, 55%) and income diversification (at least two sources, excluding cash transfers, 80%); and (iv) social empowerment: reduced domestic violence (no "misunderstanding or household conflict" in the last 30 days, 98%) and joint decision-making (decisions were made with spouses, 80%). Several indicators were dropped or amended due to the pandemic, including children's attendance at school (closed due to lockdowns), attendance at government family development sessions (paused), and seeking medical treatment (due to fear of getting infected with COVID-19).

KEY RESULTS
By the end of the pilot project in September 2020, despite severe setbacks caused by the COVID-19 pandemic, the majority of participating households had still reached key benchmarks for graduation and moved closer to their social and economic goals. The pilot project's regular monitoring system found that, on average, 71% of households met each of the nine criteria under the four pillars of graduation-social protection, financial inclusion, livelihoods promotion, and social empowerment (see footnote 8). This figure represents the graduation rate of households. Improved life skills. High rates of knowledge retention from monthly life skills training (a score of at least 80%) resulted in positive behavior change in health, nutrition, and hygiene practices. This demonstrates the success of coaching techniques designed to foster active engagement, such as small group sessions (for participants in group coaching), one-on-one sessions (for participants in individual coaching), interactive hands-on activities, and visual aids to supplement the information shared and discussed. Coaches prepared easy-to-understand content tailored to participants' literacy levels, reviewed key messages covered in previous training sessions, and followed up through regular monitoring visits. The monitoring system revealed that 98% consumed vegetables and 60% consumed fruits in the last 7 days, which suggests significantly higher spending on fruits and vegetables than at baseline in 2018. This is critical for food security and nutrition targets. Another significant improvement saw 86% reporting that they only use safe drinking water, compared with 33% at baseline. Raising awareness on waterborne illnesses and use of safe water was a significant focus of the pilot project given concerning baseline data. All respondents (100%) reported having access to a hygienic toilet, compared to only 74% at baseline. This shows substantially improved sanitation and hygiene practices, which is critical both during the pandemic as well as during periods of seasonal illness and general diarrheal diseases. Nearly all (99%) retained fully or partially correct health information during the COVID-19 crisis, which reflects the influence of the project and local government health messaging.
More resilient livelihoods. Participants were also able to launch sustainable livelihoods and generate income, even during the quarantine period due to COVID-19. In response to changing market trends during the pandemic, households were encouraged to accelerate asset liquidation and to diversify to maintain productivity. Despite the situation, 73% of group livelihoods and 60% of individual livelihoods remained fully operational as of September 2020. The rest liquidated their assets and invested in other areas to respond to market changes caused by the lockdown. More than half of households (55%) reported earning at least ₱6,000 (about $125) in the last 30 days, excluding any government cash transfers. This is sufficient to keep them above the monthly food poverty threshold. 9 Better savings and financial management. Pilot participants demonstrated increased savings and positive behavior change in financial management. Just before the pandemic, 62% of households reported being able to save in the last 30 days, compared to 29% of households who had no savings at baseline. The share of households with savings had fallen to 51% in September 2020 due to the pandemic, when many households had to dip into their reserves. The rate of indebtedness increased to 60%, compared to 56% at baseline. This is likely because many participants had to rely on loans to cope during the pandemic. However, in September 2020, 69% who reported incurring a debt also reported being able to repay all or part of the loan. Only 8% of the participants reported having unforeseen expenses, which indicates improved financial management after months of training and mentoring on reducing risk.

IMPLEMENTATION CHALLENGES
The pilot project encountered several implementation challenges, including delays in asset delivery, participant attrition, challenges associated with group livelihoods, bureaucratic hurdles and political transition, and the COVID-19 crisis. While working through government programs brought clear benefits, including a high degree of ownership and a significant amount of counterpart financing, it also presented several bureaucratic hurdles. The complex government procurement process involved delays in securing approvals for livelihood proposals, particularly for group livelihoods, leading to delays in asset procurement and delivery. As a result, participants received their assets at different times. While asset distribution began in April 2019 for households in T2 (individual livelihoods) and was 58% complete by July 2019, the participants in T1 (group livelihoods) did not receive their productive assets until December 2019. Differences in how these groups coped with the impacts of the pandemic are visible in the mobile phone survey results.
Delays in asset delivery clearly contributed to the relatively high attrition rate as participants decided they could no longer wait. Of the 1,812 households that were initially targeted for the program, 1,202 remained at the project's close in September 2020. In total, 610 participants (34%) dropped out. Other reasons cited for attrition were (i) lack of time or willingness to participate in meetings and training sessions (28%), (ii) household relocation (16%), and (iii) ineligibility where the targeting verification survey found households to either be part of another livelihood program (14%) or have a better economic status (7%). A further 6% dropped out during the project due to the COVID-19 crisis. BRAC's final report identifies several design recommendations for graduation approach implementers to prevent attrition in future programs (footnote 8). 9 The government's monthly food threshold for a family of five is ₱7,528. From the baseline survey, households received an average of ₱1,628.50 per month from the cash transfer program. All beneficiary households also received additional emergency cash transfers from the government to help them cope during the pandemic-induced lockdown.
Participants in the group livelihoods treatment arm (T1) faced unique challenges, including inactive group members, high attrition during the period between group formation in December 2018 and the launch of their businesses, complex bookkeeping, and managing group dynamics. BRAC provided additional support and coaching to T1 participants to resolve these challenges. On the ADB side, implementation challenges arose from the administrative hurdles of financing assembled from four different technical assistance funds ($1.39 million in total), each with different implementation arrangements and closing dates. 10 Major start-up delays were due to political changes in the national administration. When the original implementing agency's new leadership took over, it decided to focus on other priorities and declined to participate in the pilot project (after 2 years of preparation and consensus-building).
The most significant implementation challenge, one that could not have been anticipated, was the "enhanced community quarantine" put in place from mid-March to May 2020. Field activities had to be suspended, but project staff quickly adapted to the situation to continue to support pilot participants. Strong collaboration with partners, highly trained and qualified staff, and iterative monitoring all enabled rapid project adaptations to respond to the crisis. Coaches conducted remote monitoring and activated peer check-ins, revised the core training to include a livelihoods risk management training, adjusted the coaching format to reduce the frequency of in-person coaching and the size of coaching groups, and facilitated linkages to local support resources. As a result of these interventions, 67% of households were able to continue operating their livelihoods, 89% kept their assets during the quarantine, and 99% demonstrated at least partially correct knowledge of COVID-19 prevention measures.

HOW GRADUATION BUILDS RESILIENCE: RESULTS OF THE MOBILE PHONE SURVEY
IPA researchers conducted the mobile phone survey 11 between August and September 2020 with a target sample of 1,972 households, or 82% of the 2,400 households that were assigned to one of the pilot project's treatment arms or the control group. The final survey sample comprised 1,243 households. The most common reasons for attrition included the absence of a working phone number and inability to reach respondents despite repeated attempts. Nevertheless, it was reassuring that survey response rates did not correlate with treatment status, nor did baseline characteristics differentially predict attrition across treatment arms.
Poor and vulnerable households in Negros Occidental, like in many parts of the Philippines, were struggling even before the pandemic. However, as the mobile phone survey results indicated, the pandemic set them back even further. Among control group members (those who received only government cash transfers but no graduation program interventions) since the start of the quarantine period, survey results showed that • 58% had to deplete their savings to pay for food, health care, or other expenses; • 54% had to skip required loan payments; • 46% had to borrow money they were unsure they could repay; • 89% had to purchase cheaper food; • 68% had to reduce food consumption; • 31% had to reduce their business investment; • 29% had to skip or delay a health-care visit during the lockdown; and • 10% had to send their children to work.
Households belonging to the graduation pilot treatment groups were also not immune from the impacts of the COVID-19 crisis. However, across a range of dimensions, such as financial security, food security, and mental health, treatment households fared significantly better than those in the control group.
Financial security. Although a large share (90%) of both control and treatment households reported that they could come up with money to pay off an unexpected expense of ₱5,000 within the next 30 days, paying for an emergency was "not difficult at all" or "at most somewhat difficult" for a greater proportion of respondents in all treatment groups (31%-32%) relative to the control group (25%). This may indicate a greater degree of financial resilience among graduation program participants compared to the control group. Evidence also suggests that treatment households may have been in a better financial position to cope with the impact of income losses than the control group. As shown in Figure 1, a smaller proportion of households in the treatment group were forced to skip required loan payments to pay for food, health care, and other expenses. Similarly, a lower share of the treatment households reduced food consumption, skipped and/or delayed health-care visits, purchased cheaper food, or needed to have their children work.
Food security. While a high share of households across all groups had to reduce food consumption since the quarantine started (Figure 1), treatment households consumed on average 0.53-0.91 kilograms more meat in the past 7 days relative to the control group, which consumed 2.96 kilograms. Figure 2 provides additional evidence of greater food security among treatment households compared to the control group. In the past 7 days, households in all treatment groups had fewer days in which they had to limit portion sizes at meals and reduce the number of meals. Additionally, households in T2 and T3 consumed a greater number   of meals that included meat and had fewer days in which they limited the kinds of food eaten.
Mental health. A growing body of evidence demonstrates that the loss of income and negative income shocks adversely impact mental health. 12 Research has also established a bidirectional causal relationship between poverty and mental healthdeteriorating mental health also worsens an individual's economic outcomes. Randomized trials have found that cash transfers and other poverty reduction programs reduce depression and anxiety.
Since poverty is associated with volatile income and expenditure, providing social protection to smooth shocks reduces uncertainty and feelings of depression and anxiety. The coaching and mentoring component of the graduation approach is also likely to have an impact on mental health. The mobile phone survey's mental health module found that assignment to any of the three treatment groups increased participants' standardized mental health score (PHQ-4) by 0.12-0.2 standard deviations relative to the control group. 13 Figure 3 highlights the results from the individual mental health questions and shows that households in all treatment groups were statistically less likely than those in the control group to report feeling "nervous, anxious, or on edge." Likewise, households receiving group coaching (T1 and T2) were less likely to report "having little interest or pleasure in doing things." These results suggest that group interventions may have a particularly positive effect on mental health, especially if group members experience a sense of belonging, solidarity, and social support.

KEY LEVERS OF SUCCESS AND GLOBAL SIGNIFICANCE
The graduation pilot's key levers of success can be replicated by governments wishing to build on existing social assistance and livelihood programs. These levers were instrumental in achieving the outcomes despite serious setbacks caused by the pandemic: • strong collaboration between two national government agencies, and with local government officials; • a highly capable and diverse team of full-time facilitators (coaches) comprising former government workers, social workers, and health-care personnel; • a localized market assessment to identify market-ready and shock-resistant livelihoods tailored to household skills and resources; 12 M. Ridley, G. Rao, F. Schilbach, and V. Patel. 2020. Poverty, Depression, and Anxiety: Causal Evidence and Mechanisms. Science. 370 (6522). 13 The survey used PHQ-4 (Patient Health Questionnaire-4), a four-item questionnaire answered on a four-point Likert-type scale to provide a brief and accurate measurement of depression and anxiety. The scale measures the frequency of mental health problems as follows: 1 = never, 2 = less than half the days, 3 = more than half the days, and 4 = every day.  • structured and practical hands-on training and coaching to reinforce positive behavior change and knowledge retention and to deepen comprehension among beneficiaries; • private sector partnerships with several local actors such as vendors and suppliers; and • innovative and cost-effective digital monitoring to facilitate regular and rapid assessment of challenges on the ground, enabling field staff to adapt as needed and ensure impact.
The pilot has generated several important lessons to inform the design of future government-led graduation programs-both in the Philippines and in Asia and the Pacific more broadly. Three key recommendations derive from these lessons: (i) Explore alternative options for asset transfers, including providing "assisted cash" (where participants are accompanied by a coach or other program staff member), or vouchers (that require proof of purchase from preselected vendors), to allow participants to purchase their own assets. This will reduce the burden on government procurement systems. It may also reduce participants' attrition arising from delays in asset delivery. These alternative options for procuring assets may also prove more efficient when implementing graduation programs on a larger, potentially nationwide scale. The Philippines is exploring assisted cash in the next iteration of the graduation approach, to be launched in 2021 with 3,000 households in three provinces under the auspices of the Department of Social Welfare and Development's Sustainable Livelihoods Program.
(ii) Consider group-based coaching. Individual livelihoods with group coaching proved to be an impactful variation of a traditional graduation program. This approach avoided the challenges of group-based enterprises and leveraged the benefits of individual livelihoods, such as flexibility, autonomy, and control over the livelihood and its operation, combining them with the peer-to-peer learning, sharing, and camaraderie afforded by group coaching. Although the average attendance rate among individual coaching participants in T3 at 92% was much higher than for group coaching participants in T1 and T2 at 66%, knowledge retention was found to be roughly the same across all treatment arms. This was likely a result of innovative training and monitoring techniques applied by the coaches. Crucially, groups were intentionally kept small, with a maximum of 20 participants. This finding has significant potential cost reduction implications for government training efforts. While local culture and context should always be considered, we believe the lessons from the Philippines offer an important case study for governments looking to scale up graduation programs in a cost-effective manner.
(iii) Invest in coaching and mentoring. Whether delivered individually or through groups, this pilot project confirmed that coaching was a core factor contributing to household progress in livelihoods and overall welfare. Regular checkins with participants boosted their confidence, with coaches checking whether they understood lessons on life skills or if they had any questions, discussing both personal and financial goals and plans, providing real-time feedback and support, and monitoring the well-being of household members. Coaching also ensured that households were generating sufficient revenue to cover the costs of running their business and bringing in additional income to cover household expenses. Graduation programs require competent and dedicated staff with the right skills and enough time to deliver effective coaching and mentorship support. It is critical to provide tailored training that equips coaches with the skills and knowledge needed to support poor and vulnerable households in challenging situations.
The COVID-19 pandemic has undoubtedly caused severe stress and economic hardship for all households in the pilot project. Along with the implementation challenges that delayed the delivery of livelihood assets, many households did not have a lot of time to generate sufficient knowledge, skills, and experience with their new assets and businesses before the lockdown in March 2020. Despite these challenges, we find strong evidence that the graduation program has increased households' resilience against the COVID-19 shock along several important dimensions including financial health and security, food security, and mental health. The encouraging results and insights obtained from the mobile phone survey will help to inform the in-person end-line survey planned later in 2021. By that time, we anticipate collecting richer information on household welfare, vulnerability, and resilience that will allow us to draw conclusions on the overall impact of the graduation pilot and the cost-effectiveness of the different modalities of delivering graduation interventions. Looking ahead, we hope to see a future longitudinal study that tests the longerterm sustainability of impacts.