Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 1 MERL in the Time of COVID-19: Four Case Studies Compiled and Edited by Rodney Green, Nathan Mallonee, and Kristen Check With Contributions from Rebecca Mentzer, Dr. Lincoln Lau, Kendall Wilson, Allan Kakinda, Peter Ndungu, and Dr. Subodh Kumar Accord Network organizations are facing many new challenges due to the COVID-19 pandemic. This has required rapid adaptation in many areas, including in monitoring, evaluation, research, and learning (MERL). In this paper, four case studies of Christ-centered NGOs are presented to provide a glimpse into how different organizations are adapting their MERL approaches and utilizing their internal MERL resources in new ways. The authors then reflect on these case studies and compare the approaches taken with emerging best practices from the broader international relief and development sector. In addition, the authors share guidance on how Christ-centered organizations should adapt MERL approaches according to good principles in a broader effort to discern God’s leading in the midst of challenging external circumstances, internal organizational constraints and opportunities, and an unknown future. Introduction In April 2020, the Accord Research Alliance (ARA) hosted two webinars with Accord members to discuss and learn about monitoring, evaluation, research, and learning (MERL) responses to the COVID-19 pandemic. The information gathered from the two webinars provides a good overview of the institutional context and background to the four case studies presented in this paper. The first webinar was facilitated as an open platform where participants from the ARA community shared how COVID-19 was affecting their monitoring, evaluation, research, and learning (MERL) practices within their organizations, and how they were adapting to meet the challenges and changes required by their pandemic response. There were over fifty attendees, representing organizations such as Food for the Hungry, HOPE International, Life Water, Living Water International, Hope Walks, Water Mission, 1 The Accord Research Alliance (ARA) is an interest-based Accord Network Member Alliance designed to foster collaboration. The mission of ARA is to facilitate a professional community that collectively improves how to build cultures of learning and measure what matters in Christ-centered relief, development, and advocacy through various platforms of idea sharing to encourage, connect, equip, and/or inspire. Professionals that engage with the ARA may be part of organizations that are members of the Accord Network, but it is not required. Compassion International, CURE International, Partners Worldwide, Eido Research, and others. Members shared resources with one another and discussed solutions. For the second webinar, which was open to the entire Accord Network (not just MERL practitioners), 1 ARA facilitated a discussion on how MERL can be adapted in the age of COVID-19 and serve as a powerful tool to address current realities. At the beginning of this webinar, participants were asked a series of polling questions to show how they were adjusting their MERL practices to COVID-19 realities. A majority of the forty-four participants who responded to the poll expressed they have had to adapt their approaches to data collection or evaluation (80%), while only a small percentage reported either postponing activities (14%) or that activities were still fully operational and unchanged (7%). https://accordnetwork.org/ Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 2 For those who are adapting their approaches, responses were almost evenly split on the ways in which those adaptations are taking place. The majority (74%) of participants indicated they were re-thinking their methods for collecting data during this time, likely due to lockdown restrictions and social distancing guidelines. Interestingly, 60% of participants also said they were “re-thinking the ways the M&E function can support the wider organization.” Participants were then asked to respond to the question poll with one word that best described how they were feeling about their MERL response to COVID-19. The word cloud in Figure 1 below shows the responses, with the larger words being mentioned more frequently. The sentiments expressed most frequently were of uncertainty, being overwhelmed, and exhaustion, though there also were more positive statements around refocusing, hope, curiosity, and opportunity. Figure 1: Word Cloud of Feelings Regarding MERL Response to COVID-19 The feedback from the participants in the webinars indicated that, first, most organizations are adapting quickly to COVID-19 and are rapidly re-thinking many of their standard approaches to MERL. Also, while there is an undercurrent of optimism and hope, many staff of Accord Network and ARA organizations are also feeling overwhelmed or uncertain about the best approaches or paths forward. In an effort to provide some knowledge sharing and emerging best practices, the rest of this paper shares four case studies and additional guidance from the steering committee of the ARA for how to use MERL tools in ways that allow organizations to discern God’s direction and navigate these uncertain times. Organizational Responses Within the ARA, members are responding to the pandemic with the overall goals of continuing to learn about beneficiary and community needs, seeking to improve and learn from ever-changing program delivery models necessary in this VUCA context (volatile, uncertain, complex, and ambiguous), and, in some cases, even re-envisioning the role of MERL within their organizations. Faith is a driving force pushing many Christ- centered organizations and local faith partners to gather quality data so that they can improve their work and serve vulnerable populations more effectively. The use of data is part of a larger framework within decision- making to discern God's direction. A helpful organizational metaphor for discernment is the sailboat, because it requires action and responsibility with a reliance and sensitivity to the wind that actually moves the boat. In the same vein, as part of regular activities, spiritual practices embedded within organizational practices acknowledge a dependence on God's wisdom and guidance. This fosters a heart posture of humility, Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 3 "for unless the Lord builds the house, its builders labor in vain" (Ps.127:1). 2 Below are four case studies from three Accord Network organizations and one ARA organization that are adapting MERL practices during this time. HOPE International, an organization that has long emphasized the principle and practice of listening, shares why listening to the field is one of the first things to consider doing during this time, and how they have adapted their listening practices for this unique season. For more information on how to improve your organization’s listening practices during COVID-19, visit Feedbacklabs.org: “Tips for Improving your Feedback Practice during COVID-19”. Many organizations are switching to phone surveys or other mobile platforms in order to continue collecting data for program monitoring and evaluation or for conducting research. International Care Ministries (ICM) is one organization that is rethinking ‘business as usual’ for their data collection during COVID-19. In order to better communicate with their network of pastors and churches that serve vulnerable families, ICM further developed a Facebook messenger chat box, a tool that was already widely used. Many resources for making these adaptations exist, such as this Q&A on Mobile Phone Surveys from GeoPoll. There are also organizations looking for guidance on whether and how to carry out MERL activities safely in the context of COVID-19. Compassion International relies heavily on the best MERL practices they had established before the pandemic in order to effectively monitor and evaluate new interventions in the COVID- 19 era. Their case study offers an example of how principles and best practices are especially important in times of crisis to “discern” God’s direction. Finally, a number of organizations are repurposing their MERL data and/or staff. The case study shared by Food for the Hungry illustrates how a crisis can prompt an organization to re-purpose their MERL staff and practices to engage in different types of activities, such as big data or machine learning, in order more effectively to make decisions, move the organization forward, and serve vulnerable people. Big data provide new opportunities for organizations engaging in relief, development, or advocacy, as outlined in the World Economic Forum report (2012) “Big Data, Big Impact: New Possibilities for International Development.” 2 For principles and best practices on Christ-centered MERL, see Check, Green, and Kumar (2020). For a helpful resource on monitoring and evaluation guidance during COVID-19, see Catholic Relief Services (CRS) (2020). For a pithy overview of the importance of discernment for organizations, see Mellado (2019). Case Study 1: HOPE International The Mission HOPE International invests in the dreams of families in the world’s underserved communities as they proclaim and live the gospel. In sixteen different countries, HOPE shares the hope of Christ through biblically-based training, savings services, and loans that restore dignity and break the cycle of poverty. The Challenge Almost all countries where HOPE and HOPE’s partners operate have experienced significant disruptions to operations due to COVID-19. During lockdowns and restrictions on gatherings, many savings groups adapted how they met to abide by social distancing measures, and some stopped meeting altogether. Offices closed and transitioned to remote work, and field visits were limited. Staff and volunteers innovated in creative ways to stay in touch with clients and members. Many of the businesses of microfinance clients were shut down during government-imposed lockdowns. Programmatic Response As a development organization, HOPE savings group programs pivoted to supporting immediate relief through local church partners, while microfinance institutions demonstrated care for clients by extending grace periods and rescheduling loan terms for affected clients. Programs also developed new loan products to help clients’ businesses recover from the economic shock. Through these and additional efforts, HOPE has remained committed to rebuilding dreams together with churches and partners. MERL Adaptations HOPE’s core listening and evaluation tools and processes typically focus on better understanding stakeholder experience as defined by engagement and impact. HOPE believes listening is critical both as a posture and as a practice. HOPE paused existing activities to instead focus on the most relevant questions in this season and adapting the means of gathering information, both of which shifted significantly in light of COVID-19. The initial priority was listening to understand how clients and members were being impacted, how they were responding to the situation, and what needs they had. By listening both presently and proactively, HOPE gained information to help understand immediate relief and future recovery needs. A key aspect was ensuring https://feedbacklabs.org/blog/tips-for-improving-your-feedback-practice-during-covid-19/ https://feedbacklabs.org/blog/tips-for-improving-your-feedback-practice-during-covid-19/ https://www.geopoll.com/blog/frequently-asked-questions-mobile-surveys-faq/ http://www3.weforum.org/docs/WEF_TC_MFS_BigDataBigImpact_Briefing_2012.pdf http://www3.weforum.org/docs/WEF_TC_MFS_BigDataBigImpact_Briefing_2012.pdf https://www.hopeinternational.org/ Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 4 information was accessible and analyzed quickly to rapidly inform strategy. Adapting the process for savings groups: Field visits are the typical method to gather information from savings groups. With most field visits cancelled, HOPE designed an entirely new monitoring system. A short set of questions was developed to understand if groups were still meeting, how members were supporting one another, and what their areas of need were. These questions were asked by telephone through the ministry structure. Responses were entered into Microsoft Forms for rapid analysis by the local savings group program staff, who utilized the automatic data visualization within the platform and conducted further segmentation of results in Excel. Of over 5,000 savings groups surveyed, 85% reported that they continued meeting, many in an adapted way to abide by social distancing guidelines. In response to requests for guidance from the savings groups, HOPE developed and rolled out a toolkit to help groups adapt their meetings to adhere to best practices for group gatherings, social distancing, and hygiene during COVID-19. Adapting the process for microfinance institutions: A client impact monitoring tool was developed to understand the effects of COVID-19 on clients’ businesses. This included asking questions around whether the business was open during a government- imposed lockdown, exploring changes in income and expenses, and discussing the anticipated level of difficulty in reopening. Programs used either Microsoft Forms or Excel for data entry. In Excel, a data entry template with drop-down options for multiple choice questions and conditional formatting was developed to automatically score responses for an overall impact rating. PivotTables in the spreadsheet, needing only to be refreshed, enabled local decisionmakers to access information quickly. With feedback from over 11,600 entrepreneurs, HOPE learned that the majority of clients surveyed were not able to operate their business during country-wide lockdowns. For those who were able to continue operating, many experienced decreases in income and/or increases in expenses. For example, in Rwanda, 72% of surveyed clients were not able to operate their business during the lockdown. Of the 28% who were able to continue operating, 79% experienced decreases in sales and 41% experienced increases in expenses. This information was used to adapt budget expectations and better understand the need for recovery lending. Rather than hiring third-party enumerators for independent listening, HOPE leveraged existing relationships and listening as an opportunity to reinforce and deepen relationships. Loan officers were calling clients they support, and church volunteers were calling savings groups they support. In this sense, the listening, monitoring, and evaluation team has expanded exponentially as many staff across the globe are seeing the integral role the team continues to play in listening well. In summary, HOPE shifted its focus to answering the most relevant questions in this season — namely around understanding current realities, opportunities, and needs — and built tools and processes to listen and respond quickly. In an ever-evolving situation, relationships are critical, and timeliness essential in ensuring responses that are relevant and effective. The listening, monitoring, and evaluation team views this pandemic as a culture-shaping opportunity, where the and role of listening, monitoring, and evaluation are vital to providing relevant services in a changing world. HOPE has always believed that listening is a core part of the ministry, not just an evaluation of the ministry. Listening is an opportunity to love our neighbors. HOPE listens to care for clients and members and support them in these challenging times. With over sixteen thousand responses gathered, HOPE was equipped to actively respond at program, regional, and network levels, and continues to proactively listen as lockdowns in many countries have lifted to continue building dreams together with clients and members. Case Study 2: International Care Ministries (ICM) The Mission International Care Ministries’ core program, Transform, is targeted at households living in extreme poverty and offered in the Philippines and Uganda. It is a fifteen-week program that delivers a values, health, and livelihoods curriculum, in addition to a variety of integrated interventions that participants can receive depending on need. Each program serves around thirty families at a time and is implemented in partnership with local pastors and churches. ICM also provides the support to maintain a network of over ten thousand pastors called Thrive. Through these partnerships, Transform has reached over 1 million family members in the last ten years, and the evaluations have shown significant improvements in household savings, income, health, and psychosocial indicators. The Challenge As COVID-19 emerged and the Philippines implemented strict community quarantine measures, Transform had to be paused due to limits on gatherings. Yet communities were still in need of spiritual, nutritional, and medical assistance. https://www.caremin.com/ Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 5 Programmatic Response ICM has shifted from Transform to program interventions that deliver immediate aid and information during the acute phase of COVID-19, and indirectly support or direct organizational resources to newly identified needs and gaps among beneficiary communities. Articles and current events are frequently posted to widely utilized social media platforms in the Philippines. For example, a basic version of Facebook messenger can be used in the country without a data plan. Based on the extensive use of Facebook, ICM initiated the Thrive Network Chatbot to expand ICM’s reach within Thrive. The chatbot consists of pre- programmed answers and information that users can interact with, depending on their responses to specific questions. This format increases user engagement, allows information to be focused towards individuals meeting a specific profile, and has exciting options for automated follow-up, etc. ICM has also developed and distributed other materials, including a video (on the ICM website), a COVID-19 Information flyer (given to pastors) for infection prevention and safety, as well as health guidelines for churches. Individuals with COVID-19 infections in need of hospitalization and inpatient care can access ICM staff for medical and financial support. In partnership with a number of universities and the Canadian Government, ICM is co-leading the development of infection prevention and control guidelines for healthcare workers in hospitals and primary care settings (Manilla Bulletin 2020 and “ICM Healthcare…”), and is working to understand which communities are not being provided food and access to medical treatment so ICM can assist in alleviating these needs, or connect communities to resources. MERL Adaptations Monitoring, evaluation, research, and learning of ICM’s Transform and concurrent programs usually consists of in-person pre- and post-surveys to predominantly measure increases or decreases in early- childhood education aptitude, income and savings, hope and self-worth, and illnesses and malnutrition. With COVID-19, ICM immediately embedded a COVID-19 knowledge, attitudes and practice module into scheduled surveys (Lau, et al. 2020), as well as shifting focus to monitoring the Thrive Network Chatbot through linked working dashboards. As part of this process, ICM ensures that registrants are sent devotional material and answers to COVID-19, and that food supplies are given out after requests have been evaluated by staff. ICM has a current project that solely utilizes phone call surveys to learn how COVID-19 influences communities on a weekly basis. This project also has linked working dashboards and will again depict which communities require food provisions and will show whether future Transform surveys could occur over the phone. ICM has brought over 10 million meals to communities in dire need (“Delivering…” 2020) and is tracking food, medical and other needs in communities through the Thrive Network Chatbot and phone surveys. ICM assists with mobile phone costs for community leaders in order to maintain communication and enable continual utilization of the Thrive Network Chatbot and engagement in phone surveys. ICM is continually working with the strategy and implementation teams to determine the appropriate time for the Transform program to restart, or more likely, a variant of the program which decreases social contact and risk. ICM is also looking to develop new mediums, through videos and/or a chatbots, as a programs and strategies must be adapted post-COVID- 19. Case Study 3: Compassion International The Mission Compassion International is a Christ-centered and holistic child and youth development organization that partners with networks of over eight thousand churches in twenty-five countries to serve over two million vulnerable children through sponsorship. The mission of Compassion is to release children from poverty in Jesus’ name. The Challenge The key challenge for Compassion International in the midst of COVID-19 is how to continue holistic child development programs when children are not physically attending church-based or center-based programming. In addition, Compassion International has suspended all travel, and staff across the globe are working from home. In Kenya specifically, government restrictions to reduce the spread of COVID-19 put strains on a food security situation that was already challenging (“Why COVID-19…” 2020). Additionally, the government banned food distribution on April 12 after there was a stampede (“Kenya Bans…” 2020), and the Ministry of Health advised the use of digital payment methods instead of physical cash (“Corona Virus – Kenya…” 2020). These contextual factors provide a unique and difficult challenge to Compassion Kenya’s National Leadership in addition to challenges being faced at a global level. Programmatic Response In order to continue supporting church partners that are serving children, but without face-to-face interaction and while working remotely, modes of communication have been adapted, at least where there is internet capacity, to include more phone calls, https://www.compassion.com/ Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 6 WhatsApp communications, and Zoom calls. National Offices and church partners are adapting center-based program activities to home-based programming with children/youth and focusing their efforts on child protection, household security, spiritual and emotional counseling, health needs, and engaging beneficiaries and families in their new contexts. Interventions by church partners in the face of COVID-19 have included food and hygiene kit distribution, unconditional cash transfers, and print/digital forms of content delivery or spiritual/emotional counseling to beneficiary families. The new communication system, which has included WhatsApp or social media, where available, or radio where beneficiary phone access is limited, is being used to supplement government health messaging and deliver adapted versions of Compassion’s holistic child/youth development curriculum. In response to unique contextual factors in Kenya, the Compassion Kenya leadership implemented unconditional cash transfers (UCTs) through mobile money, agency banking, and other platforms to support income stabilization and food security among the most vulnerable beneficiary families, as selected by local church partners. Compassion was able quickly to pivot to this intervention due to the large body of evidence from randomized control trials and impact studies of UCTs in Kenya in recent years, as well as evidence from other organizations, such as UNICEF, that recommend cash transfers when markets are functioning and food and other essential goods are available (Gore and Patel 2006). MERL Adaptations Many normal MERL activities that depended upon face-to-face data collection methods were paused and postponed, including impact evaluations, process evaluations, and feedback mechanisms. Still, Compassion is specifically implementing MERL for several of the COVID-19 related interventions to ensure the organization can continuously improve. Compassion has leaned heavily on its MERL standards and specifically the “CARR” principles of data: Credibility, Actionability, Responsibility, and Relevance 3 in order increasingly to learn how to discern God’s direction during this season of COVID-19 response. A MERL toolkit and other resources were created as well to help encourage the usage of standardized indicators and questions for similar interventions, as well as establish consistent principles and effective practices with sufficient room for country contextualization, flexibility, and speed of implementation. 3 Slightly adapted from the Goldilocks Challenge CART Principles (Karlan & Gugerty, 2018). The goal of the evaluation on the UCTs in Kenya is to learn how livelihoods and food security situations are changing among beneficiary families as the transfers began and after the transfers finish. In addition, the evaluation will foster learning on how the cash was used, how satisfied the families were, whether there were unintended consequences (such as putting stress on relationships or domestic violence), and how the process could be improved. The principles of “Actionability” and “Relevance,” and a utilization- focused approach (“Utilization-Focused Evaluation” n.d.) were also applied in the logical framework and survey design phase by soliciting feedback from national office staff who would be using the results. This procedure highlighted the priority and interest in learning about the process of the cash transfers and how it could be improved. In order to apply the CARR principle of “Responsibility” and the ethic of “do no harm,” phone- based surveys have been employed in place of traditional face-to-face data collection (Mani and Barooah 2020). This allows ongoing data collection while mitigating the risk of spreading COVID-19. Phone surveys should only be utilized when information is essential for the ongoing learning and effectiveness of programming, especially new programming in Compassion such as the UCTs in Kenya. In addition, health messages are incorporated in phone surveys, especially for those who are not able to charge their phones in the home and may have to put themselves at risk of spreading COVID-19 by having to charge their battery more often as a result of spending battery life to answer the survey. Informed consent was obtained, and respondents were fully aware of the risks of charging their phone outside the home. Food security was a concern by Compassion Kenya’s National Office leadership in April 2020, especially among those most vulnerable to the economic impacts of COVID-19 and government restrictions (“COVID-19 Restrictions…” 2020), so the Household Food Insecurity Access scale (HFIAS) was used in both baseline and end-line surveys as part of the evaluation plan (Coates, Swindale, and Bilinsky 2007). The HFIAS is a widely used scale that helps apply the principle of “Credibility” and is also brief enough to be useful in a phone survey design. The ideal process would have been to conduct a baseline as part of the targeting procedure weeks before the first transfer, and to utilize recall periods of 7-days for food and market functionality related questions. With a 30-day recall period, however, the HFIAS fit within the urgent time https://www.forchildren.com/discover/monitoring-and-evaluation/monitoring-evaluation-research-and-learning-merl-standards http://www.forchildren.com/crisis-resources/ Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 7 frame of conducting a proxy baseline in tandem with the first transfers. 4 To apply the principle of data “Credibility,” four enumerators who worked with Compassion Kenya in a previous study were hired and trained to conduct phone surveys from their homes. They were trained over Zoom by the National Office M&E Specialist. A sampling methodology that combined randomization with stratified systematic sampling was employed to select beneficiaries to interview, ensure representativeness among urban and rural populations, and to account for an estimated 80% response rate with a maximum of 5 calls per household. Cash transfers were made weekly and the sampling strategy was developed to accommodate weekly updates of beneficiary lists with contact information made by churches who filled out Qualtrics forms for every transfer they completed to selected vulnerable households. As most transfers were done via mobile phone, the full target population was accessible by phone in principle and bias due to non-inclusion of those without phones was not really a concern. The enumerators recorded the number of calls to reach each respondent as part of their data collection procedure in order to help build stronger estimates for future sampling frames. The Kenya National Office M&E Specialist conducts back checks to provide accountability for the enumerators. This involves calling a small random sample of beneficiary families to confirm that they had been contacted by the enumerators. Over a 4-week period in April-May 2020, the enumerators collected data from 1,369 respondents, which was above the sampling requirement. 99% of respondents selected that they would be willing to be contacted again for the end-line survey, which will help save time in conducting the sampling for the end-line which will take place in August 2020. The response rate for the baseline was 81% with an average of 2.5 calls per beneficiary household to complete the survey. This result fit the sampling frame to ensure representativeness. In addition, 5% of households contacted had not yet received the cash transfer, which provides useful information to the Kenya leadership team for follow up and improvement for subsequent cash transfers. The results were analyzed and visualized using Excel and Tableau by the global and Kenya MERL teams. The baseline and end-line will be analyzed using Stata in order to investigate relationships between various variables using correlation or regression 4 See Innovations for Poverty Action’s RECOVR Hub for questionnaire examples: https://docs.google.com/spreadsheets/d/1HKFFWLhH6ynMhqA2NwO3LGVVYKealZDKKi6RPGnYdX8/edit#gid= 1432795253 analysis. Results on the process and outcome variables were visualized and presented to the Kenya leadership teams in order to inform follow up and improving programming. One alarming result from the baseline was that over 95% of the respondent population were moderately to severely food insecure at the time taking the survey, with over 81% of the population in the severe category. On top of that, there 29% of the respondent population was not only food insecure but was also facing food inaccessibility. In addition, 90% of urban respondent families reported missing a meal in the previous 4 weeks, whereas the Kenya Population Council’s Knowledge, Attitudes, and Practice Survey also conducted in April 2020 in five Nairobi informal settlements, reported that 68% of families had reported missing a meal in the previous 2 weeks (Population Council, 2020). This indicated that the urban beneficiaries targeted by the UCT were potentially more vulnerable than the general urban population. The results also showed a high degree of food insecurity among those with families with six or more members and who reported food inaccessibility. Specific plans were used to follow up with the church partners and regions where this challenge was more prevalent. The results of the baseline will also provide immediate and actionable feedback, to apply the “actionability” principle, on the speed and ease to which the transfers were received, and whether there were any challenges, in order to make real-time improvements in implementing UCTs. The results from the end-line will help inform decisions regarding the continuity of the intervention as well as discerning improvements to future interventions to support vulnerable households with income stabilization and food security. Case Study 4: Food for the Hungry The Mission Food for the Hungry (FH) is a Christian humanitarian organization that began in 1971 and now works in more than three thousand communities, reaching out to two million people in twenty countries in Africa, Asia, and Latin America and the Caribbean, responding to human suffering through relief and development. FH’s stated purpose is “Together we follow God’s call responding to human suffering and graduating communities from extreme poverty” (https://www.fh.org/about/values-vision-purpose/). FH implements its work with the core intervention domains https://docs.google.com/spreadsheets/d/1HKFFWLhH6ynMhqA2NwO3LGVVYKealZDKKi6RPGnYdX8/edit#gid=1432795253 https://docs.google.com/spreadsheets/d/1HKFFWLhH6ynMhqA2NwO3LGVVYKealZDKKi6RPGnYdX8/edit#gid=1432795253 https://www.fh.org/ https://www.fh.org/about/values-vision-purpose/ Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 8 of disaster risk reduction, education, food security and livelihoods, gender and child protection, health, leadership, and change in worldview. In 2019, FH's three thousand staff members impacted the lives of over twelve million children, women, and men. The Challenge As the COVID-19 pandemic unfolded in all the countries where FH works in March 2020, the normal programming of FH was interrupted due to lockdowns announced by governments around the world. Due to the elevation of the risk level because of the increasing number of cases, FH had to put in place the work from home order in alignment with many national governments of all countries where FH operates. FH also had to impose a travel embargo due either to the closure of country borders or to travelling restrictions in those countries. Financially, FH had to put in place a hiring freeze and freeze current pay rates in anticipation that there might be some challenges in the organizational cash flow. All discretionary spending was scrutinized and filtered through two questions: a. Is this essential, b. Is this essential NOW? FH had to address the interruption due to COVID 19 at the organizational level and at the program level. The response at the organizational level was more strategic in nature, while at programmatic level it was to meet the needs of the most vulnerable who were impacted by the disruption due to COVID 19. This case study will confine itself only to the organizational response. Organizational Response During the initial stages of COVID 19, FH constituted a Crisis Management Team (CMT) that met daily to monitor the situation globally and in each country where FH works. The CMT was responsible for decisions such as closing the offices, working from home, and travel restrictions, as well as providing approvals for any community level engagement such as food distribution. After only one month, the CMT was dissolved, because global data were making it increasingly clear that a new era will be defined by a fundamental schism: the period before COVID-19 and the new normal that is emerging in the post-viral era: the “next normal.” In this unprecedented new reality, there is occurring a dramatic restructuring of the economic and social order in which humanitarian organizations have traditionally operated. At the organizational level, FH started seeking answers to the question being posed by leaders across the public, private, and social sectors: What will it take to navigate this crisis now that our traditional metrics and assumptions have been rendered irrelevant? To navigate this disruption, FH determined five stages (also called 5Rs) as a strategic move that will lead FH from the crisis of today into the new reality: Resolve, Resilience, Return, Reimagination, and Reform. Resolve: to address the immediate challenges COVID-19 presents to staff and communities where we serve Resilience: to address near-term organizational cash management challenges and broader resiliency issues Return: to create a detailed plan to return the organization back to scale quickly Reimagination: to re-imagine the “next normal”— what a discontinuous shift looks like, and with implications for how FH should reinvent Reform: to imagine how the environment in the industry (regulations, role of government) might evolve. The 5Rs are a strategic response, for which FH needed implementation teams. Different teams were repurposed under an overarching “Team of teams” (also called 5Ds), with clear roles, responsibilities, and decision authority. These 5D teams are expected to continue their work for the next few years to address the 5Rs. The 5D teams are as follows: Discover: The Scenario Planning Team maintains multiple scenarios; it provides one planning scenario and facilitates future state exercises. Design: The Strategic Moves Team uses planning assumptions (& scenarios) to craft trigger-based portfolios of strategic moves. Decide: The Integrated Operations Team maintains operating cadence, risk maps, and situation reports, while also tracking progress and ensuring ownership. Deliver: The Implementation Team ensures extreme clarity and builds a cross- functional team to achieve outcomes. Data: The Data Team supports all other D teams to analyze data at different levels from different sources and works cross functionally with all the D teams to provide insights based on data and pathways to better functionality. Finally, the Executive Leadership Team serves as a “nerve center” to ensure speed without sacrificing decision quality across these five dimensions (5Rs and 5Ds). As FH considers the scale of change that the Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 9 coronavirus has engendered and will continue to engender in months and, possibly, years ahead, FH feels compelled to reflect not only on a major health crisis, but also on an imminent restructuring of the global economic order. How exactly this crisis evolves remains to be seen, but the five stages described here offered FH leaders and staff a clear path to begin navigating to the next normal, a normal that, so far, looks very different from anything in preceding years. MERL Adaptations MERL adaptations happened at two levels in FH. The Learning and Evaluation (L&E) Team had a mandate to support MERL at the Country Programs level while the Data Team, constituted as part of MERL repurposing efforts, was mandated to support other D teams by providing them data and evidence to enable them to accomplish their roles and responsibilities around the 5R strategy. As part of this mandate, the Data Team developed a Pandemic Risk Monitoring Dashboard. The intention for this dashboard is for each country in conjunction with the regional offices to understand the risk that coronavirus poses and to adopt appropriate public health recommendations across all categories of risk to mitigate the exposure to COVID- 19 for staff and beneficiaries. If the risk is HIGH, stricter precautions are advised than if risk is LOW. Two indicators of change (active cases in the last fourteen days and deviations from the peak) were monitored daily and risk status was updated weekly. To track the peak, the data team built the epidemiological data model to predict the number of cases based on the SEIR model (susceptible, exposed, infected, and recovered). The spread of infection is monitored through test positivity rate, while the pandemic reproductive rate is calculated based on mitigation measures, the testing capacity for each country, and the severity of the pandemic (the days it takes to double the number of deaths). This risk monitoring enables the country offices to take decisions on return to work or to resume field-level activities. The data team also monitors the key economic indicators at the global and country level to advise the leadership on how the economic scenario is unfolding and what impact the organization can anticipate. At the program level, all face to face data collection was put on hold as entire field operations were disrupted. The Learning and Evaluation Team, in partnership with the Relief and Humanitarian Assistance Team (RHA), which was steering the global pivoting for COVID 19, developed the Pandemic Impact Remote Survey (PIRMS). The Pandemic Impact Remote Monitoring Survey (PIRMS) is an open source, rapid, replicable assessment tool used to monitor the impacts of the COVID-19 pandemic on affected communities. This data collection tool is not a formative research (diagnostic) tool, nor should it be confused with Knowledge, Practice and Coverage (KPC) or Knowledge, Attitude and Practice (KAP) surveys, which are generally used for program monitoring. Rather, the PIRMS consists of fourteen questions that investigate respondents’ knowledge of the virus and the changes they have noticed in their communities since the onset of the pandemic (e.g. access to services, changes in movement, availability of resources). The survey takes about ten to fifteen minutes to complete and is repeated in the same geographic area every fifteen days until conditions stabilize. It is not necessary to engage the same respondents each time the survey is administered. Also, the interviewer may select any remote tool (e.g. mobile phone, WhatsApp) to administer the survey, but responses are collected in remote data collection tools such as Open Data Kit (ODK). Stakeholders, such as community leaders, local and regional government offices, and implementing partners should be considered for data dissemination. Data can be presented in a manner that is appropriate for each stakeholder. Based on the results, partners, alongside national stakeholders can use data to: ● Initiate more detailed needs assessments in specific sectors and locations to design interventions and develop funding proposals. ● Help fill information gaps expressed by donors and humanitarian agencies on community-level conditions and develop strong cases for targeted funding responses. ● Engage partners to generate and share more data to gain a better understanding at broader levels (national, regional, and perhaps even global levels). The results of PIRMS can be accessed by visiting the PIRMS google site. Analyses are presented as snapshots for any time interval, as well as trend and cross analysis. The PIRMS contains an interactive dashboard that helps country programs to gain insights from their data and adapt programs to address emerging needs due to COVID-19. One key result from PIRMS thus far is that there has been a widespread and significant decline of income due to COVID-19. Respondents who indicated that they have witnessed or experienced a decline in income have higher prevalence of anxiety and instances of abuse. The survey also found that an increase in anxiety and abuse resulted in fewer children participating in home learning. Findings and insights through this PIRMS will enable Country Program to pivot in response to arising needs as they plan for next year. One more tool FH designed and tested is a Rapid Remote Assessment (RRA) tool, a formative research tool to inform program pivots. Data is collected through telephone interviews. A resource guide was developed https://sites.google.com/fh.org/pandemic-impact-survey/ https://sites.google.com/fh.org/pandemic-impact-survey/ https://sites.google.com/fh.org/pandemic-impact-survey/ https://sites.google.com/fh.org/pandemic-impact-survey/ https://sites.google.com/fh.org/pandemic-impact-survey/ Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 10 that explains the purpose, objectives, methodology, and scope of the tool. This resource will be used in virtual training for personnel who will collaborate in evaluations. In conclusion, through the repurposing of Monitoring and Evaluation in FH, the L&E and Data teams have been able to support FH’s response to COVID-19 by providing data and enabling evidence- based pivoting. This repurposing also brought new capabilities to the organization that will serve the mission in the long-term, such as predictive and prescriptive analysis that enable program pivoting. While the L&E team supports Country Offices through remote data collection and informing the need for pivoting, the Data team takes a more strategic approach by relying on “big data” and machine learning. Many global websites have large and complex data sets on COVID 19. These and Economic Indicator data permit the Data team to develop analytical models for decision making. These global data sets are so voluminous that traditional data processing software cannot manage them. Other engines have been created for these massive volumes of data that are now available to address business problems in new ways. This is a new venture for FH and every day requires learning. The need to put all program evaluations on hold due to COVID-19 provided the space necessary to develop these capabilities for advanced analysis and visualization that would otherwise not have been possible. These examples of repurposed MERL in FH will enable meeting short-term and long-term challenges in an uncertain future. Conclusions The Accord member organizations featured in this paper offer a glimpse into the different postures, principles, and practices that have informed MERL approaches in the midst of the global COVID-19 pandemic. Each has had to make quick decisions to respond to different contexts and challenges posed by the pandemic, and each offer lessons learned to consider for MERL into the future. In addition, the cases provide inspiration on how organizations can emerge from the pandemic with stronger programs and approaches to MERL as a result of the innovation and flexibility necessary for implementing development programs in a volatile setting. While these four cases are a small subset of a much larger network and industry, the cases reviewed together offer some unique insights for organizations seeking to utilize MERL in order to serve people even more effectively in years ahead. 1. MERL is more important now than ever – but it must deliver As many organizations adapt their programs and pilot new initiatives in response to the global pandemic, and other emerging challenges, MERL offers the ability to listen to communities, stay relevant, prioritize learning, continue improving, and build trust and relationship. At the same time, the stakes are high for MERL to deliver useful data and provide value. For some organizations, there may be a temptation to move too quickly in response to crisis and fail to prioritize appropriate MERL design and investment. Michael Quinn Patton, former president of the American Evaluation Association and evaluation expert, challenges evaluation professionals to “prepare to make the case for evaluation’s value…prepare by working now to make evaluation all the more useful and real-time data essential so that the evaluation value proposition reframes evaluation as an essential activity not as a mundane bureaucratic or luxurious function when times are good” (Patton 2020). 2. MERL must keep adapting In a global pandemic, business as usual for MERL was completely disrupted. In all four organizations, regular planned MERL activities were postponed. In a matter of a few days, not only were the approaches no longer relevant or even safe for communities, but the very purpose of the activities and many of the questions being asked were no longer appropriate. The four cases provide examples of organizations adapting tools to provide more real-time feedback to decision-makers, adapting methods of data collection to remain as rigorous as possible while keeping staff and communities safe, adapting and rethinking the purpose of evaluations and what is really essential to best fulfill the mission of the organization, and adapting the function of MERL to include leveraging big data and machine learning techniques. 3. MERL must apply good principles in a crisis The challenge for Christ-centered organizations during COVID-19 is to discern God’s leading and make consequential decisions in a rapidly changing external environment. These decisions should be informed by monitoring and listening to what is happening both externally and internally with beneficiaries, staff, and many other stakeholders. Yet, the challenge is compounded because most of the standard MERL approaches organizations have used in the past no longer apply. Nevertheless, good MERL principles must continue to underpin the new and adapted approaches. These principles must be applied consistently and proactively, even if the actual approaches may differ. As a priority, Christ-centered Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 11 organizations should be clear on what their principles are, especially in the middle of a crisis. Some examples of key principles highlighted in these case studies are the following: • HOPE International applies an important principle to listen well to clients through MERL as a form of ministry and service to people who bear God’s image, especially in a crisis. • ICM applies the principle to leverage what communication mechanism is already available for the majority of its network to make services and MERL activities more relevant, effective, and timely. • Compassion International’s CARR principles seek to balance what would lead to “Credible” data with what is “Responsible” in order to gather data that supports decision-making and serves children and families loved by God. • Food for the Hungry applies the principle to utilize already available “big data” to provide the best insights for decision-making, because those who are vulnerable deserve the best insights available as a form of service. For further information on good principles, Check, Green, and Kumar (2020) provide a framework of principles to consider for Christ-centered organizations conducting MERL that interacts with the American Evaluation Association Guiding Principles. One example of how organizations could apply the same principles differently would be a decision on whether to hire enumerators or not. One organization could decide to hire and manage enumerators remotely to collect data through phone surveys in order to be more rigorous as an approach, while another organization could decide to train and rely on staff to collect data in order to save money, provide work to staff and/or utilize the opportunity to build a culture of learning. Given the situation and constraints, each approach may be appropriate in the given scenario and if they are also applying good principles. It will be a challenging journey for Christ-centered organizations to increasingly deliver value through MERL, adapt MERL to provide even more useful data, and utilize good principles to guide MERL approaches in ever shifting and volatile environments, like the one COVID-19 presents. It is a journey worth taking, because it will inform responses to COVID-19 and also improve the ability for organizations to discern God’s direction for providing quality services, seeking justice, and empowering communities to face an unknown future. References Catholic Relief Services. 2020. CRS COVID-19 Recommendations: Guidance on Monitoring and Evaluation and Learning. Online resource from CRS. Check, Kristen, Green, Rodney, and Kumar, Subodh. (2020). “Toward Laying the Foundation for Christ-Centered Monitoring & Evaluation.” Christian Relief, Development, and Advocacy: The Journal of the Accord Network 1(2), 61-69. https://crdajournal.org/index.php/crda/article/view /417. Coates, Jennifer; Swindale, Anne; and Bilinsky, Paula. 2007. “Household Food Insecurity Access Scale (HFIAS) for Measurement of Food Access: Indicator Guide.” USAID. http://www.fao.org/fileadmin/user_upload/eufao- fsi4dm/doc-training/hfias.pdf. “Coronavirus – Kenya: Use Digital Payment Methods Instead of Cash as a Precautionary Measure Against the Coronavirus Outbreak.” April 11, 2020. CNBCAfrica. https://www.cnbcafrica.com/africa-press- office/2020/04/11/coronavirus-kenya-use-digital- payment-methods-instead-of-cash-as-a- precautionary-measure-against-the-coronavirus- outbreak/. “COVID-19 Restrictions and Flooding from Above- Average March-May Rains Drive Acute Food Insecurity.” May 2020. Famine Early Warning Systems Network (FEWS NET). https://fews.net/. “Delivering 10 million meals to 1.7 million people in 100 days.” June 28, 2020. Inquirer.net. https://globalnation.inquirer.net/188835/deliverin g-10-million-meals-to-1-7-million-people-in-100- days. Gore, Radhika and Patel, Mahesh. 2006. “Cash Transfers in Emergencies: A Review Drawing Upon the Tsunami and Other Experience.” UNICEF. https://www.unicef.org/policyanalysis/files/Cash_tr ansfers_in_emergencies_- _A_review_drawing_upon_the_tsunami_and_oth er_experience.pdf. “ICM Healthcare Worker Video Training Series.” https://www.caremin.com/COVID-training. Karlan, Dean and Gugerty, Mary Kay. (2018). “The Goldilocks Challenge: Right Fit Evidence for the Social Sector.” Oxford University Press. “Kenya Bans ‘Uncoordinated Direct’ Food Donations after Friday Stampede.” April 12, 2020. CGTN. https://newsaf.cgtn.com/news/2020-04-12/Kenya- bans-uncoordinated-direct-food-donations-after- Friday-stampede-PBYDhtTthS/index.html. https://ics.crs.org/node/898-COVID_MEAL.pdf https://ics.crs.org/node/898-COVID_MEAL.pdf https://ics.crs.org/node/898-COVID_MEAL.pdf https://crdajournal.org/index.php/crda/article/view/417 https://crdajournal.org/index.php/crda/article/view/417 http://www.fao.org/fileadmin/user_upload/eufao-fsi4dm/doc-training/hfias.pdf http://www.fao.org/fileadmin/user_upload/eufao-fsi4dm/doc-training/hfias.pdf https://www.cnbcafrica.com/africa-press-office/2020/04/11/coronavirus-kenya-use-digital-payment-methods-instead-of-cash-as-a-precautionary-measure-against-the-coronavirus-outbreak/ https://www.cnbcafrica.com/africa-press-office/2020/04/11/coronavirus-kenya-use-digital-payment-methods-instead-of-cash-as-a-precautionary-measure-against-the-coronavirus-outbreak/ https://www.cnbcafrica.com/africa-press-office/2020/04/11/coronavirus-kenya-use-digital-payment-methods-instead-of-cash-as-a-precautionary-measure-against-the-coronavirus-outbreak/ https://www.cnbcafrica.com/africa-press-office/2020/04/11/coronavirus-kenya-use-digital-payment-methods-instead-of-cash-as-a-precautionary-measure-against-the-coronavirus-outbreak/ https://www.cnbcafrica.com/africa-press-office/2020/04/11/coronavirus-kenya-use-digital-payment-methods-instead-of-cash-as-a-precautionary-measure-against-the-coronavirus-outbreak/ https://fews.net/ https://globalnation.inquirer.net/188835/delivering-10-million-meals-to-1-7-million-people-in-100-days https://globalnation.inquirer.net/188835/delivering-10-million-meals-to-1-7-million-people-in-100-days https://globalnation.inquirer.net/188835/delivering-10-million-meals-to-1-7-million-people-in-100-days https://www.unicef.org/policyanalysis/files/Cash_transfers_in_emergencies_-_A_review_drawing_upon_the_tsunami_and_other_experience.pdf https://www.unicef.org/policyanalysis/files/Cash_transfers_in_emergencies_-_A_review_drawing_upon_the_tsunami_and_other_experience.pdf https://www.unicef.org/policyanalysis/files/Cash_transfers_in_emergencies_-_A_review_drawing_upon_the_tsunami_and_other_experience.pdf https://www.unicef.org/policyanalysis/files/Cash_transfers_in_emergencies_-_A_review_drawing_upon_the_tsunami_and_other_experience.pdf https://www.caremin.com/covid-training https://newsaf.cgtn.com/news/2020-04-12/Kenya-bans-uncoordinated-direct-food-donations-after-Friday-stampede-PBYDhtTthS/index.html https://newsaf.cgtn.com/news/2020-04-12/Kenya-bans-uncoordinated-direct-food-donations-after-Friday-stampede-PBYDhtTthS/index.html https://newsaf.cgtn.com/news/2020-04-12/Kenya-bans-uncoordinated-direct-food-donations-after-Friday-stampede-PBYDhtTthS/index.html Christian Relief, Development, and Advocacy 2(1), Summer 2020 Green, Mallonee, and Check, MERL in the Time of COVID-19: Four Case Studies 12 Lau, Lincoln Leehang; hung, Natalee; Go, Daryn Joy; Ferma, Jansel; Choi, Mia; Dodd, Warren, and Wei, Xiaolin. 2020. “Knowledge, attitudes and practices of COVID-19 among income-poor households in the Philippines: A cross-sectional study.” Journal of Global Health 10 (1), 1-11. Manila Bulletin. July 21, 2020. “Protecting Filipino Healthcare Workers from COVID-19.” https://mb.com.ph/2020/07/21/protecting-filipino- healthcare-workers-from-COVID-19/ Mani, Subha and Barooah, Bidisha. April 9, 2020. “Phone surveys in developing countries need an abundance of caution.” International Initiative for Impact Evaluation. https://www.3ieimpact.org/blogs/phone-surveys- developing-countries-need-abundance-caution. Mellado, Santiago “Jimmy.” 2019. “Why Good Discernment is like a Sailboat.” (https://www.facebook.com/watch/?v=265608856 1122124). Patton, Michael. March 23, 2020. “Evaluation Implications of the Coronavirus Global Health Pandemic Emergency.” Blog post. Blue Marble Evaluation. https://bluemarbleeval.org/latest/evaluation- implications-coronavirus-global-health-pandemic- emergency. Population Council. 2020. “Kenya: COVID-19 Knowledge, Attitudes, Practices & Needs.” Kenya Ministry of Health. https://www.popcouncil.org/uploads/pdfs/2020PG Y_CovidKenyaKAPStudyResultsBriefRound2.pd f “Utilization-Focused Evaluation.” N.d. Better Evaluation. https://www.betterevaluation.org/en/plan/approac h/utilization_focused_evaluation “Why COVID-19 Is Another Blow for Kenya’s Food Security.” April 20, 2020. The Conversation. https://theconversation.com/why-COVID-19-is- another-blow-for-kenyas-food-security-135567 World Economic Forum. 2012. “Big Data, Big Impact: New Possibilities for International Development.” http://www3.weforum.org/docs/WEF_TC_MFS_ BigDataBigImpact_Briefing_2012.pdf. Additional Resources Accord Network COVID-19 Resources Accord Research Alliance COVID-19 MERL Resources Authors/Editors Rodney Green is a Monitoring, Evaluation and Listening Specialist with Compassion International with ten years of experience in International Development. He holds a MSc in Development Management and co-chairs the Accord Research Alliance steering committee. Author email: RGreen@us.ci.org Nathan Mallonee is the Senior Director of Program Development at Living Water International. He holds a master’s degree in international policy and development from Georgetown University and co-chairs the Accord Research Alliance steering committee. Author email: nmallonee@mac.com Kristen Check is a Senior Program Effectiveness Research Specialist with Compassion International and owner of Edessa Research, LLC. She holds an MA in Anthropology, has over ten years of experience in International Development, and co-chairs the Accord Research Alliance steering committee. Author email: Kcheck@us.ci.org Contributors for Case Studies Rebecca Mentzer serves at HOPE International as Senior Listening, Monitoring, and Evaluation Advisor. She is a member of the ARA steering committee. Lincoln Lau, based in Manila, is an infectious disease epidemiologist, Director of Research at International Care Ministries, and Assistant Professor of Public Health at University of Toronto. Kendall Wilson is an ICM Research Associate with a background in Epidemiology, a passion for M&E and healthcare, and an eagerness to launch her career in medical research. Allan Kakinda is a Global Monitoring and Evaluation Specialist at Compassion International, based in Uganda. Peter Ndungu is a National Monitoring and Evaluation Specialist for Compassion International in Kenya. Dr. Subodh Kumar is Global Director of Monitoring and Evaluation with Food for the Hungry and has been working in M&E for the last 25 years. He is a member of the ARA steering committee. https://mb.com.ph/2020/07/21/protecting-filipino-healthcare-workers-from-covid-19/ https://mb.com.ph/2020/07/21/protecting-filipino-healthcare-workers-from-covid-19/ https://www.3ieimpact.org/blogs/phone-surveys-developing-countries-need-abundance-caution https://www.3ieimpact.org/blogs/phone-surveys-developing-countries-need-abundance-caution https://www.facebook.com/watch/?v=2656088561122124 https://www.facebook.com/watch/?v=2656088561122124 https://bluemarbleeval.org/latest/evaluation-implications-coronavirus-global-health-pandemic-emergency https://bluemarbleeval.org/latest/evaluation-implications-coronavirus-global-health-pandemic-emergency https://bluemarbleeval.org/latest/evaluation-implications-coronavirus-global-health-pandemic-emergency https://www.popcouncil.org/uploads/pdfs/2020PGY_CovidKenyaKAPStudyResultsBriefRound2.pdf https://www.popcouncil.org/uploads/pdfs/2020PGY_CovidKenyaKAPStudyResultsBriefRound2.pdf https://www.popcouncil.org/uploads/pdfs/2020PGY_CovidKenyaKAPStudyResultsBriefRound2.pdf https://www.betterevaluation.org/en/plan/approach/utilization_focused_evaluation https://www.betterevaluation.org/en/plan/approach/utilization_focused_evaluation https://theconversation.com/why-covid-19-is-another-blow-for-kenyas-food-security-135567 https://theconversation.com/why-covid-19-is-another-blow-for-kenyas-food-security-135567 http://www3.weforum.org/docs/WEF_TC_MFS_BigDataBigImpact_Briefing_2012.pdf http://www3.weforum.org/docs/WEF_TC_MFS_BigDataBigImpact_Briefing_2012.pdf https://accordnetwork.org/resources/covid-19-resources/ https://docs.google.com/spreadsheets/d/1T6RBQ-1yy-34XglanpjXQoZLGVZ3UJeXw276OD7TczI/edit?ts=5ea9cd00#gid=0 https://docs.google.com/spreadsheets/d/1T6RBQ-1yy-34XglanpjXQoZLGVZ3UJeXw276OD7TczI/edit?ts=5ea9cd00#gid=0 mailto:RGreen@us.ci.org mailto:nmallonee@mac.com mailto:Kcheck@us.ci.org