Research, interrupted: Addressing practical and methodological challenges under turbulent conditions Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 30/10/2023 Facebook: /EPAAA Revisions received: 23/2/2024 Twitter: @epaa_aape Accepted: 24/2/2024 education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 32 Number 21 April 9, 2024 ISSN 1068-2341 Research, Interrupted: Addressing Practical and Methodological Considerations under Turbulent Conditions1 Susan Bush-Mecenas RAND Corporation Jonathan D. Schweig RAND Corporation Megan Kuhfeld NWEA Louis T. Mariano RAND Corporation & Melissa K. Diliberti RAND Corporation United States Citation: Bush-Mecenas, S., Schweig, J. D, Kuhfeld, M., Mariano, L. T., & Diliberti, M. (2024). Research, interrupted: Addressing practical and methodological challenges under turbulent conditions. Education Policy Analysis Archives, 32(21). https://doi.org/10.14507/epaa.32.8436 1 The research reported here was sponsored by the Institute of Education Sciences, U.S. Department of Education, through grant R305U200006. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. http://epaa.asu.edu/ojs/ https://doi.org/10.14507/epaa.32.8436 Education Policy Analysis Archives Vol. 32 No. 21 2 Abstract: The COVID-19 pandemic caused tremendous upheaval in schooling. In addition to devasting effects on students, these disruptions had consequences for researchers conducting studies on education programs and policies. Given the likelihood of future large-scale disruptions, it is important for researchers to plan resilient studies and think critically about adaptations when such turbulence arises. In this article, we utilize qualitative analysis of interviews with research study leaders to illuminate practical and methodological challenges, as well as promising practices that arose during the pandemic period. We find that researchers made pivots to address practical challenges a nd protect the feasibility of their studies. We also find that researchers took precautions, where possible, to understand and bolster internal validity. However, these pivots frequently surfaced additional threats to construct and external validity. We co nclude with recommendations for future studies conducted in times of prolonged unplanned school closures or other large-scale disruptions. Keywords: education effectiveness; school disruptions; study design Investigación interrumpida: Abordar desafíos prácticos y metodológicos en condiciones turbulentas Resumen: La pandemia de COVID-19 provocó una enorme agitación en la educación. Además de los efectos devastadores para los estudiantes, estas perturbaciones tuvieron consecuencias para los investigadores que realizaban estudios sobre programas y políticas educativas. Dada la probabilidad de futuras perturbaciones a gran escala, es importante que los investigadores planifiquen estudios de resiliencia y piensen críticamente sobre las adaptaciones cuando surjan tales turbulencias. En este artículo, utilizamos un anális is cualitativo de entrevistas con líderes de estudios de investigación para iluminar los desafíos prácticos y metodológicos, así como las prácticas prometedoras que surgieron durante el período de la pandemia. Descubrimos que los investigadores tomaron med idas para abordar desafíos prácticos y proteger la viabilidad de sus estudios. También encontramos que los investigadores tomaron precauciones, cuando fue posible, para comprender y reforzar la validez interna. Sin embargo, estos pivotes a menudo sacaron a la superficie amenazas adicionales a la validez externa y de constructo . Concluimos con recomendaciones para futuros estudios realizados en tiempos de cierres escolares prolongados e imprevistos u otras perturbaciones a gran escala. Palabras-clave: eficacia de la educación; interrupciones escolares; diseño del estudio Pesquisa interrompida: Enfrentando desafios práticos e metodológicos em condições turbulentas Resumo: A pandemia de COVID-19 causou uma tremenda reviravolta na escolaridade. Além dos efeitos devastadores sobre os estudantes, estas perturbações tiveram consequências para os investigadores que conduziam estudos sobre programas e políticas educativas. Dada a probabilidade de futuras perturbações em grande escala, é importante que os investigadores planeiem estudos resilientes e pensem criticamente sobre as adaptações quando tal turbulência surgir. Neste artigo, utilizamos a análise qualitativa de entrevistas com líderes de pesquisas para iluminar desafios práticos e metodológicos, bem como práticas promissoras que surgiram durante o período pandêmico. Descobrimos que os investigadores fizeram pivôs para enfrentar desafios práticos e proteger a viabilidade dos seus estudos. Descobrimos também que os investigadores tomaram precauções, sempre que possível, para compreender e reforçar a validade interna. No entanto, estes pivôs frequentemente trouxeram à tona ameaças adicionais à validade externa e de Research, interrupted 3 construção. Concluímos com recomendações para estudos futuros realizados em tempos de encerramentos prolongados e não planeados de escolas ou outras perturbações em grande escala. Palavras-chave: eficácia da educação; interrupções escolares; design de estudo Research, Interrupted: Addressing Practical and Methodological Considerations under Turbulent Conditions Policymakers in the United States (US) have increasingly called for education intervention and policy adoption to be guided by evidence-based research (e.g., Slavin, 2002). In 2001, the No Child Left Behind Act (U.S. Department of Education [U.S. ED], 2002) mandated that scientific evidence serve as the basis for federally funded interventions. Since that time, the Institute for Education Sciences (IES) of the U.S. ED has invested in studies that use experimental and quasi- experimental designs to provide rigorous evidence of what works (e.g., Slavin, 2008). To support this agenda, U.S. ED created the What Works Clearinghouse (WWC) and the Standards for Excellence in Education Research (SEER) principles (Institute of Education Sciences [IES], 2022a). Collectively, WWC and SEER codify expectations that education studies should mitigate threats to validity, thereby increasing the credibility of such research. This helps ensure researchers and policymakers draw appropriate conclusions of which interventions and policies are likely to be effective in a specific school or district context (e.g., Joyce & Cartwright, 2020). Large-scale2 disruptions to schooling—including those during the COVID-19 pandemic and at other time points—have important implications for the researchers who planned and continued to conduct research on education interventions and policies under these turbulent conditions. Atypical schooling conditions potentially jeopardize the integrity of findings from individual studies conducted under these conditions and impede efforts to advance our collective understanding of “what works” in education, thus compromising decision-making regarding education policy. Although the COVID-19 pandemic-related school closures were unprecedented in scale, large-scale disruptions to schooling were already common by 2020. Between 2011 and 2019, over 13 million students in the US were impacted by prolonged unplanned school closures primarily due to extreme weather phenomena (including hurricanes), natural disasters (including wildfires), teacher strikes, environmental issues, and violence (Jahan et al., 2022). Furthermore, there is an emerging consensus that further disruptions to schooling are inevitable in the future (e.g., Inter-agency Network for Education in Emergencies [INEE] 2020; Salman, 2022). Given the likelihood of future large-scale disruptions, it is important for researchers seeking to evaluate education interventions and policies to plan resilient studies and think critically about possible adaptations when such turbulence arises. A systematic assessment of common challenges and successes can help to ensure that future studies meet standards for high-quality research (IES, 2022a). In this article, we examine the COVID-19 pandemic period as one particularly salient case of a large-scale disruption to schooling that exemplifies challenges faced by researchers evaluating education interventions and policies under turbulent conditions. We utilize qualitative analysis of interviews with research study leaders to illuminate some promising practices that arose during the pandemic period and provide recommendations for future education research conducted during times of profound disruption such that educational policy decisions may be rooted in sound research. 2 We define large-scale disruptions as those that are extensive in terms of duration and far-reaching in terms of the number of students affected. Education Policy Analysis Archives Vol. 32 No. 21 4 Background Literature: Practical Challenges Posed by Large-Scale School Disruptions To understand the challenges involved in conducting research amid turbulence conditions, we draw on a growing body of research on the topic, including a burgeoning literature in international development about conducting research amid humanitarian crises (Bakrania et al., 2021; Hassnain et al., 2021; Nene Odjidja & Alves do Reis, 2021; Puri et al., 2017). This international development literature is particularly useful to understand how multiple forms of disruption affect research, as few examples exist in the United States context. We use this literature to guide our analysis of the pivots and adaptations made by education researchers who conducted studies during the COVID-19 pandemic. This literature has identified several practical challenges that are commonly encountered by researchers in humanitarian settings. The same literature has also identified methodological issues that threaten the “validity of an impact evaluation” (Puri et al., 2017, p. 523) conducted under these conditions. Building on the international development literature, we defined four specific practical challenges that are distinguishing features of conducting education research in the midst of large- scale school disruptions like the COVID-19 pandemic including: complex settings marked by turbulence and instability, difficulty in accessing research populations and collecting data, lack of local capacity to support research, and limited availability of good-quality data (see Bakrania et al., 2021; Hassnain et al., 2021; Nene Odjidja & Alves do Reis, 2021; Puri et al., 2017). Complex Settings Marked by Instability Schools and districts are complex, dynamic systems (Groff, 2013) marked by turbulence and instability even in normal school years (Myers, 2014). Turbulence regularly arises for a variety of reasons, including changes to district capacity and priorities, organizational culture, curriculum, standards, and assessment. However, large-scale disruptions like the COVID-19 pandemic introduce additional complexities, disrupting even the most durable aspects of schooling—things like instructional mode, timing and length of the school year, student enrollments, and grade progressions. For example, schools in all 50 states closed to in-person instruction in spring 2020. They reopened to in-person instruction on different timelines and had different policies on instructional modes. In sum, the COVID-19 pandemic caused widespread breakdowns in both logistics and school infrastructure (Puri et al., 2017). Difficulty in Accessing Research Populations and Collecting Data Education researchers often rely on direct observation, interviews, and focus groups with students and school staff—often conducted in-person at schools—in their studies. School closures presented serious difficulties for researchers looking to access research populations, as researchers were prohibited from visiting schools or interacting in person with students or school staff throughout the pandemic. Even as schools reopened for in-person instruction in the 2020–2021 school year, many state departments of public health recommended that schools continue to limit nonessential visitors (e.g., California Department of Public Health, 2022). While researchers may have increasingly pivoted to conduct their data collections virtually, this shift may have excluded particularly vulnerable populations from participating in research because of systematic inequities in home internet access (e.g., Stelitano et al., 2020). Lack of Local Capacity to Support Research Even under ideal conditions, conducting school-based research involves the input and support of a variety of system stakeholders, including (but not limited to) district administrators, Research, interrupted 5 principals, teachers, and parents. At minimum, these stakeholders need to grant researchers permission to access research populations. But often these stakeholders support research in much more involved ways, including participating in implementation trainings, facilitating the distribution of consent forms, establishing data use agreements, fulfilling administrative data requests, and reviewing study procedures and protocols to ensure that they meet standards for ethical research (e.g., Bartlett et al., 2017). During the pandemic, schools were faced with serious staffing shortages (e.g., Schimtt et al., 2022), and many teachers felt increased job burnout and stress, in part because they believed the pandemic forced them to work more hours (Diliberti et al., 2021). These conditions have serious implications for the ability of schools to support research for two reasons. First, there are simply diminished staff resources to focus on research. Second, even among remaining staff, research is likely to be deprioritized to make room for other more immediate challenges (such as assuring compliance with local health mandates or ensuring that school lunches are distributed). Additionally, staff turnover has potential implications for human capital and institutional knowledge; those individuals who were familiar with the particulars of study planning or implementation may no longer be working in a particular school or district. Limited Availability of Good-Quality Administrative Data In addition to the primary source data described above, education research studies regularly make use of a variety of administrative data and standard measures routinely collected by schools and districts, including assessments, course grades, attendance, student surveys, among many others. The pandemic limited researchers and educators’ ability to collect high-quality data on many, if not all, of the measures central to education research. Beyond issues with the quality of assessment data (see below), a nationally representative survey conducted by the Center for Reinventing Public Education (CRPE) found that only half of all districts monitored engagement through attendance tracking or one-on-one check-ins, and only around 40% of districts set expectations that teachers collect student work, grade it, and include it in final course grades (Gross & Opalka, 2020). Conceptual Framework: Methodological Challenges Posed by Large-Scale School Disruptions Building upon this international development literature, we analyze our findings using the lens of Campbell and colleagues’ (1963; 1972) widely-referenced validity framework (Campbell & Stanley, 1963; Cook & Campbell, 1979; Shadish et al., 2002). Specifically, we frame the practical and associated methodological challenges encountered by researchers during an extreme period of school disruptions in terms of specific threats to internal validity, construct validity, statistical conclusion validity, and external validity. The framework developed by Campbell and colleagues (Campbell & Stanley, 1963; Cook & Campbell, 1979; Shadish et al., 2002) indexes 37 distinct threats to internal validity, construct validity, statistical conclusion validity, and external validity. While these 37 threats provided the foundation for our qualitative data analysis, we focus on a subset of 10 validity threats that emerged inductively in our qualitative analysis and that are particularly salient to large-scale school disruptions such as the COVID-19 pandemic (see Table 1). Below, we briefly describe the 10 validity threats and explain how COVID-19 era schooling conditions might have heightened these threats in the studies conducted under these conditions. Education Policy Analysis Archives Vol. 32 No. 21 6 Table 1 Categories of Validity Threats Type Definition Specific Threats Internal Validity The extent to which the estimated effects of a program or intervention correspond to true causal effects Selection bias Attrition bias Instrumentation bias History bias Construct Validity The extent to which the enacted treatments and measurement constructs faithfully represent the intended treatments and measurement constructs (Gorin 2006) Inadequate explication of constructs Treatment diffusion Statistical Conclusion Validity The extent to which statistical methods are used appropriately Low statistical power Unreliability of treatment implementation External Validity The extent to which inferences about effectiveness can be generalized to other individuals, settings, instruments, or program variations, (Briggs 2008) Interaction of the causal relationship with settings Interaction of the causal relationship over treatment variations Source: Shadish et al. (2002) Threats to Internal Validity Internal validity describes the extent to which the estimated effects of a program or intervention correspond to true causal effects (Shadish et al., 2002). There are four primary threats to internal validity that are particularly relevant for our analysis. Selection bias is likely to arise if there are systematic differences between those who participate in an intervention and those who do not. This will occur, for example, if schools or districts that have more resources, more stable or resilient infrastructure, lower exposure (e.g., lower COVID transmission rates) or greater organizational capacity consent to participate in a study and implement an intervention. Attrition bias is likely to arise because of shifts in enrollment and staffing. Large numbers of students transferred out of public schools during and after pandemic-induced school closures (Dee & Murphy, 2021). Beyond changes in enrollments, student and staff absenteeism increased and schools reported trouble adequately staffing their positions (IES, 2022b). Dropout and absenteeism can lead to attrition bias if students or staff who remain in schools may be systematically different from those who withdrew. Instrumentation bias is likely associated with disruptions to state assessment systems. There were drastic changes to states’ summative assessment programs in 2019–2020 and 2020–2021. Spring 2020 summative assessments were suspended in all states, and Spring 2021 assessments were administered in a variety of modes (e.g., remotely, in-person). Some states administered abbreviated summative assessments while others used benchmark assessments like MAP Growth (Bruno & Goldhaber, 2021). These changes in instrumentation may be conflated with an exposure effect (internal validity). History bias is a particularly salient threat to internal validity if an intervention or Research, interrupted 7 policy was enacted over a period that aligned with the pandemic. In such a situation, it is difficult— if not impossible—to disentangle the effects of the intervention on student outcomes from the confounding effects of the pandemic itself. Threats to Construct Validity Construct validity describes the extent to which the enacted treatments and measurement constructs faithfully represent the intended treatments and measurement constructs (Gorin, 2006; Shadish et al., 2002). There are two primary threats to construct validity that are particularly relevant for our analysis. Inadequate explication of constructs may arise because of the disruptions to assessment programs described above. When assessments are altered or adapted, there is the possibility that these revised assessments do not capture the same aspects of the construct as the intended measures. Treatment diffusion may be induced by the drastic and sudden changes in school contexts and procedures that occur during unplanned and prolonged disruptions to schooling. As one specific example, the federal government allocated billions of dollars to state education agencies through the Elementary and Secondary School Emergency Relief (ESSER) Fund with the goal of supporting school restart and recovery. This money was allocated to programs (including high dosage tutoring programs, see Robinson et al., 2021) that, in some cases may alter or overlap with the intervention activities that comprise the treatment or control conditions. Threats to Internal Validity Statistical conclusion validity describes the extent to which statistical methods are used appropriately (Shadish et al., 2002). There are two primary threats to statistical conclusion validity that are particularly relevant for our analysis. Low statistical power may be induced by the shifts in enrollment and staffing described above. Heighted attrition can reduce statistical power. Attrition bias may arise if there is differential attrition, even if there are not systematic differences in study withdrawal or data availability across study conditions (e.g., if those that are participating in an intervention are more likely to have missing assessment scores or to withdraw from school entirely). Widespread missing data or heightened attrition reduces the statistical power of a study to detect a relationship between an intervention or policy and an outcome. Unreliability of treatment implementation may arise because of how schools and systems altered interventions in response to the pandemic. For example, interventions designed for conventional classroom settings were frequently reconfigured to be implemented online. Furthermore, because schools reopened for in-person instruction on different timelines, implementation may have varied from setting to setting. Threats to External Validity External validity describes the extent to which inferences about effectiveness can be generalized to other individuals, settings, instruments, or program variations (Briggs, 2008; Shadish et al., 2002). In the international development literature, Nene Odjidja & Alves do Reis (2021) note that because humanitarian settings are so dynamic and idiosyncratic, the generalizability of findings from research conducted in such settings may be limited and that findings may not replicate in different geographic areas or among different populations. There are two primary threats to external validity that are particularly relevant for our analysis. Interaction of the causal relationship with settings may occur because effects found in one site may not hold if the study were conducted in another site or at another point in time (including timepoints more temporally distant from the height of the pandemic). As mentioned above, because schools reopened for in-person instruction on different timelines, implementation of interventions may have varied from setting to setting given differences in instructional modes. Communities and schools were also differentially vulnerable to pandemic effects based on a combination of ecological, Education Policy Analysis Archives Vol. 32 No. 21 8 social, health and economic factors (Nayak et al. 2020). This raises questions about the extent to which effects identified by studies conducted in this era could be expected to hold in other settings or with other program instantiations. Interaction of the causal relationship over treatment variations may occur because effects found with one specific variation of an intervention or policy might not hold with other variations of that intervention or policy. Many schools altered interventions in response to COVID-19, inducing some of the idiosyncrasy in program implementation noted by Nene Odjidja & Alves do Reis (2021). The Current Study Throughout the pandemic, researchers had to think creatively about how to approach the design and implementation of education studies, often having to make adaptations and design pivots in response to practical and methodological challenges. Because these pivots were made reactively in the context of a worldwide crisis, there was little opportunity to think collaboratively about the best ways to ensure study feasibility and mitigate the threats to validity. This study describes common challenges and successes with the objective of improving decision making by education researchers during future periods of disruption. Drawing on interviews with researchers who were fielding or planning IES-funded Exploration, Development and Innovation, Initial Effectiveness and Efficacy and Replication3 studies during the pandemic, we address three research questions: 1) How did the COVID-19 pandemic present practical and methodological challenges for education researchers fielding or planning IES-funded Exploration, Development and Innovation, Initial Effectiveness and Efficacy and Replication studies? 2) How and why did such researchers adjust their studies in response to the challenges presented by these turbulent conditions? 3) What consequences and implications arose from these methodological decisions, in terms of threats to validity? We find that that researchers made pivots to address practical challenges and protect the feasibility of their studies. We also find that researchers took precautions, where possible, to understand and bolster internal validity. However, while the adaptations researchers made to their studies in response to practical and methodological challenges mitigated some threats to validity, the adaptations they chose frequently surfaced additional validity threats—particularly to construct validity and external validity. The remainder of this article is structured as follows: First, we describe our data and methods. Then, we summarize our findings, including the practical challenges faced by researchers such as issues with intervention feasibility, difficulty with recruitment, challenges with data availability and data quality, and concerns with extrapolation and generalization. In the next section, we discuss these results, connecting our findings back to the practical and methodological challenges 3 Though IES has altered the goal structure several times since 2017, we use the goals and project types that appeared in the project summaries on the IES website throughout this analysis. Exploration grants, as defined by IES, identify malleable factors associated with student outcomes. Development and Innovation grants seek to develop interventions and improve existing interventions. Initial Efficacy studies include those that 1) examine interventions that have not been rigorously evaluated previously, 2) test the longer-term impact of interventions shown to have beneficial impacts on student outcomes, and 3) rely on retrospective data to test the impacts of past interventions. Efficacy and Replication studies examine the extent to which interventions found to have beneficial education outcomes on student outcomes replicate in other contexts and under different study conditions. A fifth goal, Measurement, is not included in the current study. Research, interrupted 9 described in the international development literature. We conclude with recommendations for future studies conducted in times of prolonged unplanned school closures or other large-scale disruptions. Data and Methods In this qualitative study, we describe patterns from interview data to understand how researchers responded to turbulent environments and why they made the decisions they did. Qualitative methods are, on the whole, well-suited to address these particular questions, as “[t]he interest [in qualitative research] is in process rather than outcomes” (Merriam, 1988, p. xii). That is, qualitative methods are particularly well-suited to answer questions of how or why (Yin, 2009). Uniquely, this study utilized qualitative approaches to examine the decision-making, moves, and rationale of quantitative researchers, and analyzes how these behaviors may have affected validity. Sampling Strategy We purposively elected to focus on research studies funded through IES’s National Center for Education Research (NCER), both because of the breadth of research sponsored by NCER and because the details of funded studies are publicly available. We intentionally included studies on a wide range of topics, grade levels, and subject areas (e.g., math, reading, history, science). We used a three-step process to purposively random sample interview participants. First, using publicly available information on IES-funded Exploration, Development and Innovation, Initial Efficacy, and Efficacy and Replication studies in fiscal years 2017 through 2021, we created a sampling frame from which to sample studies. Relevant studies for potential inclusion were identified through searches of the IES website, which contains project summaries for all funded projects. Available information includes principal investigator, grant type, award period, research design, and key measures. We excluded studies that were scheduled to conclude (meaning outcome data had already been collected) before the initial pandemic school closures in March 2020 as well as studies that did not rely on school or system administrative data (including state assessment data) for at least one study outcome. In total, we identified 85 studies that met our initial inclusion criteria. Second, informed by the common study features outlined in Hedges and Tipton (2020), we conceptualized studies as generally including five research stages, include recruitment, random assignment, intervention or policy implementation, implementation data collection, and outcome data collection. We developed five categories that were characterized by the interaction between the research stage and the time point at which the pandemic began to influence them (Table 2). These categorizations primarily reflect different starting points in time (rather than presenting distinct individual or study characteristics). Category 1 includes single cohort studies that were influenced during outcome data collection phase. Category 2 includes studies that had been planned (and perhaps study sites had been recruited) but had not yet gotten to the intervention implementation stage until after Spring 2020 school closures. Category 3 includes multi-cohort studies and longitudinal studies where outcome data collection was intended to happen both pre-COVID and after COVID-19 school closures. Category 4 includes studies that had recruitment, randomization, implementation, and outcome data collection happen after initial school closures, but had some part of their implementation stage happen when students in participating schools were learning remotely. Finally, Category 5 includes studies that were conducted after the most acute impacts of the pandemic subsided and schools were largely re-opened for in-person learning. Education Policy Analysis Archives Vol. 32 No. 21 10 Table 2 Categories of Cases for Sampling Case Category Description Category 1 Studies that began prior to Spring 2020 school closures and were designed to include only one outcome data collection opportunity Category 2 Studies that were planned prior to Spring 2020 school closures but implemented in the Fall after Spring 2020 school closures Category 3 Studies that began prior to Spring 2020 school closures and were designed to include both pre-pandemic and post-school closure outcomes Category 4 Studies that were planned and implemented after Spring 2020 school closures Category 5 Studies that began after schools reopened in Spring 2021 Third, we mapped each of the included studies from our sampling frame to one of these five categories. We reached out to leads of randomly selected studies within each category, until we reached three studies in each category. In total, we reached out to 18 study leaders and 15 agreed to participate. Our sample size falls within Hennink and Kaiser’s (2022) findings regarding adequate sample size in qualitative research, which suggests that 9–17 interviews are sufficient to reach saturation within relatively homogenous populations and narrowly defined objectives. Table 3 highlights some descriptive information about the research participants. The majority of the studies included were Efficacy studies, and all but the two Exploration studies included a planned experimental or quasi-experimental design. About half of the included studies examined the efficacy of interventions designed to improve student outcomes, and the other half were designed to improve teaching through coaching and professional development. The included studies also involved a wide range of student populations, including students in rural suburban and urban areas, and students from across the K–12 grade span. States from all geographic regions (Northwest, Southwest, Northeast, Midwest, Southeast) were represented, and two studies were multi-state studies. Many studies did not report demographic information on their participants, but those that did often reported samples as racially, ethnically, and economically diverse. Table 3 Descriptions of Included Studies Type Category and Count Goal Exploration (n=2) Development and Innovation (n=4) Initial Efficacy (n=4) Efficacy and Replication (n=5) Research, interrupted 11 Type Category and Count Student or Teacher Focus Student outcomes (n=7) Teacher development (n=8) Grade Level Elementary grades (K–5) (n=8) Middle grades (6–8) (n=4) High school grades (9–12) (n=3) Urbanicity Urban (n=7) Rural (n=3) Multiple settings (n=5) Race/Ethnicity Three studies mention a focus on a racially diverse sample Poverty Four studies mention serving students experiencing poverty Outcomes Math and reading (n=8) Reading (n=3) Math (n=3) History (n=1) Data Collection and Analysis In Spring and Summer 2022, we conducted in-depth, semi-structured interviews with principal investigators (n=15, with three studies in each of the five categories, as noted above) and transcribed audio recordings prior to coding and analysis. Our interview protocol asked respondents about their study design and context, changes to school conditions, recruitment and sampling strategies, research design pivots, use of outcomes data, comparison of pre- and post-COVID-era outcomes, generalizability, and general successes and challenges in conducting research in turbulent settings. We began by coding interview data across different levels of abstraction, using Dedoose qualitative research software. For example, we coded for elements of study context (study intent, school level, study design, timeline), changes in instructional context (changes in conditions, modification of metrics and data, instructional model, staff turnover, fidelity of implementation, district priorities, use of funds), recruitment and sampling (sampling plan, recruitment efforts, issues, attrition), design pivots (research questions, hypotheses, counterfactual conceptualization, design and analysis plan), outcomes data (planned data sources, administration issues, data modifications), pre/post pandemic comparison (plans to compare or combine data, issues, modifications), and extrapolation and generalization (to pre-pandemic, to future). We then analyzed coded data using cross-case meta-matrices (Bush-Mecenas & Marsh, 2018; Miles et al., 2020) to understand prevalence and patterns among these quotes, paired with analysis along the categories of validity (see Table 1, Shadish et al., 2002). To enhance the internal validity and accuracy of findings, we compared interview data with existing grant documentation. Specifically, we integrated grant proposal documentation to document study design and compare to described planned design and activities. We present our findings by the three common challenges that emerged from our data, describing the challenges and how researchers responded. We then Education Policy Analysis Archives Vol. 32 No. 21 12 present cross-cutting findings on the topic of extrapolation and generalizability, across these practical and methodological challenges and pivots. Limitations There were several limitations to our data collection and analyses. We focused our study on IES-funded research that relied on state or district administrative data for key measures of study outcomes. Because of this, most studies included in this analysis are Initial Efficacy or Efficacy and Replication studies. These studies are not meant to be representative of all IES studies or studies funded by other agencies. Relatedly, while we randomly selected studies for interviews, there was some nonresponse, and we cannot be sure that those we did interview are representative of all studies funded by IES that were under the Exploration, Develop and Innovation, Initial Efficacy, and Efficacy and Replication agency goals. Third, we were unable to collect additional information on study implementation from school or district partners. While such data would have provided a richer understanding of how research programs were altered during the pandemic, the researchers we interviewed were reluctant to grant permission for such interviews, given concerns about capacity, burden, and sustaining research partnerships. Finally, because many of these research programs are actively funded by IES, confidentiality and anonymity were assured to all research participants. This included mitigating the possibility that the included studies could be identified by inference. As such, we are limited in the amount of study information and context we can provide. Although this study is best understood as exploratory, it is an important first attempt to describe common challenges and successes with the objective of improving decision making during future periods of disruption. Findings First, we find that researchers experienced three common and compelling practical challenges with methodological implications: (1) issues with intervention feasibility caused by situational complexity, (2) difficulty with study recruitment, and (3) issues with data availability and concerns about data quality. Although our interview design and sampling approach was guided by the assumption that researchers’ responses to the COVID-19 pandemic would vary based on research stage and the time point at which the pandemic began to influence them, our analyses instead demonstrated that these three challenges were common across most, regardless of where they were in study planning and implementation. Additionally, while the existing literature in international development describes practical and methodological challenges separately, the researchers we interviewed for this study frequently described them as inherently interconnected. Specifically, the researchers we interviewed framed the issues they faced as practical challenges with methodological implications. Next, we explore each of these challenges in detail. For each, we describe the challenge and identifying common patterns that emerged regarding researchers’ pivots and adaptations. We then describe the associated methodological challenges that arose, framing those findings in terms of potential threats posed to validity. Finally, we conclude this section with a cross-cutting findings on researchers’ reflections on extrapolation and generalization, and the extent to which they believed that research conducted in the context of the pandemic was likely to be comparable to previously conducted studies, or to studies conducted several years from now, when the immediate impacts of the pandemic have subsided. Research, interrupted 13 Challenge 1. Limited Intervention Feasibility Due to Situational Complexity The first major challenge identified by our respondents was limited intervention feasibility, largely caused by instability in the institutional environment. About half of our respondents indicated that instability in the institutional environment (including instructional resources, staffing, working conditions, technology, structure of the school day, and the school building) during the pandemic decreased the feasibility of their intended interventions. Most obviously, school closures as well as virtual and hybrid instruction interrupted or modified interventions by altering the structure of the school day and the physical spaces in which teaching and learning occur (especially for studies using traditional models of school-based classroom interaction as a variable of interest). One researcher referred to these aspects of the institutional environment as the underlying organizational infrastructure of schools, and noted, The infrastructure in schools has been stressed…I think we’ve learned a lot more about … how fragile that infrastructure is and how unchangeable and inflexible it is. And that’s true of high schools, I think elementary and middles can flex easier than high schools. Their structure is years old and rarely has looked any different. is an example of a first paragraph of an extended quotation (even if this is not extended. Most respondents shared similar observations about the full, system -level disruptions brought on by the pandemic: teacher absenteeism and retirements, lack of substitute teachers, shifting daily work for administrators to cover classrooms, increased student absenteeism, and students’ heightened need for support all were identified as factors affecting the feasibility of ongoing research. One researcher noted the pandemic led to widespread “instability” due to “constant shifts” in staffing. This researcher continued, For me the bottom line is we need to learn how to do applied research within these kinds of settings. As a researcher, I really wish it wasn’t like this. It’s really hard to control for all the variables that come into play, or even know how to... I think the big question for me when we meet as a team is, “What else should we be measuring that we’re not measuring?” because we’re not collecting data informally or formally to be able to understand and explain the context. Researchers Attempted to Mitigate Feasibility Challenges through Flexibility in Research Design In response to feasibility challenges, researchers took a variety of approaches, often depending upon the intervention itself and the phase of research. First, almost all respondents reported that the researchers need to modify study timelines. Over half of the studies in our sample were delayed, with two studies ending prematurely due to inability to implement or measure the intervention under virtual instruction. Other researchers, especially those with studies scheduled to begin later in the pandemic, opted for a lengthened development period and delayed implementation. For three respondents, pilot studies and implementation analyses were conducted while awaiting suitable conditions for experimental research. Second, where possible, researchers simplified or modified interventions “to be less burdensome.” Modifications addressed limitations on teachers’ time (e.g., shorter professional learning segments), increased student need (e.g., a greater proportion of students needing tiered academic or behavioral interventions), and schools’ use of virtual instruction (e.g., modifying instruction for use in virtual settings). In one example, a researcher described moving their Education Policy Analysis Archives Vol. 32 No. 21 14 intervention to an online platform and how this benefitted the implementation of the intervention, research data collection, and future research. As this researcher explained, Making the materials accessible via the internet was very successful and I think it can be long-lasting. … You can update your materials very easily when you’re dealing with an online environment. You can get to students who may be absent more easily because you get the response right away and so you can follow up with the teachers to work on your response rates that way. I think that moving over to an online environment for the intervention and the data coll ection was something really positive that happened. On the whole, almost all the researchers in our sample described flexibility as a strength in addressing shifting contexts. In the words of one researcher, Here it is important to be flexible with the folks that were working on projects, to recognize … that we’re all wrestling with challenges. Institutional Instability and Researcher Responses to Feasibility Challenges Posed Particular Threats to Internal, Construct, and External Validity Overall, the instability in the institutional environment identified by study participants poses threats to construct, internal, and external validity. As regards internal validity, the shifts to the basic structures of schooling present threats of history bias as it is difficult, if not impossible, now to disentangle the effects of the intervention from those of the pandemic. Turbulence in school settings also has the potential to affect the treatment or control condition, presenting the possible threat of treatment diffusion. Researchers who modified study timelines by delaying recruitment or implementation took steps to mitigate these validity threats by waiting for more “normalized” conditions to diminish the threat of history bias and implement an intervention or a policy with more fidelity to the intended model. Researchers that simplified or modified interventions, on the other hand, may have solved some practical issues with launching and conducting a study at the cost of raising some additional threats to construct validity and external validity. Treatment adaptations present challenges to construct validity because the effects of the enacted treatment might not reflect the effects of the intended treatment. Additionally, there is the possibility of and interaction of the causal relationship over treatment variations, such that an effect found with the treatment as modified for the pandemic context might not hold with other variations of that treatment (e.g., Shadish et al., 2002), posing a threat to external validity. Challenge 2. Difficulty with Study Recruitment Caused by Limited School and District Resources and Shifting Priorities Researchers experienced serious challenges to recruitment, exacerbated by limited district and school staff time as well as competing priorities that often superseded research participation. In about a quarter of the studies in our sample, recruitment challenges manifested as districts or schools dropping out entirely or refusing to enforce randomization. At the time of our interviews, many of the studies were actively recruiting for the experimental portions of their research and continuing to find research access and recruitment very challenging. Three respondents reported that, while districts were generally supportive in concept, they limited researchers from directly contacting schools or teachers, leading to substantial recruitment challenges. Another researcher noted that the influx of federal stimulus funds may have made “free” interventions accompanied by research, which are often offered as part of evaluations, less attractive, which also presented recruitment challenges. Research, interrupted 15 Situations arose where the district, or sometimes school leaders, sought to protect teachers from increased demands on their time amid widespread teacher burnout. One researcher identified the scope of excessive burden on teacher time as follows: School-based personnel are so overwhelmed with not only trying to do their own jobs but trying to do their own jobs plus the 50% of the job that somebody else was doing that left. … So they’re doing really a tremendous number of things just to keep the school afloat. … they also have less time than ever to dedicate to getting them [interventions] to work. …That’s like balancing the burden and knowing, even for research questions that the schools still see as important, how to carve out that time seems to be a real challenge. Substitute teacher shortages further constrained teachers’ participation in research. One researcher summarized this tension: “I think the biggest thing is schools are so overwhelmed right now that I think that the hardest thing is just to do research in schools is to be really respectful of where the schools are at.” Researchers Addressed Recruitment Challenges by Focusing on Partnerships and Allocating Funding to Support Staffing and Incentives To address these challenges, researchers took the approach of developing stronger partnerships within districts to build support for research and participation. First, most of the researchers in our sample discussed recognizing the main competing priorities facing districts. One researcher described this approach as, “realizing that research is not at the top of the agenda of districts and schools who are trying to put out fires.” In response, researchers sought to incentivize and support participation through other methods. Five researchers reported offering or increasing incentives for participation. Noting that administrators often had less time to advocate for research, three researchers said they created new liaison positions to help with recruitment. In these studies, researchers hired retired school staff as research liaisons or teacher recruiters. These individuals were knowledgeable about the district and school context and priorities and better able to communicate the importance of the research to local staff. Two respondents stated that the nature of researchers’ relationships with districts reportedly shifted. As one researcher described their pivot: I think we really pivoted to try to do more like a Research -Practice-Partnership approach in terms of asking the schools, “Well what do you want? What do you need? What would be helpful for you?” And I think that was well received in general and I hope that that resulted in, maybe not us collecting research data, but in the school being able to utilize some of the resources that have been created. These conversations sometimes led researchers to supplement their planned studies with pilot studies or dosage studies, or implementation and lessons learned analyses . Recruitment Difficulty and Researcher Responses to Recruitment Challenges Posed Particular Threats to Internal, External, and Statistical Conclusion Validity Overall, the instability in the institutional environment identified by study participants poses threats to internal, external, and statistical conclusion validity. In particular, schools and systems that have the capacity to support research may differ systematically from those who do not. This has two different implications for validity. First, in multi-site studies (those conducted in multiple schools or districts), if a subset of sites opts not to participate because of a lack of capacity, this potentially Education Policy Analysis Archives Vol. 32 No. 21 16 leads to attrition bias or selection bias. Second, if a planned or intended site is unable to participate and a new site is recruited, there are implications for external validity because the characteristics of those systems who do participate might restrict the populations to which results can be generalized. Another threat to validity was explicitly identified by study participants. Over half of the researchers we spoke with noted that recruitment challenges were likely to result in smaller realized study samples. This potentially poses a threat to statistical conclusion validity by making the study insufficiently powered to detect effects. Challenge 3. Difficulties with Data Availability, Concerns about Data Quality Due to Changes to State Assessment Programs A common challenge was the lack of data availability during the pandemic. In March 2020, state testing holds meant that no statewide assessment data were available, which presented a substantial challenge to studies that were designed to make use of (if not solely rely upon) these data. However, a second challenge commonly described by the researchers we interviewed was low or uneven participation rates in assessments even after testing holds were lifted in Spring 2021. One researcher described the challenge of limited student participation in state testing, stating, “If attrition [from our study] means that they [students in our sample] didn’t take the state tests, that has nothing to do with the actual study; that has everything to do with COVID.” Another researcher emphasized the importance of interrogating the complexity of noise in state testing data. As this researcher shared, It was pure chaos, but we still got impact results even amongst that chaos so in the grand scheme of things, I think it was a successful implementation, we’re just trying to figure out how to make sense of the noise. It wasn’t random noise, it was definitely COVID noise. Challenges arose with use of other forms of data as well. About a quarter of researchers noted that virtual instruction precluded planned classroom observation data crucial to studies of instructional practice and classroom interaction. Virtual observations, in these studies, were inadequate measures of interaction and practice, given the substantive differences to instruction in virtual versus in-person settings. Even administrative data, like attendance, were sometimes altered by districts in response to the pandemic context. For example, one researcher noted they needed to request more fine-grain data than usual, given shifts in metrics. As this researcher explained, Everybody measured it [attendance] differently. And even now, in reality, there’s differences. You have to be really careful when you get that data and know exactly what they count as a day and not a day … At this point, we don’t get summarized data; we ask for the individual data on the individual kids so that we can do what we want to do with it. Researchers Addressed Challenges in Data Quality and Availability by Modifying Their Outcome Measures In response to data quality challenges, researchers took two main approaches. About a quarter of respondents reported altering the key outcome measures of the study, shifting away from a reliance on statewide summative assessments and placing more emphasis on other assessments such as the PSAT. Three researchers noted they built their own assessments or surveys, making use of internet and smartphone-based technology to facilitate high response rates. Researchers who pivoted towards purpose-built assessments and surveys administered online, believed these data Research, interrupted 17 better reflected the aims of the intervention. Other forms of data collection included recorded classroom instruction (where school access was limited), as well as qualitative data collection to better understand implementation. Data Quality and Researcher Responses to Data Quality Challenges Posed Particular Threats to Internal, Construct, and External Validity Overall, the instability in the institutional environment identified by study participants poses threats to construct, internal, and external validity. Many of these threats were explicitly identified by study participants. About a third of researchers raised concerns about the implications of missing data for the internal validity of these state test results, with a particular focus on concerns of attrition bias and selection bias. For example, one researcher noted patterns in which students did not complete the state test due to parent opt-out. In this study, the researcher noted that students who had selected in-person instruction were more likely to take the state test than their peers who opted to continue virtual instruction. These two populations of students, however, differed systematically both on prior performance and in race/ethnicity. As one researcher summarized, “the problem is that through our analysis of state results is like, that missing data isn’t like random.” Another researcher raised concerns about the use of state tests for research given possible attrition bias, as students who were tested may have differed systematically from those who did not take state tests, and instrumentation bias, as students might be tested virtually versus in-person based on whether they were accessing virtual or in person instruction. As this researcher shared, “We’re concerned not just, ‘Will the state tests be there?’ but ‘how well we can trust those results?’” Modifying outcome measures is one way to mitigate the threats to validity posed by concerns that missing state testing data is likely to induce attrition bias and selection bias. However, modifying outcome measures also raises some threats to validity that merit consideration. First, there is the threat of instrumentation bias that arise any time an assessment program is changed, because changes in assessment may be conflated with a treatment exposure effect. Second, it is unclear to what extent new assessments might vary from the intended constructs for the study, potentially raising questions about construct validity. Finally, using new assessments may have consequences for external validity, because using study-specific measures may make it difficult to compare effects across studies, or impede future researchers’ capacity to replicate study results or include such studies in meta-analyses. Concerns with Extrapolation and Generalization Across the challenges described above, we find immediate implications for the quality and rigor of individual studies, as identified by our respondents. While researchers were able to adjust research plans to sustain individual studies, another concern is external validity, or the extent to which research findings can be generalized across studies or to other settings. About half of the researchers in our sample expressed concerns that research studies from the pandemic period could not be generalized to other contexts, due to the pandemic’s impacts on school learning environments, staffing, and differential impacts on specific student populations. As one researcher noted, “The learning environment was so incredibly different than anything we’ve seen in the past and hopefully anything we see in the future”—referring not only to the shift in conditions for the treatment condition, but also to the counterfactual. Indeed, understanding the counterfactual itself was difficult. Three researchers noted that district responses to the pandemic (e.g., shutdowns, virtual, hybrid, in-person) amplified existing inequities. In such settings, it is difficult to isolate the effect of the intervention, where the counterfactual itself may have shifted in varied and inconsistent ways. While thoughtful about the shifts in the counterfactual, no researchers Education Policy Analysis Archives Vol. 32 No. 21 18 in our study described systematically measuring the business-as-usual conditions under the pandemic. Systematic analysis of how hybrid or virtual settings might have influenced peer effects were uncommon, despite awareness of such potential issues among researchers. Nonetheless, six researchers suggested that changes like virtual learning and limited school staffing that would likely persist in the future, enhancing generalizability of pandemic research. One such researcher shared, In some ways the information during the pandemic is unique and those conditions aren’t exactly the same, but some of those conditions are going to continue to be part of schooling and should be addressed. Four researchers believed that the pandemic conditions merely narrowed the range of generalizability, such that findings could only be generalized to similar settings, grade levels, or types of interventions. Conclusion It is widely accepted that, even before the pandemic, there were myriad practical and methodological challenges that threatened both the viability of education research and the validity of research findings (e.g., Myers, 2014). As one researcher summarized, “logistics are the things that break almost every study.” However, there has never been a disruption to schooling in modern history as profoundly impactful and widespread as the COVID-19 pandemic. And while there is a growing body of work in international development about conducting research in fragile contexts, including in times of conflict or in the midst of humanitarian crises (Bakrania et al., 2021; Hassnain et al., 2021; Nene Odjidja & Alves do Reis, 2021; Puri et al., 2017), there is little research evidence on conducting studies of education policies and programs when faced with large-scale school disruptions. This study begins to fill this gap by utilizing interview data to describe common challenges faced by researchers conducting studies during the COVID-19 pandemic, as well as the strategies employed by researchers to successfully navigate this turbulent time. Overall, this study finds the disruptions to schooling caused by the COVID-19 pandemic showed how difficult it can be for researchers to maintain a balanced focus on potential threats to internal, construct, statistical conclusion, and external validity when conducting an education study under exceedingly turbulent conditions. In other words, the uncertain and unstable schooling conditions presented by the pandemic inevitably surfaced tensions among these aspects of high-quality studies, as researchers were forced to address issues reactively, without opportunities for collaboration or collective sensemaking (Grissom & Condon, 2021). Based on our findings and supporting literature we conclude by offering three suggestions to help educational policy researchers plan ahead for planning and conducting education studies in the context of future large-scale school disruptions. Considering these suggestions while planning mitigate both practical and methodological challenges posed by large-scale school disruptions. In particular, these suggestions encourage researchers to proactively consider issues of extrapolation and generalization, maximizing the potential for such studies to contribute to our broader understanding of how to improve teaching and learning. Flexible and Adaptive Designs Are Important for Ensuring that Research Remains Feasible and that Findings Remain Useful Part of a flexible or adaptive design involves contingency planning, which may involve including a “cushion” in the study implementation schedule, or planned adaptations to data Research, interrupted 19 collection procedures if situations arise that compromise the feasibility of a particular procedure. Adaptive design also involves planning on using multiple measures or mixed methods for data collection. Several authors note that planning to use a mix of qualitative and quantitative methods can be a productive way to proactively respond to the practical and methodological challenges of conducting research during a large-scale disruption (Chirambwi, 2023; Nene Odjidja & Alves do Reis, 2021). Flexible planning may also include allocating additional project resources to support recruitment, including financial resources for incentives, or products that are of immediate use to schools. This kind of planning may allay logistical issues, while ensuring research is ethically conducted. Establish and Cultivate Trusting Partnerships with Schools and Districts Trusting partnerships ensure that researchers and educators agree with and are committed to the objectives of the study (López Turley and Stevens 2015). This can be critical in mitigating recruitment challenges and ensuring that research remains beneficial to participants as well as to the field. In particular, potential research participants may be more trusting of school system officials than outside researchers (Bartlett et al., 2017; Bruzzese et al., 2009), and some research suggests that school partnerships may increase the likelihood of successfully recruiting historically underserved students, including English learners and students experiencing poverty (Alibali & Nathan, 2010; Bartlett et al., 2017; Lamb et al., 2001). Partnerships focused on recruitment can help to mitigate the possibility of selection bias. Additionally, strong partnerships can also mitigate other threats to validity by creating opportunities for effective communication between researchers, program implementers, and other district personnel (López Turley & Stevens, 2015) so that researchers can better document and transparently report shifts in implementation and to understand counterfactual conditions more clearly. Document Modifications to Study Designs, Changes to Instruments, and Information about Students’ Learning Contexts Research conducting during the pandemic was conducted in the midst of systemic turbulence and instability. Our study found that generalizability and extrapolation were of significantly lower priority for researchers than practical issues of study feasibility, or mitigating threats to internal validity. However, documentation about these key variables is critical for understanding how these aspects of the study shape inferences about the effectiveness of a program or policy (Hill et al., 2022), and for increasing the credibility of local effectiveness predictions made by practitioners and policymakers (Joyce & Cartwright, 2020). In general, researchers suggest that such information is under-reported in academic papers (Hill et al., 2022), and though SEER standards do mention the importance of documenting implementation, there are not clear and consistent standards for publishing such contextual information across journals (Hill et al., 2022). Ultimately, this kind of documentation provides consumers crucial information to consider the external validity of the study and how this study may fit into a broader body of literature, including replication studies and meta-analyses. In addition, in documenting these shifts, we suggest that researchers intentionally and explicitly consider the potential implications that in-stream adjustments to study design, implementation and analysis may have for internal validity, construct validity, statistical conclusion validity and external validity. While the complex ecosystem of schools and turbulent conditions will always necessitate making compromises, reporting these trade-offs will help to strengthen the field through more comprehensive knowledge in the presence of complex research conditions. Education Policy Analysis Archives Vol. 32 No. 21 20 References Alibali, M. W., & Nathan, M. J. (2010). Conducting research in schools: A practical guide. Journal of Cognition and Development, 11(4), 397-407. https://doi.org/10.1080/15248372.2010.516417 Bakrania, S., Balvin, N., Daidone, S., & De Hoop, J. (2021). Impact evaluation in settings of fragility and humanitarian emergency (Report No. DP-2021-02). UNICEF Office of Research. https://www.unicef-irc.org/publications/pdf/Impact-Evaluation-in-Settings-of-Fragility- and-Humanitarian-Emergency.pdf Bartlett, R., Wright, T., Olarinde, T., Holmes, T., Beamon, E. R., & Wallace, D. (2017). Schools as sites for recruiting participants and implementing research. Journal of Community Health Nursing, 34(2), 80-88. https://doi.org/10.1080/07370016.2017.1304146 Briggs, D. C. (2008). Comments on Slavin: Synthesizing causal inferences. Educational Researcher, 37(1), 15-22. https://doi.org/10.3102/0013189X08314286 Bruno, P., & Goldhaber, D. (2021). What pandemic-related test waiver requests suggest about states’ testing priorities. Phi Delta Kappan, 103(3), 48-53. https://doi.org/10.1177/00317217211058525 Bruzzese, J. M., Gallagher, R., McCann‐Doyle, S., Reiss, P. T., & Wijetunga, N. A. (2009). Effective methods to improve recruitment and retention in school‐based substance use prevention studies. Journal of School Health, 79(9), 400-407. https://doi.org/10.1111/j.1746- 1561.2009.00427.x Bush-Mecenas, S., & Marsh, J. A. (2018). The DIVE approach: Using case-ordered meta-matrices and theory-based data displays to analyze multiple case study data. In C. R. Lochmiller (Ed.), Complementary research methods for educational leadership and policy studies (pp. 33–56). Palgrave MacMillan. https://doi.org/10.1007/978-3-319-93539-3_3 Campbell, D. & Stanley, J. (1963). Experimental and quasi-experimental designs for research. Rand-McNally. Chirambwi, K. (2023). Rethinking research methods in protracted violent conflicts in Mozambique: Fieldwork in complex emergencies. Journal of Humanitarian Logistics and Supply Chain Management. https://doi.org/10.1108/JHLSCM-08-2022-0088 Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Houghton Mifflin. Dee, T. S., & Murphy, M. (2021). Patterns in the pandemic decline of public school enrollment. Educational Researcher, 50(8), 566-569. https://doi.org/10.3102/0013189X211034481 Diliberti, M., Schwartz, H., & Grant, D. (2021). Stress topped the reasons why public school teachers quit, even before COVID-19 (Report No. RRA1121-2). RAND Corporation. https://www.rand.org/pubs/research_reports/RRA1121-2.html Gorin, J. S. (2006). Test design with cognition in mind. Educational Measurement: Issues and Practice, 25(4), 21-35. https://doi.org/10.1111/j.1745-3992.2006.00076.x Grissom, J. A., & Condon, L. (2021). Leading schools and districts in times of crisis. Educational Researcher, 50(5), 315-324. https://doi.org/10.3102/0013189X211023112 Groff, J. (2013). Dynamic systems modeling in educational system design & policy. Journal of New Approaches in Educational Research, 2(2), 72-81. http://dx.doi.org/10.7821/naer.2.2.72-81 Gross, B., & Opalka, A. (2020, June). Too many schools leave learning to chance during the pandemic. Center on Reinventing Public Education. https://crpe.org/too-many-schools-leave-learning-to- chance-during-the-pandemic/ https://doi.org/10.1080/15248372.2010.516417 https://www.unicef-irc.org/publications/pdf/Impact-Evaluation-in-Settings-of-Fragility-and-Humanitarian-Emergency.pdf https://www.unicef-irc.org/publications/pdf/Impact-Evaluation-in-Settings-of-Fragility-and-Humanitarian-Emergency.pdf https://doi.org/10.1080/07370016.2017.1304146 https://doi.org/10.3102/0013189X08314286 https://doi.org/10.1177/00317217211058525 https://doi.org/10.1111/j.1746-1561.2009.00427.x https://doi.org/10.1111/j.1746-1561.2009.00427.x https://doi.org/10.1007/978-3-319-93539-3_3 https://doi.org/10.1108/JHLSCM-08-2022-0088 https://doi.org/10.3102/0013189X211034481 https://www.rand.org/pubs/research_reports/RRA1121-2.html https://doi.org/10.1111/j.1745-3992.2006.00076.x https://doi.org/10.3102/0013189X211023112 https://crpe.org/too-many-schools-leave-learning-to-chance-during-the-pandemic/ https://crpe.org/too-many-schools-leave-learning-to-chance-during-the-pandemic/ Research, interrupted 21 Hassnain, H., Kelly, L. & Simona Somma, S. (Eds.). (2021). Evaluation in contexts of fragility, conflict and violence: Guidance from global evaluation practitioners. IDEAS. https://ideas-global.org/wp- content/uploads/2021/06/EvalFCV-Guide-web-A4-HR.pdf Hedges, L., & Tipton, E. (2020). Addressing the challenges to educational research posed by COVID-19 (Report No. WP-20-47). Northwestern Institute for Policy Research. https://www.ipr.northwestern.edu/our-work/working-papers/2020/wp-20-47.html Hill, H., Lynch, K., Polanin, J., Stuart, E. & Ochsendorf, R. (2022, September 21–24). Making meta- analyses easy: Guidelines for reporting on classroom-level instructional interventions. [Conference presentation] SREE 2022 Conference, Washington, D.C., United States. https://sree.confex.com/sree/2022/meetingapp.cgi/Session/1624 Institute of Education Sciences. (2022a). Standards for excellence in education research. https://ies.ed.gov/seer/ Institute of Education Sciences. (2022b). School pulse panel. https://ies.ed.gov/schoolsurvey/spp/ Inter-agency Network for Education in Emergencies. (2020). 20 years of INEE: Achievements and challenges in education in emergencies. https://inee.org/resources/20-years-of-inee Jahan, F. A., Zviedrite, N., Gao, H., Ahmed, F., & Uzicanin, A. (2022). Causes, characteristics, and patterns of prolonged unplanned school closures prior to the COVID-19 pandemic—United States, 2011–2019. PLoS ONE, 17(7), Article e0272088. https://doi.org/10.1371/journal.pone.0272088 Joyce, K. E., & Cartwright, N. (2020). Bridging the gap between research and practice: Predicting what will work locally. American Educational Research Journal, 57(3), 1045-1082. https://doi.org/10.3102/0002831219866687 Lamb, J., Puskar, K. R., & Tusaie-Mumford, K. (2001). Adolescent research recruitment issues and strategies: application in a rural school setting. Journal of Pediatric Nursing, 16(1), 43-52. https://doi.org/10.1053/jpdn.2001.20552 López Turley, R. N., & Stevens, C. (2015). Lessons from a school district–university research partnership: The Houston Education Research Consortium. Educational Evaluation and Policy Analysis, 37(1), 6S-15S. https://doi.org/10.3102/0162373715576074 Miles, M. B., Huberman, A. M., & Saldaña, J. (2020). Qualitative data analysis: A methods sourcebook. Sage. Myers, E. L. (2014). Taming turbulence: An examination of school leadership practice during unstable times. Professional Educator, 38(2), n2. Nayak, A., Islam, S. J., Mehta, A., Ko, Y., Patel, S. A., Goyal, A., & Quyyumi, A. A. (2020). Impact of social vulnerability on COVID-19 incidence and outcomes in the United States. medRxiv. https://doi.org/10.1101/2020.04.10.20060962 Nene Odjidja, E., & Alves do Reis, A. (2021, December 7) Learning from adaptive evaluation approaches in dynamic conflict-afflicted and humanitarian settings. Evaluation Matters Magazine: Learning from Successes and Failures in Evaluation. https://idev.afdb.org/en/content/learning-adaptive-evaluation-approaches-dynamic- conflict-afflicted-and-humanitarian Peele, H., & Riser-Kositsky, M. (2021). Map: Coronavirus and school closures in 2019–2020. [Data set]. Education Week. https://www.edweek.org/leadership/map-coronavirus-and-school- closures-in-2019-2020/2020/03 Plucker, J. A., & Makel, M. C. (2021). Replication is important for educational psychology: Recent developments and key issues. Educational Psychologist, 56(2), 90-100. https://doi.org/10.1080/00461520.2021.1895796 https://ideas-global.org/wp-content/uploads/2021/06/EvalFCV-Guide-web-A4-HR.pdf https://ideas-global.org/wp-content/uploads/2021/06/EvalFCV-Guide-web-A4-HR.pdf https://www.ipr.northwestern.edu/our-work/working-papers/2020/wp-20-47.html https://sree.confex.com/sree/2022/meetingapp.cgi/Session/1624 https://ies.ed.gov/seer/ https://ies.ed.gov/schoolsurvey/spp/ https://inee.org/resources/20-years-of-inee https://doi.org/10.1371/journal.pone.0272088 https://doi.org/10.3102/0002831219866687 https://doi.org/10.1053/jpdn.2001.20552 https://doi.org/10.3102/0162373715576074 https://doi.org/10.1101/2020.04.10.20060962 https://idev.afdb.org/en/content/learning-adaptive-evaluation-approaches-dynamic-conflict-afflicted-and-humanitarian https://idev.afdb.org/en/content/learning-adaptive-evaluation-approaches-dynamic-conflict-afflicted-and-humanitarian https://www.edweek.org/leadership/map-coronavirus-and-school-closures-in-2019-2020/2020/03 https://www.edweek.org/leadership/map-coronavirus-and-school-closures-in-2019-2020/2020/03 https://doi.org/10.1080/00461520.2021.1895796 Education Policy Analysis Archives Vol. 32 No. 21 22 Puri, J., Aladysheva, A., Iversen, V., Ghorpade, Y., & Brück, T. (2017). Can rigorous impact evaluations improve humanitarian assistance?. Journal of Development Effectiveness, 9(4), 519-542. https://doi.org/10.1080/19439342.2017.1388267 Robinson, C. D., Kraft, M. A., Loeb, S., & Schueler, B. E. (2021). Accelerating student learning with high- dosage tutoring. The Annenberg Instititue for School Reform. https://annenberg.brown.edu/sites/default/files/EdResearch_for_Recovery_Design_Princi ples_1.pdf Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin. Slavin, R. E. (2008). Perspectives on evidence-based research in education—What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5-14. https://doi.org/10.3102/0013189X08314117 U.S. Department of Education. (2002). No Child Left Behind: A desk-top reference. https://eric.ed.gov/?id=ED471334 Yin, R. K. (2009). Case study research: Design and methods (Vol. 5). Sage. About the Authors Susan Bush-Mecenas RAND sbushme@rand.org https://orcid.org/0009-0002-3705-7652 Susan Bush-Mecenas is the policy researcher at the RAND Corporation. Her research examines the implementation of PreK–12 education policies and practices, with attention to the interaction of school systems, their organizational and institutional context, and educators and school leaders as agents of change. Jonathan D. Schweig RAND jschweig@rand.org https://orcid.org/0000-0003-1950-0358 Jonathan D. Schweig is a senior social scientist at RAND. His research focuses on the measurement of teaching, and how safe and supportive learning environments can nurture positive adolescent development. Megan Kuhfeld NWEA Megan.kuhfeld@hmhco.com https://orcid.org/0000-0002-2231-5228 Megan Kuhfeld is the Director of Growth Modeling and Analytics at NWEA, a division of HMH. Her research seeks to understand students’ academic and social -emotional learning trajectories and the school and neighborhood influences that promote optimal growth. Louis T. Mariano RAND loum@rand.org https://orcid.org/0000-0001-5163-7384 https://doi.org/10.1080/19439342.2017.1388267 https://annenberg.brown.edu/sites/default/files/EdResearch_for_Recovery_Design_Principles_1.pdf https://annenberg.brown.edu/sites/default/files/EdResearch_for_Recovery_Design_Principles_1.pdf https://doi.org/10.3102/0013189X08314117 https://eric.ed.gov/?id=ED471334 mailto:sbushme@rand.org https://orcid.org/0009-0002-3705-7652 mailto:jschweig@rand.org https://secure-web.cisco.com/15-1KcF9eLl4LdhxXBDnMf69OqOs_Y01T1s4W_6oEOBTMYqqJ6dN74_Hr0Rm9KxySgQk78C0faC5q9cb_Sbqy9LCNARkOo4bu0xUa_tVeezk1Z1qsFn0VBelyGJn3_eZ_sYfXg4OEO_dKt6uwVjSaOWEzL-83bbCA-lq8U8QNSnyGwsOzzfrfvDNLZXVp_512Pw4E2-hyu6gHdNX7rjXMV6tztSW-RBqyAorPemAGSwbViEKpqT847Cr-uB5O_BE-woEuU586fdMuLdKrfujgs412tSLbMt-ZGhk5J8IzakhzTxRPWq18CAVUn-OOPViuB1fJLMLafUmVkOT-YPF9UqljWrRUYCc-rJA6oqKEyMPR4b2MwfwSg2Eq96XpiweJ/https%3A%2F%2Forcid.org%2F0000-0003-1950-0358%3Flang%3Den mailto:Megan.kuhfeld@hmhco.com https://secure-web.cisco.com/1buKj9D7c6tlgQ_6zxXWkhMrQHDOPGkKCwOWYsM_05Fn4OH3CbbdO2fOo_egDinFPm4Pz_buVg0mM2n8CjvTCnIuZEEoGSuGauIKGl4tqM7erlx_LAvIblo7dSe-WOEQ_Cbvq7ykVB-12NPGKge85u-i5RdhKQ9sEXzhXpgAL0jvToUzJmVt86TXjhIRKixZGWzdYKPQeuw6njqJxWzhX9KzuAmwzetJRv4e6kfvnx8_DQjF2dWk9Hz0JNiyfJOP0qLmGt5z5mGzLvQHJSHaLyR9b_qIRBIql-BLt-hFfJiPImo70u4Z6_tHn7y0jUCchT5AHsjUkqpVMl-3IW_dvFfO-CIm8egQ35fhffeVhplZl05L2U5uaJntuvr9K7jPV/https%3A%2F%2Forcid.org%2F0000-0002-2231-5228 mailto:loum@rand.org https://orcid.org/0000-0001-5163-7384 Research, interrupted 23 Louis T. Mariano is a senior statistician at RAND. His research focuses on evaluation of the efficacy of education policy programs and reforms. Melissa K. Diliberti RAND mdiliber@pardeerand.edu https://orcid.org/0009-0002-6035-3097 Melissa Diliberti is an assistant policy researcher at RAND and a Ph.D. candidate at the Pardee RAND Graduate School. Her research uses nationally representative surveys of educators to investigate how the COVID-19 pandemic has changed the U.S. education system. education policy analysis archives Volume 32 Number 21 April 9, 2024 ISSN 1068-2341 Readers are free to copy, display, distribute, and adapt this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, the changes are identified, and the same license applies to the derivative work. More details of this Creative Commons license are available at https://creativecommons.org/licenses/by-sa/4.0/. EPAA is published by the Mary Lou Fulton Teachers College at Arizona State University. Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A1 (Brazil), SCImago Journal Rank, SCOPUS, SOCOLAR (China). About the Editorial Team: https://epaa.asu.edu/ojs/index.php/epaa/about/editorialTeam Please send errata notes to Jeanne M. Powers at jeanne.powers@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. mailto:mdiliber@pardeerand.edu https://orcid.org/0009-0002-6035-3097 https://creativecommons.org/licenses/by-sa/4.0/ http://www.doaj.org/ http://www.doaj.org/ https://epaa.asu.edu/ojs/index.php/epaa/about/editorialTeam mailto:mailtojeanne.powers@asu.edu https://www.facebook.com/EPAAAAPE