© 2021 Children, Youth and Environments Children, Youth and Environments 31(1), 2021 Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM Framework Dana Engel Gerstein Agriculture and Natural Resources, Nutrition Policy Institute University of California Berkeley Carolyn R. Bates Division of Developmental and Behavioral Health Children’s Mercy Kansas City Amy M. Bohnert Department of Psychology, Loyola University Chicago Citation: Gerstein, D. E., Bates, C. R., & Bohnert, A. M. (2021). Evaluating a green schoolyard transformation: A protocol utilizing the RE-AIM framework. Children, Youth and Environments, 31(1), 187-198. Retrieved from http://www.jstor.org/action/showPublication?journalCode=chilyoutenvi Abstract Increasing access to quality outdoor space in low-income communities may lead to a host of child, community, and environmental benefits. The Space to Grow initiative transforms barren schoolyards in low-income communities across Chicago into vibrant green spaces for children and community members to play and learn among natural elements. This paper presents the Space to Grow health and wellness evaluation protocol designed to assess the five dimensions of the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, Maintenance; King et al, 2010), which assesses the comprehensive public health impact of green schoolyards on children and their communities and the transferability of the various elements of the initiative. Keywords: green schoolyards, RE-AIM evaluation framework, equitable built environment, access to green space, health promotion strategy http://www.jstor.org/action/showPublication?journalCode=chilyoutenvi Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 188 Introduction Low-income communities have less access to quality green space than higher- income communities, leading to fewer opportunities for outdoor recreation, physical activity, relaxation and restoration, learning, and connectedness with nature (Cohen, Inagami, & Finch, 2008; Chawla, Keena, Pevec, & Stanley, 2014; Gordon- Larsen, Nelson, Page, & Popkin, 2006; Moore, Diez Roux, Evenson, McGinn, & Brines, 2008). Increasingly, interventions are addressing social and physical inequities in communities to support healthy behaviors (Wolfenden et al., 2014). Health-promotion strategies targeting the social and built environment have the potential not only to provide a multitude of individual-level benefits (such as increasing access to resources), but also to positively impact community-level outcomes (such as increasing social capital and social cohesion; King, Glasgow, & Lee-Castillo, 2010). Thus, increasing access to green space may be one way to improve social and health equity in low-income urban communities (Stevenson et al., 2020). The purpose of this study was to investigate how building green schoolyards in low-income communities may address health inequity by increasing access to quality green space. Green schoolyards are defined as multi-functional school grounds, designed by and for the entire school community, which include places for students, teachers, parents and community members to play, learn, explore and grow. During out-of- school time, these schoolyards are intended to be used as community spaces (Children & Nature Network, 2018). By increasing access to quality outdoor space and promoting its use, green schoolyards promote: (1) physical activity, both during and outside of school hours (Gordon-Larsen et al., 2006; Raney, Hendry, & Yee, 2019); (2) stress reduction and increased resilience (Chawla et al., 2014), (3) prosocial interactions (Bates, Bohnert, & Gerstein, 2018); and (4) reduced absences and disciplinary actions in school (Children & Nature Network, 2018). Additionally, green schoolyards may serve as gathering spaces for community members. Community gathering spaces enhance community well-being through increased collective efficacy, an indicator of neighborhood social cohesion and social capital (Cohen et al., 2008). Although foundational research efforts suggest several benefits of green schoolyards, more comprehensive studies are needed to assess the individual, school, and community impact of green schoolyards as a built environment intervention. This field report documents a multi-method, theory-driven health and wellness evaluation protocol for a green schoolyard transformation initiative, Space to Grow (STG), which has been taking place in Chicago, Illinois. STG is co-managed by two not-for-profit local organizations, Healthy Schools Campaign and Openlands, and linked capital funds, expertise, and leadership from Chicago Public Schools (CPS), the Chicago Department of Water Management, and the Metropolitan Water Reclamation District of Greater Chicago. The aim of the STG initiative is to transform Chicago public schoolyards in low-income communities into vibrant green spaces for students, parents and community members to play, learn, and be outside (Figures 1 and 2). Schoolyard transformations prioritize creating spaces that promote physical activity, outdoor learning, and community engagement. In addition to physically transforming the schoolyard, STG utilizes a multi-year support Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 189 plan that focuses on engaging the entire community in the design process, providing hands-on garden activities to students, training teachers to incorporate the schoolyard into lesson plans, and organizing community workshops and events to promote schoolyard utilization. To date, STG has transformed 20 green schoolyards at elementary schools in Chicago, and the partners expect to complete a total of 34 schoolyards by 2022 (Healthy Schools Campaign and Openlands, 2020). Figure 1. John Cook Elementary School before and after the Space to Grow schoolyard transformation Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 190 Figure 2. Wadsworth Elementary School before and after the Space to Grow schoolyard transformation The protocol described in this field report assessed the impact of green schoolyard transformations on student, school, and community well-being. The protocol was developed by an interdisciplinary team of social scientists utilizing the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, Maintenance; King et al, 2010). Developed to enhance the impact of interventions on population health, the RE-AIM framework seeks to evaluate the five dimensions considered most relevant to real-world implementation (King et al., 2010). Using the RE-AIM framework allowed decision makers and others to focus on the elements of STG impacting children and their communities the most and to compare the public health impacts across the five framework dimensions. This field report describes a Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 191 multi-method assessment strategy utilizing both quantitative and qualitative methods (e.g., observational coding, self-reporting, and interviews) to evaluate the impact of STG green schoolyard transformations on students, schools, and communities. Protocol The current study adapted the RE-AIM framework dimensions to fit the scope of this built environment initiative. The dimensions of the RE-AIM framework, as well as the study outcomes, assessment tools, and data collection timeline, are described below and depicted in Table 1. RE-AIM Framework Dimensions Reach, the first dimension of the adapted RE-AIM framework, focused on measuring characteristics of children and their communities receiving the initiative. Using publicly available, aggregate city- and school-level data, we examined of the location and demographics of the schools and surrounding neighborhoods to characterize the communities where green schoolyard transformations took place, including racial/ethnic characteristics, socio-economic status (i.e., proportion of students qualifying for free or reduced-price lunch), and neighborhood crime statistics. To further characterize the community, we interviewed school principals, collected geographical data via geographic information system (GIS) technology, and interviewed key school and neighborhood personnel to learn about the surrounding community (school key-informant interviews). Effectiveness, the second dimension of the adapted RE-AIM framework, considered the effects of the initiative on children and their schools and communities. To address this dimension, we examined changes in student health behaviors (e.g., physical activity), academic outcomes, the school environment (e.g., bullying), and school-community engagement and cohesion from pre- to post-schoolyard transformation. We minimized the impact of data collection on students by collecting de-identified, observational data on physical activity and social interactions on the schoolyard via behavioral mapping methodology (Cosco, Moore, & Islam, 2010; Cosco, Moore, & Smith, 2014). In addition, we surveyed caregivers/parents, school staff, and community members to assess perceptions of schoolyard safety (e.g., injuries and bullying) and the neighborhood environment (e.g., collective efficacy, perceptions of safety and cohesion). Finally, we mined publicly available, aggregate city- and school-level data to evaluate outcomes over time (e.g., standardized test scores, number of school suspensions and expulsions, crime statistics, surrounding real estate values). The adapted RE-AIM framework dimension of Adoption focused on how often the renovated schoolyard was being used both during school and in out-of-school times. We again prioritized minimizing burden to children by objectively measuring schoolyard utilization via observational data using the behavioral mapping methodology at various points throughout the day and week (before, during and after school as well as weekends). Additionally, we surveyed caregivers/parents, school staff, and community members to assess how often they utilized the schoolyard, and at what times of day. Finally, we interviewed key school informants Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 192 (e.g., principals), who shared their perceptions of change in schoolyard utilization over time as well as usage barriers and facilitators. The Implementation dimension of the RE-AIM framework measured the degree to which the STG initiative was implemented as intended across schools (i.e., intervention fidelity). We assessed implementation on two levels: (1) the community-engagement process (i.e., community involvement at each stage of the transformation); and (2) the amenities and features of the newly transformed green schoolyard (e.g., adherence/fidelity to school and community needs). We utilized a robust and multi-method approach to assess community engagement. First, we collected attendance records from community-engagement activities and administered a process survey to individuals who attended pre-transformation, community engagement events. The surveys were developed drawing on social, organizational, and educational psychology literature to assess community needs, wants, and engagement around the schoolyard transformation. Second, we conducted structured, “process checklist” interviews with interventionists to assess that actual project’s fidelity to initially stated goals. This interview was developed using the STG School Manual which detailed the community-engagement process from schoolyard planning and design to transformation and maintenance. The interview-administered tool combined a quantitative assessment of each step of the community engagement process with a qualitative assessment of the interventionist’s experience implementing the specific step of the process. We additionally asked key school informants (e.g., school principals) to report on perceptions of intervention fidelity during semi-structured school key-informant interviews. Finally, to assess fidelity to the amenities and features of the transformed schoolyards, as well as their condition, researchers objectively assessed schoolyard characteristics using a structured observation form, which was adapted from a well- validated observational tool from the Robert Wood Johnson Bridging the Gap research program (2012). Maintenance, the fifth adapted RE-AIM framework dimension, examined the long- term sustainability of the initiative by considering individual and community-level health outcomes, as well as the sustainability and maintenance of the physical space. To assess the sustainability of student- and community-level health and wellbeing outcomes, we collected follow-up survey data from caregivers/parents, teachers and school staff, and community members on utilization, physical activity, academic outcomes, the school environment, and neighborhood cohesion (Caregiver/Parent, School Staff & Community Stakeholder Survey). Follow-up data were collected at regular interval periods after the completion of the intervention (8, 24, and 32 months post-intervention). To assess the maintenance of the schoolyard features, researchers collected follow-up assessments of the physical space and features using the schoolyard observation form. Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 193 Table 1. Space to Grow health and wellness evaluation protocol Assessment Tool RE-AIM Dimension Outcome Data Collection Timeline Publicly available, aggregate city- and school-level data Reach Reach, Effectiveness & Maintenance Community characteristics (e.g. crime statistics) School characteristics (e.g. student demographics), academic performance & disciplinary action Pre- & Post- schoolyard transformation Geographic Information System (GIS) technology Reach Access to other outdoor green space Pre-schoolyard transformation School key-informant interview Reach Adoption Implementation Community contextual variables (e.g., community assets & priorities) Usual schoolyard utilization: barriers and facilitators to utilization Fidelity of the community- engagement process Pre- & Post- schoolyard transformation Behavioral mapping methodology Effectiveness & Maintenance Effectiveness & Maintenance Adoption Observed physical activity (5-point Child Activity Rating Scale) Observed social interactions Observed schoolyard utilization & characteristics of persons utilizing schoolyard space Pre- & Post- schoolyard transformation Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 194 Assessment Tool RE-AIM Dimension Outcome Data Collection Timeline Caregiver/parent, school staff & community stakeholder survey Effectiveness & Maintenance Effectiveness & Maintenance Adoption & Maintenance School environment: • Perceptions of schoolyard injuries & bullying • Trust in institution Neighborhood environment: • Collective efficacy (Social cohesion, & Trust subscale • Perceptions of neighborhood safety • Neighborhood cohesion Perception of schoolyard utilization Pre- & Post- schoolyard transformation Process survey Adoption & Implementation Fidelity of the STG community– engagement process Pre-schoolyard transformation Attendance records from community- engagement activities Implementation Number of students, school staff and community members that participate in the community- engagement process Pre- & Post- schoolyard transformation Process checklist interviews Implementation Qualitative and quantitative assessment of the STG community- engagement process collected from the interventionist Pre- & Post- schoolyard transformation Schoolyard observation form Implementation Maintenance Observed outdoor amenities & features Observed incivilities & features Pre- & Post- schoolyard transformation Discussion Low-income communities have less access to safe, quality outdoor spaces than higher-income areas, resulting in unequal opportunities for physical activity, stress relief, and community engagement (Moore et al., 2008). Increasingly, health- promoting strategies include designing more equitable built environments (Wolfenden et al., 2015). We used the RE-AIM framework, adapted for environmental change (King et al., 2010), to guide the STG health and wellness evaluation. The framework addressed inequities and supported children and their communities by collecting robust data, with representation from various members of the community, and with minimal burden to children. Throughout the evaluation Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 195 process, we utilized the RE-AIM framework to address inequities by gathering perspectives from both authoritative figures in the schools and communities (e.g., principals, members of the parent-teacher alliance, district aldermen), as well as lay community members (e.g., school staff, crossing guards, community neighbors), aiming to capture a breadth of perspectives and minimize evaluation bias. The evaluation’s minimal burden to children was exemplified in the ways that we capitalized on observational methodologies, secondary data analysis, and data from a wide range of adult informants. One aspect of this evaluation that worked particularly well was the observational data collection (behavioral mapping methodology). Observational methodologies allow researchers to capture objective data from youth, while minimizing the youth research burden. We initially wondered about the acceptability of observational methodologies on the schoolyards; however, we found that the schools, communities, and children were accepting of the presence of researchers in the schoolyard when they learned that research was related to the STG schoolyard transformation. Moreover, children appeared to quickly adjust to the presence of researchers and were not observed to notably augment their play to avoid or engage the researchers. The acceptance of this data collection approach resulted in a wealth of rich data on schoolyard utilization, physical activity, and social behaviors of children and community members on the schoolyard. We would recommend use of observational methodologies for any evaluator that is interested in capturing these variables in a naturalistic setting. A critical and formative experience during this project involved learning how to adapt to changes in school administrator, staff, and community leadership over time. When one of the study schools lost a beloved principal in the middle of the pre- to post- evaluation, our research team realized that we needed to collect information from multiple reporters (e.g., school key-informant interviews) in order to ensure that our pre-post data would be reliable over the years, even if personnel changed. On the other hand, we observed the impact that a single individual may have on a community, and the need to capture changes for individuals, schools, and communities that may occur due to variables outside of the schoolyard transformation itself. The goal of the STG health and wellness evaluation was to design a comprehensive evaluation protocol that could be used to compare the impact of the STG initiative across communities and over time. Further, we aimed to gather multi-disciplinary data to support cross-sector efforts to mitigate inequities in the built environment via green schoolyard transformations. The protocol met the stated needs with a multi-method approach that provided rich qualitative and quantitative data at the individual, school, and community levels, across all five RE-AIM dimensions. Ultimately, the project provided evidence-based support for investing in green schoolyards and has supported the growth of the STG initiative in the Chicago area. We argue green schoolyard initiatives would benefit from utilizing this methodology in the future, in order to build a strong and cohesive evidence base. Adopting a shared protocol and approach to data collection for evaluating initiatives would Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 196 advance understanding of the benefits and outcomes of green space interventions, as well as best practices for implementation. Ultimately, more evidence is needed to determine if reducing inequities in the built environment can reduce health disparities. To replicate this evaluation plan, it is recommended that researchers: • Collaborate with other researchers who offer unique expertise or experience. Capitalize on individual strengths to carry out the multi-method strategy. Enlist seasoned evaluators to train data collectors in order to promote the reliability of data collection and input. • Invest time in establishing rapport with community leaders, local organizations, parent groups, and school staff/administrators. Well- established relationships are critical for community engagement. • Provide ongoing feedback to organizations leading and/or supporting the initiative. The evaluation protocol was designed to provide immediate feedback that should be used to adjust the process as deemed necessary. Limitations The most rigorous study design would have included the collection of these data in both intervention and non-intervention communities (e.g., STG and non-STG communities), allowing for the distinction of the effect between the green schoolyard transformation and no change to the built environment. However, limitations to funding and time needed for human subjects approval prevented a controlled study design. Another design limitation that is worthy of note pertains to the lack of data collected directly from youth. Other than process data collected from students attending the STG community-engagement events, youth-reported data was not collected in this evaluation. This was purposeful, promoting the use of evaluation methods across different communities without encountering the challenges of adhering to human subject ethics. Aside from non-identifiable observational data, data on student-level behaviors were collected from caregiver and/or teacher reports, as well as open-access, secondary sources that provide aggregated, student-level data. Of note, the STG health and wellness evaluation plan did not include a fiscal assessment of the STG initiative; however, the STG interventionists have been collecting this information to accompany dissemination of study findings and recommendations. Additionally, a separate evaluation is now being conducted to assess the environmental impact of the green schoolyard initiative (e.g. stormwater management), because such variables were beyond the scope and expertise of the STG health and wellness evaluation team. These findings will also accompany the dissemination of the STG health and wellness evaluation findings and recommendations. Conclusion The STG health and wellness evaluation team sought to develop an evaluation protocol to assess the public health impact of green schoolyards in Chicago utilizing the RE-AIM framework, and in doing so developed a protocol that can help build an evidence base for the green schoolyard movement across the country. Utilizing a rigorous, theory-driven, multi-method assessment strategy resulted in robust data that can inform public policy, future public green space funding and allocation, other funding sources, and park and program priorities. The overall goal is to Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 197 reduce built environment inequities and positively impact the health and wellbeing of all communities. Acknowledgement This research was supported by Healthy Schools Campaign. Dana E. Gerstein, RD, MPH, worked as a social scientist at the Nutrition Policy Institute at the University of California. Her research interests include community and school-based health and wellness initiatives to support nutrition and physical activity behaviors among underserved populations. Carolyn R. Bates, Ph.D., is a psychology post-doctoral resident at Children’s Mercy Kansas City. Her research explores contextual influences on child and family health and aims to identify modifiable factors that promote resilience. Amy M. Bohnert, Ph.D., is a professor of clinical and developmental psychology at Loyola University Chicago. Drawing on an ecological framework, her work examines individual, familial, and other contextual contributions to psychological and health- related outcomes among youth. A central focus of this work is to delineate how basic developmental processes in normative contexts, such as organized activities, facilitate better outcomes, particularly among at-risk populations. This work has entailed several evaluations of local community-based programs that seek to improve youth health and well-being. References Bates, C. R., Bohnert, A. M., & Gerstein, D. E. (2018). Green schoolyards in low- income urban neighborhoods: Natural spaces for positive youth development outcomes. Frontiers in Psychology, 9(805), 1-10. https://doi.org/10.3389/fpsyg.2018.00805 Bridging the Gap Research Program (2012). Park observation form. Robert Wood Johnson Foundation. Retrieved from http://www.bridgingthegapresearch.org/_asset/vnb0e7/BTGCOMP_Park_201 2.pdf Chawla, L., Keena, K., Pevec, I., & Stanley, E. (2014). Green schoolyards as havens from stress and resources for resilience in childhood and adolescence. Health & Place, 28, 1-13. https://doi.org/10.1016/j.healthplace.2014.03.001 Children & Nature Network (2018). Retrieved from https://www.childrenandnature.org https://doi.org/10.3389/fpsyg.2018.00805 http://www.bridgingthegapresearch.org/_asset/vnb0e7/BTGCOMP_Park_2012.pdf http://www.bridgingthegapresearch.org/_asset/vnb0e7/BTGCOMP_Park_2012.pdf https://doi.org/10.1016/j.healthplace.2014.03.001 http://www.childrenandnature.org/ Evaluating a Green Schoolyard Transformation: A Protocol Utilizing the RE-AIM… 198 Cohen, D.A., Inagami, S., & Finch, B. (2008). The built environment and collective efficacy. Health & Place, 14(2), 198-208. https://doi.org/10.1016/j.healthplace.2007.06.001 Cosco, N. G., Moore, R. C., and Islam, M. Z. (2010). Behavior mapping: A method for linking preschool physical activity and outdoor design. Medicine & Science in Sports & Exercise, 42, 513–519. https://doi.org/10.1249/MSS.0b013e3181 cea27a Cosco, N. G., Moore, R. C., & Smith, W. R. (2014). Childcare outdoor renovation as a built environment health promotion strategy: Evaluating the preventing obesity by design intervention. American Journal of Health Promotion, 28(Suppl. 3), S27–S32. https://doi.org/10.4278/ajhp.130430-QUAN-208 Gordon-Larsen, P., Nelson, M. C., Page, P., & Popkin, B. M. (2006). Inequality in the built environment underlies key health disparities in physical activity and obesity. Pediatrics, 117(2), 417–424. https://doi.org/10.1542/peds.2005- 0058 Healthy Schools Campaign and Openlands (2020). Sharing Chicago’s Space to Grow Model. Retrieved from http://www.spacetogrowchicago.org/ King, D. K., Glasgow, R. E., & Lee-Castillo, B. (2010). Reaiming RE-AIM: Using the model to plan, implement, and evaluate the effects of environmental change approaches to enhancing population health. American Journal of Public Health, 100, 2076-2084. https://doi.org/10.2105/AJPH.2009.190959 Moore, L. V., Diez Roux, A. V., Evenson, K. R., McGinn, A. P., & Brines, S. J. (2008). Availability of recreational resources in minority and low socioeconomic status areas. American Journal of Preventive Medicine, 34(1), 16-22. https://doi.org/10.1016/j.amepre.2007.09.021 Raney, M. A., Hendry, C. F., & Yee, S. A. (2019). Physical activity and social behaviors of urban children in green playgrounds. American Journal of Preventive Medicine, 56(4), 522–529. https://doi.org/10.1016/j.amepre.2018.11.004 Stevenson, K. T., Moore, R., Cosco, N., Floyd, M. F., Sullivan, W., Brink, L., ... & Zapalatosch, J. (2020). A national research agenda supporting green schoolyard development and equitable access to nature. Elementa: Science of the Anthropocene, 8(1), 406. https://doi.org/10.1525/elementa.406 Wolfenden, L., Wyse, R., Nichols, M., Allender, S., Millar, L., & McElduff, P. A. (2014). Systematic review and meta-analysis of whole community interventions prevention excessive population weigh gain. Preventive Medicine, 62, 193-200. https://doi.org/10.1016/j.ypmed.2014.01.031 https://doi.org/10.1016/j.healthplace.2007.06.001 https://doi.org/10.1249/MSS.0b013e3181%20cea27a https://doi.org/10.4278/ajhp.130430-QUAN-208 https://doi.org/10.1542/peds.2005-0058 https://doi.org/10.1542/peds.2005-0058 http://www.spacetogrowchicago.org/ https://doi.org/10.2105/AJPH.2009.190959 https://doi.org/10.1016/j.amepre.2007.09.021 https://doi.org/10.1016/j.amepre.2018.11.004 https://doi.org/10.1525/elementa.406 https://doi.org/10.1016/j.ypmed.2014.01.031