© 2019 Children, Youth and Environments Children, Youth and Environments 29(2), 2019 “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating Nature School Effectiveness Mahala Volpe Victoria Derr Environmental Studies Program, California State University Monterey Bay Steven Kim Department of Mathematics and Statistics, California State University Monterey Bay Citation: Volpe, M., Derr, V., & Kim, S. (2019). “And it made me feel good inside”: Initial evidence and future methods for evaluating nature school effectiveness. Children, Youth and Environments, 29(2), 5-26. Retrieved from http://www.jstor.org/action/showPublication?journalCode=chilyoutenvi Abstract Nature school settings provide opportunities for experiential learning across many developmental domains. This study focused on an afterschool nature program in Santa Cruz County, California, which serves kindergarten through fifth grade students. The study was designed as a pilot to both learn about the program’s effectiveness and to test the feasibility of methods for sustained program evaluation. The study showed that the program produced positive and statistically significant results for most developmental areas and that teachers considered most effective those lessons that engage multiple developmental domains. Future program assessments should examine the number of developmental domains lessons engage and should partner with a local university or expert for periodic in- depth evaluation. Keywords: forest schools, nature schools, outdoor education, child development, children http://www.jstor.org/action/showPublication?journalCode=chilyoutenvi “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 6 Introduction Forest schools emerged in Germany during the initial formation of kindergartens in the mid-19th century, and emphasized the interaction between children and nature (Sobel, 2014). Forest schools provide an immersive curriculum in which young children would observe nature’s cycles, learn care for animals, imitate their sounds, and walk in and explore nature (Sobel, 2014). “Rain or shine” schools in Denmark emerged more than 40 years ago, and forest schools have spread to Norway, Sweden, Switzerland, the United Kingdom, Australia and New Zealand (Elliot, Ten Eycke, Chan, & Müller, 2014; O’Brien & Murray, 2007; Knight, 2009). More recently, forest kindergartens and nature schools have emerged in the United States and Canada (Elliot et al., 2014; Kahn, Weiss, & Harrington, 2018; Sobel, 2014). In these settings, young children are fully immersed in nature for most or all of the day, and they largely engage in unstructured free play or in responsive activities guided by teachers (O’Brien & Murray, 2007; Waite, Bølling, & Bentsen, 2017). Forest schools, and outdoor nature schools more generally, draw upon philosopher John Dewey’s ideas of experiential learning: children learn well by following their own interests and investigations in local nature through place-based education (Änggård, 2010; Derr, Chawla, & Pevec, 2017; Miettinen, 2000). The benefits of nature to children’s health are increasingly documented, with access to greenspaces and “close-to-home” nature supporting increased attention, reduced stress, increased physical activity, and improved cognitive function (Chawla, 2015; Faber Taylor & Kuo, 2006; Faber Taylor, Kuo, & Sullivan, 2001; McCurdy, Winterbottom, Mehta, & Roberts, 2010; Wells, 2000; Wolch et al., 2011). In addition, children benefit from play in nature for its affordances of imaginative and creative play (Änggård, 2010) and increased social interactions (Chawla, 2015). In her recent review, Chawla (2015) notes that when children are given the appropriate space involving earth, sand, water, and vegetation, there is a dialog that emerges with peers and teachers about the environment through sensory exploration and imaginative play. Forest schools have been asserted to provide a wide range of benefits, including motor coordination and physical development, self-awareness and regulation, social skills, academic readiness and performance, self-confidence, empathy, flexibility in thinking, nature relatedness, and environmentally responsible behavior (Asah, Bengston, Westphal, & Gowan, 2018; Elliot et al., 2014; Fjørtoft, 2004; Knight, 2009; Müller et al., 2017; Sobel, 2014). Some research suggests that when low-income children attend forest kindergartens there is a “ripple effect” in which children and their families who did not previously have exposure to nature begin to seek nature outside school hours (Knight, 2009; O’Brien & Murray, 2007). Perhaps least explored are the benefits of forest schools to children’s environmental attitudes, behaviors, and ways of knowing (Chawla & Derr, 2012; Elliot et al., 2014). Kahn and Weiss (2017) pose the question why, if we know nature is so important in human lives, are we not doing more to bring people and nature closer together? Their belief is that the answer is rooted in “environmental generational amnesia” in which we increasingly tolerate degraded worlds because with each generation, children experience degraded environments as normal and therefore lack the experience, knowledge, or motivation to bring people and nature together “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 7 in more robust and sustained ways. Kahn and Weiss (2017, p. 10) assert that the primary way to counteract this is through deep engagements with what they call “big nature,” and the associated pleasures and benefits that such interactions yield. They use the term “big nature” to refer to wild places that are long distances from human artifice, places that can be feared, and loved, but they also use the term to refer to what “big” might mean in an urban center—more wild than a child usually encounters. Forest schools are one way to provide such exposure, with potentially significant and critical benefits not only for understanding that nature matters for health, but in cultivating children who have the ability to benefit from such spaces in an increasingly urban world: If on a national and global level we lose the ability to embody hundreds if not thousands of these seemingly simple interaction patterns—walking on ground, climbing a hill, holding onto a branch to keep one’s balance for a moment, swimming in ocean waves, digging in soil, identifying wild huckleberries to eat on a hot summer’s day, gazing up at the summer’s night sky—it will continue to set into motion this insidious cycle where less interaction with nature leads to less valuing of nature, which leads to less nature being protected and less nature to interact with, which leads to even less interaction with nature: downward and downward we go (Kahn & Weiss, 2017, p. 20). It is within this context that we explored how a nature school influences children’s development. We use the term “nature school” because it includes a broader range of physical environments in which children can experience nature than the term “forest schools” implies. The first author, Mahala Volpe, set out initially to explore the benefits of a specific nature program with which she worked in Santa Cruz County, California. In theory, accessing nature in Santa Cruz County should not be difficult because nature is abundant within the region; however, families often have difficulty finding the time to get out into nature. Many parents make the effort to go outdoors with their children on the weekend, but it can be much harder for them to do so during the school week. Ramblin’ Adventure Club (Ramblin’) is a place-based outdoor afterschool adventure program that picks students up from their schools and then shuttles them to a different natural location every day of the school week. Ramblin’ serves kindergarten through fifth graders in families from Santa Cruz County. The primary objective of the program is for students to gain a sense of appreciation for and awareness of the local ecology of the county by connecting children with the outdoors and teaching science through imaginative play and exploration. Like many nature schools, this program lacked systematic methods for evaluation of its effectiveness. After reviewing the literature on both the benefits of nature schools and systematic evaluations, we established a mixed-methods approach to evaluation. While seeking to identify specific ways children were benefiting from nature interaction, we also recognized that children function in a holistic way, and that nature schools support integrated interactions that allow development across multiple domains. Methods to assess the program included participant observation, child and parent questionnaires, child interviews, and teacher diaries. This research “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 8 was designed as a pilot to both learn about the program’s effectiveness, and to test the feasibility and effectiveness of methods for sustained program evaluation. Nature School Evaluation Research to evaluate the effects of forest schools began in the United Kingdom (O’Brien & Murray, 2007), including Scotland (Borradaile, 2006) and England (Knight, 2009). Consistently, these studies sought to evaluate the effects of forest kindergartens on low-income children who otherwise had little access to nature. The research focused on eight outcomes: confidence, social skills, language and communication, concentration, physical skills, new perspectives, and ripple effects to families (Knight, 2009). These outcomes were evaluated through child- and parent questionnaires, storyboarding with school staff, and direct observations of children in the outdoor setting. In Knight’s (2009) research, parents reported increases in their children’s confidence, social and physical skills, and environmental knowledge. She also found ripple effects, with families seeking out more nature experiences outside of school hours. O’Brien and Murray (2007) tracked changes within three case study areas over an eight-month period and found similar positive impacts in the same domains. Änggård (2010) used an ethnographic approach to study children’s imaginative play in an outdoor classroom over a 10-month period, using participant observation, video observations, informal conversations with children, interviews with parents and staff members, and photographs. In the context of her research setting—a Swedish school where children were outside all the time—she found that nature was used in three primary ways: • as a classroom where children learn about nature in different ways; • as a home—a peaceful place in which to eat, sleep, socialize and play; and • as an enchanted world—a fairyland populated by fairy figures and animals with human traits (Änggård, 2010). Änggård (2010) described how this school provided opportunities for children to learn through active exploration of the natural environment and that these experiences alternated with “a more traditional pedagogical approach in which the leader mediates facts” (p. 12). Learning was thus an interplay between children and teachers, with emphasis given to promoting sensory exploration. Änggård also described how the school used nature as an outdoor fairyland. In some cases, children ascribed human traits to animals, such as snails, that they encountered in the forest, as well as when teachers drew on cultural stories to dramatize or explain aspects of nature. To Änggård, “the stories, songs and imaginary characters help the children create a relationship to nature. They make the forest more exciting and interesting to the children and provide them with symbols they can use in their play” (p. 20). Elliot and colleagues (2014) used a “rich documentation” or “pedagogical narration” approach to evaluating a British Columbian nature kindergarten’s impact on children’s ecological awareness. While the researchers were interested in the effects of the nature kindergarten curriculum as a pedagogical approach to achieving “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 9 environmental education goals, they also recognized the importance of nature exposure as a means to develop emotional connections to nature, termed “nature relatedness” (Nisbet, Zelenski, & Murphy, 2009). Their evaluation methods included observation, interviews, photography by children and researchers, drawings, and children’s narrations of their experiences in nature. They emphasized the importance of the role of children themselves in voicing their perspectives as a means to document their experiences of the environment. This research also evaluated the effects of the nature kindergarten on a range of developmental domains, including social, physical, and environmental. The environmental assessment was designed to evaluate nature relatedness and environmentally responsible behaviors through a game-like assessment that was compared with a control group. Nature relatedness was defined as a mix of affective, cognitive, and experiential aspects of a child’s affinity for nature. The qualitative assessment carried out by Elliot and colleagues (2014) identified themes as “a community of safety,” “a community of learners,” “empathy,” and “deep engagement.” In the empathy theme, children demonstrated relationships with life forms in the forest and created “kinship relationships” and “rules for sharing and caretaking” (p. 111). Deep engagement involved intense play in nature but also practicing compassion for nature and processing death of wildlife, such as a decomposing owl at the side of the trail. Children in the nature kindergarten showed significantly higher nature relatedness scores by the end of the school year whereas the control group’s scores actually declined. No effects were detected for environmentally responsible behavior over the course of the year. Kahn, Weiss, and Harrington (2018, p. 1) characterized child-nature interactions in a forest preschool through “interaction patterns: characterizations of essential ways of interacting with nature.” They described these interaction patterns in an effort to understand how experiences with nature in the forest preschool can counteract environmental generational amnesia. Drawing from Christopher Alexander and colleagues’ (1977) seminal book A Pattern Language to describe patterns in architecture and the built environment, Kahn and colleagues identified patterns of interaction between children and nature. They have collectively identified more than 150 such patterns. These patterns elaborate the three primary elements Änggård (2010) described as well as the ideas of empathy and deep engagement described by Elliot and colleagues (2014). This theoretical approach to describing interaction patterns also gives merit to the qualitative evaluation approaches that identify the diverse ways children interact with nature in forest schools. In a sense, Kahn and colleagues (2018) suggest that evidence abounds for the value of these interactions, and that deep and pervasive child-nature interactions can serve to reawaken people to a vital and essential natural world. Program Background Ramblin’ Adventure Club (Ramblin’) began in 2014 and has served approximately 1,000 students since its inception. All Ramblin’ activities occur outside and last from about one-and-a-half to four or more hours, depending on the grade and program type. Generally, the structured curriculum lasts 20-30 minutes, and the remaining time provides opportunities for unstructured free play in nature. Thus, Ramblin’ “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 10 provides structure for learning while allowing freedom for exploration, play, and adaptive teaching and learning to the context of the day. Locations are rotated quarterly and vary each day of the week, ranging from coastal scrub to redwood forest ecotypes. For example, the kindergarten group attended Wilder Ranch State Park each Monday and Arana Gulch each Tuesday within a single quarter. Students thus visit new locations each weekday but also repeat these visits for several weeks, developing a degree of familiarity with each location. In a single day, first to fifth grade students spend two hours outdoors with the program (or four hours during the one day a week that students in the county are released early from school), equating to 12 hours of outdoor exploration per week if the student attends five days per week. Kindergarten students spend four hours per day with the program, or 20 hours per week if they attend five days per week. Most of the kindergarten students attend four to five days per week with the program whereas the first to fifth grade students attend two to four days per week. The program mixes structured program activities with unstructured free time. Activities include art, cooking, and place-based nature and science exploration. Specific structured activities vary for each grade-level group based on age and abilities. For example, if the activity was to journal about nature, the kindergartens would be expected to draw, the first graders would draw but would also write a basic sentence, and the second to fifth graders would draw and write several sentences. There are three groups comprised of two teachers per group, or six teachers each day. During unstructured time, students are given the freedom to do nearly whatever their imaginations can think of, as long as they follow the three Ramblin’ Adventure Club rules: be safe, be respectful, and have fun. They are taught to play in nature in a respectful way because they understand (or are learning) the impact they have on the environment. For example, if in a moment of destructive fun, they destroy the fallen log in which they love to hunt bugs and jump from, they no longer have that log to play with, and they learn that the other creatures also lose a home or food source. While many terms have emerged to describe these types of programs including forest kindergartens, forest schools, and nature preschools (Sobel, 2014), we refer to this program as a nature school whose primary purpose is taking children outdoors for immersive experiences for the entirety of the afterschool day. This is in keeping with Waite and colleagues (2016) review of outdoor learning, with the Ramblin’ Adventure Club sharing pedagogical alignment with “forest schools.” However, while some of the environments children visit with Ramblin’ are redwood forests of the Santa Cruz Mountains, they also explore many coastal areas with diverse habitats (Figures 1 and 2), including oak woodlands, dunes, and coastal scrub. Thus, the term “nature school” is more inclusive of the environments and intent of the Ramblin’ program. “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 11 Figures 1 and 2. Diverse habitats that are part of the Ramblin’ nature school program setting Image: Mahala Volpe During the 2018-19 school year, there were 74 students enrolled in the Ramblin’ program. Ramblin’ does not collect demographic data from its students. However, according to the U.S. Census in 2018, 57 percent of families in Santa Cruz County were white, not Hispanic or Latino; 34 percent were Hispanic or Latino; 5 percent were Asian; 4 percent were two or more races; 2 percent were Black or African American, and 2 percent American Indian or Alaskan Native (U.S. Census, 2018). According to the same census, 12.6 percent of families lived in poverty within the county; however, census data does not account for the income disparities that some families face from the region’s high cost of living combined with low wage jobs, or those families who are undocumented immigrants, primarily working in the county’s agriculture sector. During the 2018-2019 school year, Ramblin’ Adventure Club had 186 in-session days, with tuition costing $37 per day. Tuition for a fully enrolled student would be $8,000 U.S. for the school year. Since the program is currently fee-based, it largely serves families with the resources and means to pay for afterschool care, although Ramblin’ does provide several full and partial scholarships per quarter. The scholarship form is the only time a family is asked their income, so the only household income data comes from these documents. For scholarship applicants, the household incomes range from $12,000 per year to $67,000 per year. In 2018-2019, there were ten students receiving a full scholarship and two receiving a partial scholarship. There was a disproportionate number of males attending the program compared to females; of the 74 students enrolled, 50 were male and 24 were female. “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 12 Methods The goals of this research were to identify the benefits of the Ramblin’ Adventure Club and to pilot evaluation methods that would be feasible for them to carry out over time. We approached our evaluation with two research questions: i) what are the benefits of the nature school for child participants? and ii) which assessment methods are feasible for the program to carry out over time? It was important to develop effective methods that also acknowledge the limited resources of a small nature school. Qualitative evaluations are clearly valuable in assessing how nature schools affect children’s development. However, we also wanted realistic methods that we could test and that could be implemented long term by the staff of the Ramblin’ Adventure Club. Replicability and analysis time were factors in establishing evaluation protocols as well. Therefore, we drew from Sara Knight’s (2009) evaluation of forest schools in the U.K., and developed questionnaires for both child participants and their parents that we felt could potentially be replicated over time. The first author also conducted interviews with 12 of the participating children: six interviews from the kindergarten group and six interviews from the first- to fifth- grade group. Because narrative descriptions such as those provided by Änggård (2010) and Elliot et al. (2014) provide rich details of children’s engagement with nature, we wanted to provide a source of qualitative assessment as well. In addition to the child interviews, qualitative methods included participant observation by the first author as a teacher in the Ramblin’ program (Emerson, Fretz, & Shaw, 2011), and daily teacher journal entries. This pilot research was conducted between August and December, 2018. We issued pre-term questionnaires within the first two weeks of the program in August, and mid-term questionnaires in early November. Questions were adapted from Knight’s (2009) evaluation and consisted of questions in eight areas including children’s confidence, communication, social skills, concentration, physical abilities, knowledge and understanding, new perspective taking, and ripple effects. Knight (2009) developed these eight question areas based on a self-appraisal with forest school providers in the U.K., with one question per area. Her instrument was not tested for construct validity. The first author asked eight questions of the group of children using a blind vote process in which the children closed their eyes and raised their hands for a count in response to the questions. Our adapted questions were as follows: 1. Confidence: How many of you feel confident, like you are able to do the things you try, in your school classroom? 2. Social Skills: Raise your hand if you think you are a helpful group member. 3. Communication: How many of you feel comfortable talking with your classmates when there is a problem or challenge or when you don’t agree? 4. Concentration: Raise your hand if you can focus on things at school when you are asked to. 5. Physical skills: Raise your hand if you like running/climbing/jumping/swimming/balancing. 6. Knowledge and understanding: Everyone take a look around you. Now, “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 13 raise your hand if you can name three of the trees around you right now? Raise your hand if you know the name of a newt that has a yellow tummy? If you know the name of the creek/river we visit when we are at (insert name of location)? If you know the name of the tree that gives us acorns? 7. New perspectives: Raise your hand if you know my name? The other teacher? Three Ramblin’ teachers? Five? All? Raise your hand if you know your teacher’s favorite food? Teacher’s pet peeves? Teacher’s favorite thing to do in the forest? 8. Ripple effects: Raise your hand if you like to go outside when you have free time? Such as going into the forest, the beach, the creek, etc. By matching the group of children (by grades and days of the week), the proportion of children who raised their hands in answer to the questions was compared between the pre-term (August) and the post-term (November). A total of 73 children participated in the pre-term questionnaires, and 83 in the mid-term. However, the total sample size was reduced to 72 children due to missing data. The proportions were calculated at class-level (i.e., individual students were not identified), and the sample size in the analysis was n = 8 classes (three kindergarten classes, one first-grade class, and four second- to fifth-grade classes) for the comparison. To test for the change in the proportion of children who raised their hands between the pre-term (August) and the mid-term (November), we used the paired t-test. By denoting the average change in the proportion by µD, the hypothesis testing was formulated with the null hypothesis H0: µD = 0 and the alternative hypothesis H1: µD > 0 (i.e., one-sided right-tailed test) for each of the eight areas. The sample size of n = 8 for each paired t-test is quite small, so the significance level was fixed at α = 0.1 (instead of α = 0.05) as a trade-off between Type I error rate and Type II error rate. In addition, we estimated µD using a 90% confidence interval (CI) for each of the eight areas. At the end of the mid-term questionnaire, the first author also conducted two sets of group interviews to supplement the questionnaires. Participants were randomly selected, and group interviews were conducted on the program site with six children in each group, for a total of 12. Interviews asked children the same five questions, which were: 1. Do you know where we are today? 2. What do you like to do here? 3. Is there anything about this spot that you do not like? 4. What do you like about Ramblin’ Adventure Club? 5. Is there anything you do not like about Ramblin’? Responses to these questions are included in the results when they help to illustrate findings from the questionnaire itself. The parent questionnaire we administered was also adapted from Knight’s (2009) “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 14 evaluation and included 26 scaled questions and two open-ended questions. Scaled questions asked parents to indicate if their child’s abilities were increasing, decreasing, or about the same as at the start of the program. Questions asked about children’s abilities across social, physical, cognitive, and environmental domains. These domains were not tested for construct validity in Knight’s (2009) research. The first open-ended question asked parents to identify the top three benefits their child has received from the list of 23 items and to explain why they identified those three. A second open-ended question asked parents to provide any additional benefits that were not listed. Parents were also asked to list the number of years their child had attended the program. Parent questionnaires were issued in November. A total of 23 parents completed the questionnaire. All questionnaire data were recorded in an Excel spreadsheet, and the raw data were cleaned appropriately to perform statistical analysis. For the statistical analysis of the scaled questions, “+1” was assigned to a positive response (i.e., increased ability), and “0” was assigned to a non-positive response (i.e., decreased ability or the same). The responses of each parent were scored between zero and one by dividing the number of positive responses by the total number of scaled questions in each area (i.e., the proportion of positive responses). To test for the association between the parent’s score and the years of the child attending the program (which is treated as a continuous variable in regression analysis), a generalized linear regression model was used (Kutner, Nachtsheim, Neter, & Li, 2004). Since the age of child (which is also treated as a continuous variable) was a potential confounding variable, the age was adjusted in the model. For the proportion of positive responses, which is bounded between zero and one, the logistic regression model was used with the linear predictor β0 + β1 X1 + β2 X2, where X1 denotes years of attending the program and X2 denotes the age of child. The hypothesis testing was formulated with the null hypothesis H0: β1 = 0 and the alternative hypothesis H1: β1 > 0 (i.e., one-sided right-tail test). The significance level was fixed at α = 0.05. We also calculated 95% CI for β1. All statistical analyses were performed in statistical software R version 3.5.0 (R Core Team, 2018). Open-ended questions from the parent questionnaire were entered into NVivo, a qualitative data analysis program, and the researchers coded and counted the responses for frequency. We also used NVivo to generate a word frequency graphic (word cloud) of most frequent responses. The study also asked teachers to complete a daily activity journal. This amounted to two teacher journals per group per day because there are two teachers for every class. These short entries asked teachers to rank how successful they felt the activity was using a three-point scale (very successful, somewhat successful, or not at all successful). Teachers were also asked to write a short entry reflecting on why they rated each day’s activities the way they did. The entries were treated individually to allow for difference of opinion between the two teachers. Teachers completed a total of 142 entries from the middle of August to the beginning of November. “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 15 We input all teachers’ entries into NVivo. Based on participant observation and our interest in generating feasible evaluation procedures, we decided to code the data according to four developmental domains: social, physical, cognitive, and environmental. Therefore, teacher reflections were coded based on the teacher’s effectiveness score for the lesson as well as for the domains with which children engaged in that lesson. After coding, we sorted the data into the three categories of lesson effectiveness (very effective, somewhat effective, or not very effective) and calculated a mean score for the number of domains coded at each lesson. For example, if a teacher entry described social and physical activities, the lesson would receive a code of “2” for the social and physical domains. If a second teacher entry described activities that primarily involved learning facts about nature, that journal entry would be coded as “1” for engaging a single, cognitive, domain. All teacher entries were coded in this way within lesson effectiveness categories. Results Children’s Questionnaires and Interviews When we compared the children’s mid-term to the pre-term responses for each of the eight areas, the observed mean change was positive in all areas except for ripple effects. In particular, at the significance level α = 0.1, the areas of confidence (M = .17, SD = .15, t(7) = 3.16, p = .008), concentration (M = .07, SD = .13, t(7) = 1.59, p = .078), knowledge and understanding (M = .11, SD = .20, t(7) = 1.61, p = .076), and new perspectives (M = .13, SD = .11, t(7) = 3.37, p < .006) were statistically significant based on the sample of n = 8 classes. Table 1 provides the statistics from the paired t-test for each of the eight areas. In some areas, such as physical ability and ripple effects, the proportion of children who raised their hands was already high in the pre-term (.94 and .86, respectively), so there could be a ceiling effect. Figure 3 graphically shows the comparison between the children’s pre-term and the mid-term responses. Table 1. Mean change in the proportion of children who raised their hands in each of the eight areas between pre-term and mid-term) Area M SD t p 90% CI Confidence .17 .15 3.16 .008 [.07, .27] Social Skills .06 .34 .48 .322 [-.17, .28] Communication .06 .22 .82 .219 [-.08, .21] Concentration .07 .13 1.59 .078 [-.01, .16] Physical Abilities .03 .10 .94 .188 [-.03, .10] Knowledge and Understanding .11 .20 1.61 .076 [-.02, .24] New Perspectives .13 .11 3.37 .006 [.06, .20] Ripple Effects -.06 .24 -.71 .751 [-.22, .10] “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 16 Figure 3. Comparison of the proportion of children who raised their hands in each of the eight areas between pre-term and mid-term responses The statistical analysis demonstrates generally positive changes, and the statistical results are supported by interviews with the children. For example, an 8-year old boy who had been attending the program for three years stated that his favorite part of Ramblin’ is that they get to go to “cool new spots with friends” and that they “get to play.” Similarly, a 10-year old boy that has been attending for five years said that his favorite part of Ramblin’ is “it’s just fun going to new places every day and hanging out with people.” These interview comments show that the students were engaging in social situations and learning new perspectives through prolonged exposure to different environments. An 8-year-old girl that has been attending the program since kindergarten said that her favorite part of going to Wilder Ranch State Park is that there are “frogs, a lot of cool plants, and an ‘octopus tree’” (Figure 4). She said there are also “aloe forts” that are fun to play in (Figure 5). This shows her development of knowledge because she is gaining specific knowledge about plants and animals that live within a specific habitat. It also shows her development of physical skills gained by climbing in and navigating the tangled aloe forts and octopus tree. “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 17 Figure 4. (left) An “octopus tree” that children like to climb Figure 5. (right) “Aloe forts” that children found fun to play in Image source: Mahala Volpe Parent Questionnaire According to the parents’ responses (n = 23), the mean number of years that children had been attending the program was M = 1.29 years with SD = 1.05 years. When the years were rounded to integers, seven parents answered 0 years; five answered 1 year; nine answered 2 years; one answered 3 years; and one answered 4 years. For all four domains (cognitive, social, environmental, and physical), the proportion of positive responses tended to increase with respect to years of attending the program (Figure 6). In Figure 6, the years of attendance is treated as a continuous variable, and we fixed the children’s age at the sample mean (M = 6.48, SD = 1.01). Statistically significant results were observed in the social domain (b1 = .71, SE = .27, z = 2.62, p = .004) and physical domain (b1 = .34, SE = .21, z = 1.67, p = .048) after adjusting the children’s age (Table 2). The statistical significances were relatively weak in the cognitive domain (b1 = .20, SE = .15, z = 1.40, p = .081) and the environmental domain (b1 = .43, SE = .29, z = 1.47, p = .071). Table 2. Estimated regression parameter, standard error (SE), test statistic (z), p-value, and 95% CI from the logistic regression model for each of the four domains Domain b1 SE z p 95% CI Cognitive .20 .15 1.40 .081 [-.08, .49] Social .71 .27 2.62 .004 [.18, 1.24] Environmental .43 .29 1.47 .071 [-.14, 1.00] Physical .34 .21 1.67 .048 [-.06, .75] “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 18 Figure 6. The estimated logistic regression models demonstrate the positive association between the proportion of positive responses from the parents and of the length of time children have attended the Ramblin’ Adventure Club for each of the four domains (cognitive, social, environmental, and physical) The importance of growth in the social domain was supported by the child interviews and parent responses to open-ended questions. Parent responses included: Improved respecting of boundaries of others, and I attribute this to the wonderful direct communication skills of the teachers. They have increased friendships and social development. They have learned social skills such as storytelling. In the open-ended questions, the two aspects of child development that parents most frequently attributed to the program were an increased level of self- confidence and increased respect for the environment. This can also be shown by a word cloud generated from NVivo of the most frequent words used in answers to the open-ended question on the questionnaire (Figure 7). “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 19 Figure 7. Key word frequency for open-ended responses to parent questionnaire Teacher Daily Activity Journals Of the 142 entries from teachers’ activity journals, 90 activities (58 percent) were rated by teachers as “very successful,” 42 (30 percent) were “somewhat successful,” and 10 (7 percent) were “not very successful” (Table 3). Table 3. Teacher activity journal entries, showing teacher ratings of the activities Very Successful Somewhat Successful Not Very Successful Total Entries 90 42 10 Percentage of Data 58% 30% 7% When we coded the data in NVivo, we marked each entry as meeting one or more of the four domains of study (social, physical, cognitive, or environmental). The activities that were rated “very successful” predominately included the social domain, with 82 of the 90 total entries. The next most common entry was physical with 69 entries, then cognitive with 64 entries, and environmental with 58 entries. The activities that were rated “somewhat successful” predominately included environmental, with 27 of the 42 entries. The next most common was physical with 23 entries, then social with 22 entries, and cognitive with 15 entries. There was no dominant activity type in the “not very successful” ratings. After coding each entry and sorting them into activity ratings (very successful, somewhat successful, or not very successful), mean scores were calculated for the number of domains associated with each success category (Table 4). A mean score “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 20 of four, for example, would mean that all activities included development of all four domains. The higher the mean score, the more developmental domains were included in each success category (Table 4). Table 4. The number of domains coded within each activity “success” category Very Successful Somewhat Successful Not Very Successful Total Domain Score 286 82 19 Mean Score 3 2 1 During coding, we observed the trend that the “very successful” activities also most frequently engaged more developmental domains. In other words, very successful activities often were developing the children’s social, physical, cognitive, and environmental aspects during the outing. Calculation of mean score shows this trend, with “not very successful” activities having a mean score of engaging only one developmental domain (Table 4). As an example for how “very successful” activities engage multiple domains, one entry described an “art in nature” kindergarten activity: Today we talked about art in nature and reviewed what we have done on previous days. Then we talked about food being an art form and the different foods in nature we can eat. Then we foraged for bay tree nuts to make roasted chocolate treats. This activity was rated by the teacher as “very successful.” Breaking it into component domains, the activity engaged all four developmental domains under study, often in such a way that at least two domains are integrated at any given time: Social and Cognitive: “Today we talked about art in nature and reviewed what we have done on previous days.” Here, the entry shows that students were having a group discussion, which utilizes the social domain, while reviewing past information which requires pulling information from memory, which is the cognitive domain. Social, Cognitive, and Environmental: “Then we talked about food being an art form and the different foods in nature we can eat.” The students were discussing new information as a group (social domain) while encoding new information (cognitive domain). They were also using the environmental domain because they were talking about edible plants that the students could actually see (or smell or feel) while they were learning about them. Physical and Environmental: “Then we foraged for bay tree nuts to make roasted chocolate treats.” The students were physically looking for a plant in their immediate environment, using their bodies in diverse ways to look, peer, and gather, and using environmental knowledge to understand the tree’s shape, smell, form, and its means of reproduction. “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 21 These types of examples abound for the “very successful” activities. We assert that it is the cross-integration of domains within an activity that makes nature schools effective in achieving many of the benefits identified. Toward the end of this pilot study, students in the first author’s activity session wrote a collective poem, in which each student contributed a stanza to the whole. This poem demonstrates the ways that children experience nature in the program, and illustrates their own individualized ways of processing what is valuable about nature experiences. It also demonstrates how children themselves experience “big nature” in this program, and the varied ways that this affects their social, cognitive, physical, and environmental development: Around the rocky cliffs Up the pointy trees I finally get to the top of a mountain. Around the pointy rocks There is a tiny river That leads to the ocean. The ocean water is green, My favorite color The cool animals swim. The sun is warming me up, Playing in the maze, More than one way Makes me happy Playing with friends in nature, Really makes me happy. Because there is always cool things to find Like that hollow log That you can jump on. Flowers are sweet And we saw animals These animals were Great Horned Owls And rats And we went to a really fun beach And played in the maze They made me feel happy And made me feel good inside. Among the trees, Lots of flowers bloom And the leaves swaying in the wind Across the Bay And then out And when people walk by, “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 22 They never hear the rustles, And everyone knows the moon is made of cheese. I saw two great horned owls On our way to the beach We went to play Now it is time to go home And say goodbye to the beach. I saw a beautiful sunset, Hiking back to the cars. This is from Ramblin’ Adventure Club And for all. Discussion This study sought to assess the effectiveness of an afterschool nature program and to test the feasibility of various assessment methods. Our evaluation demonstrated the potential effectiveness of a nature school approach that is holistic by intentionally addressing multiple developmental domains while also allowing for flexibility, responsiveness, and learning “in the moment.” This is supported by previous research that identifies social, physical, and cognitive benefits from nature schools or nature play (Chawla & Derr, 2012; Elliot et al., 2014; Fjørtoft, 2004; Knight, 2009; Müller et al., 2017; O’Brien & Murray, 2007). It also demonstrated a means of assessing children’s learning within the environmental domain, an area less developed within existing evaluation studies (Chawla & Derr, 2012; Elliot et al., 2014). Pre- and mid-term questionnaires showed positive and statistically significant changes in seven of the eight development areas. Responses regarding the presence of a ripple effect due to students’ participation in the program decreased slightly. Physical activity and ripple effect responses already were high in the pre- term assessment, and thus this result could reflect a ceiling effect in which no additional growth can be detected. It is also possible that there is a greater possibility for ripple effects in the summer months when children responded to the pre-term survey, than during the busier school year reflected in the mid-term assessment. All assessment methods pointed to social development as one of the most significant benefits of the nature school. The parent questionnaire found that positive responses to children’s growth in the cognitive, social, environmental, and physical domains tended to increase with the number of years the child had attended the program. Thus, continued participation in the program over multiple years may be significant in influencing the benefits children develop. This is consistent with other studies that have found that extended and repeated exposure to nature education increases program impacts (Chawla & Derr, 2012). Analysis of teacher journals showed that the most effective lessons engaged multiple developmental domains. Effective lessons should intentionally engage three or more developmental domains “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 23 over the duration of the lesson. The program can collect annual child- and parent- questionnaires and engage a local university to assist with analysis. While observations and teacher journals yield rich description, they are also labor intensive to analyze. Therefore, these methods could be employed periodically with university or consulting assistance. Each of these findings is discussed in turn below. The generally positive program benefits found across methods in part could be due to prolonged exposure to the consistent locations for program outings; children develop relationships with the environment through visits to local places that become familiar to them. Each student is also with the same cohort of students in the program for one to five days per week after school, which promotes a connection to the other students. This connection to others and the environment provides a space for children to be confident in their actions, as they continually receive support from their friends and teachers. For example, one student at the beginning of August was a little timid and had never climbed a tree. By October she was running and jumping off trees, rocks, and benches with her friends. That “very successful” program days were those with high engagement of multiple domains for learning supports the development of “nature relatedness” because it can draw upon children’s affective, cognitive, and experiential interactions with nature. Our research demonstrates the importance to children of experiencing Kahn and Weiss’ (2017) “big nature.” Children in this program not only developed knowledge of specific plants and animals, but also an affinity for them, similar to the ways that Änggård (2010) described for children’s fantasies in nature, with creative names given to special places that children enjoyed visiting, like the “octopus tree” or the “aloe fort.” The collective poem also shows how “big nature” works, with many salient examples, including that it “made me feel good inside.” This evaluation contributes to existing research to support the benefits of nature schools for children: when left to explore on their own, and with adults to guide learning in a child-centered way, nature as a classroom provides rich environments for development. However, to assess this on a regular basis is onerous for programs where the primary focus for staff is taking children outside. We therefore considered the feasibility of each evaluation method. Self-reports, particularly by young children, can be limited in effectiveness as a research tool, but in this study, the questionnaires did indicate ways that children and parents perceived the program to be increasing children’s development across various domains. That children and parent reports resulted in a range of both quantitative and qualitative results may suggest that both children and parents are in fact capable of discerning and reporting on aspects of the program that seem more salient or beneficial. This approach to evaluation is also supportive of the push to provide children more “voice” in program evaluation and research (Elliot et al., 2014). Statistical analysis suggests that the length of time children spend in a program (over multiple years) is positively associated with some domains, if not all. Though some results were not statistically significant, the direction of association was fairly consistent with our expectation. For program evaluation, these “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 24 questionnaires were the easiest to conduct, and when collected over time could indicate trends in program effectiveness. Like other research on nature schools, this study found some of the program’s benefits are best told through narratives of interactions. Qualitative research is time-consuming and can be hard to replicate without a dedicated researcher to analyze and present the data. One approach could be to collect questionnaires and teacher activity diaries each year and to engage a qualitative researcher at intervals for more in-depth documentation, drawing from some of the approaches reviewed earlier in this article. This would also support Müller and Liben’s (2017) call for more integrative research that could support ways that nature supports children’s development across disciplines. While it was time-consuming to enter all the teacher diaries into NVivo for coding, this coding was useful in thinking about how the program was meeting four developmental domains as a measure of success. One way to simplify long-term evaluation could be to focus on the identification and evaluation of specific program activities and the extent to which they integrate multiple developmental domains. The evaluation would thus consist of ensuring that program activities consistently are meeting the “best practices” in nature school education because research into nature experiences increasingly identifies benefits, and the integration of experience, across multiple domains. This was illustrated by the plant identification and chocolate treat-making example, which integrated two to three domains at any point in time. For small schools and non-profits with budget constraints, assessment of the program in terms of its integration of the four domains into each day’s activities could be a useful tool for evaluation. Teacher entries and notes could also be digitized into a survey form in a way that might facilitate easier evaluation, particularly if the focus is given to the documentation of domains, with teacher reflections that provide short pieces of evidence that show how the activity and associated domains led to student learning. We present these results at an exciting time of exploration of the values and benefits of nature schools. Nature schools are burgeoning across the globe (Kahn et al., 2018), with the paired goals of providing opportunities for children to realize the benefits of nature while also providing an integrated educational approach to fostering nature relatedness and environmental care in children. This article offers a grounded approach to evaluating how programs can demonstrate these benefits, helping to grow our collective understanding of the rich affordances nature schools can offer. Mahala Volpe studied child development at San Jose State University and environmental education at California State University Monterey Bay, where she completed her B.A. in Environmental Studies. A childhood spent immersed in the Santa Cruz Mountains fostered her love of the natural world and her passion to share the wonders of nature with others. She has been a teacher at Ramblin’ Adventure Club since 2016. Victoria Derr is Assistant Professor of Environmental Studies at California State University Monterey Bay, where her teaching and research focus on the “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 25 intersections between sustainable communities, place-based environmental education, and social justice. For more than 20 years, Victoria has engaged children, youth, and communities in participatory research in both rural and urban settings within tribal, Spanish land grant, recent immigrant, and international settings. She holds a masters and Ph.D. from Yale University. Steven Kim is Assistant Professor of Statistics at California State University Monterey Bay. He studies and teaches statistical methods and theories. His primary research areas are statistical methods applied to dose-response modeling and experimental designs in small-sample studies. He has collaborated with researchers in various disciplines such as environmental studies, kinesiology, plant sciences, and health sciences. References Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I., & Angel, S. (1977). A pattern language. New York: Oxford University Press. Änggård, E. (2010). Making use of ‘nature’ in an outdoor preschool: Classroom, home, and fairyland. Children, Youth and Environments, 20(1), 4-24. Asah, S. T., Bengston, D. N., Westphal, L. M., & Gowan, C. H. (2018). Mechanisms of children’s exposure to nature: Predicting adulthood environmental citizenship and commitment to nature-based activities. Environment and Behavior, 50(7), 807-836. DOI: 10.1177/0013916517718021. Borradaile, L. (2006). Forest School Scotland: An evaluation. Edinburgh: Forestry. Retrieved from www.forestry.gov.uk Chawla, L. (2015). Benefits of nature contact for children. Journal of Planning Literature, 30(4), 433-452. Chawla, L. & Derr, V. (2012). The development of conservation behaviors in childhood and youth. In S. Clayton (Ed.), Handbook on environmental and conservation psychology (pp. 527-555). Oxford, England: Oxford University Press. Derr, V., Chawla, L., & Pevec, I. (2017). Early childhood education. In A. Russ & M. Krasny (Eds.), Urban environmental education review (pp. 155-164). Ithaca, New York: Cornell University Press. Elliot, E., Ten Eycke, K., Chan, S., & Müller, U. (2014). Taking kindergartners outdoors: Documenting their explorations and assessing the impact on their ecological awareness. Children Youth and Environments, 24(2), 102-122. Emerson, R. M., Fretz, R. I., & Shaw, L. L. (2011). Writing ethnographic fieldnotes (2nd ed.). Chicago: The University of Chicago Press. http://www.forestry.gov.uk/ “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 26 Faber Taylor, A., & Kuo, F. E. (2006). Is contact with nature important for healthy development? State of the evidence. In C. Spencer & M. Blades (Eds.), Children and their environments (pp. 124–40). Cambridge, UK: Cambridge University Press. Faber Taylor, A., Kuo, F. E., & Sullivan, W. C. (2001). Coping with ADD: The surprising connection to green play settings. Environment and Behavior, 33(1), 54-77. Fjørtoft, I. (2004). Landscape as playscape: The effects of natural environments on children's play and motor development. Children, Youth and Environments, 14(2), 21-44. Kahn Jr., P. H., & Weiss, T. (2017). The importance of children interacting with big nature. Children, Youth and Environments, 27(2), 7-24. Kahn, P. H., Weiss, T., & Harrington, K. (2018). Child-nature interaction in a forest preschool. In A. Cutter-Mackenzie, K. Malone, & E. Barrat Hacking (Eds.), Research handbook on childhoodnature: Assemblages of childhood and nature research (pp. 1-24). Springer, Cham: Springer International Handbooks of Education. Knight. S. (2009). Forest school and outdoor learning in the early years (2nd ed.). Thousand Oaks, CA: Sage Publications. Kutner, M., Nachtsheim, C., Neter, J., & Li, W. (2004). Applied linear statistical models (5th ed.). New York: McGraw-Hill/Irwin. McCurdy, L. E., Winterbottom, K. E., Mehta, S. S., & Roberts, J. R. (2010). Using nature and outdoor activity to improve children's health. Current Problems in Pediatric and Adolescent Health Care, 40(5): 102-117. DOI: 10.1016/j.cppeds.2010.02.003 Miettinen, R. (2000) The concept of experiential learning and John Dewey's theory of reflective thought and action. International Journal of Lifelong Education, 19(1), 54-72. DOI: 10.1080/026013700293458 Müller, U., & Liben, L. S. (2017). Introduction: Natural spaces and development. Children, Youth and Environments, 27(2), 1-6. Müller, U., Temple, V. A., Smith, B., Kerns, K., Ten Eycke, K., Crane, J., & Sheehan, J. (2017). Effects of nature kindergarten attendance on children's functioning. Children, Youth and Environments, 27(2), 47-69. Nisbet, E. K., Zelenski, J. M., & Murphy, S. A. (2009). The nature relatedness scale: Linking individuals’ connection with nature to environmental concern and behavior. Environment and Behavior, 41(5), 715-740. http://paperpile.com/b/GE4WLD/ax6G http://paperpile.com/b/GE4WLD/ax6G http://paperpile.com/b/GE4WLD/ax6G http://paperpile.com/b/GE4WLD/ax6G http://paperpile.com/b/GE4WLD/ax6G “And it made me feel good inside”: Initial Evidence and Future Methods for Evaluating… 27 O’Brien, L., & Murray, R. (2007). Forest School and its impacts on young children: Case studies in Britain. Urban Forestry & Urban Greening, 6(4), 249-265. R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. Retrieved from https://www.R-project.org/. Sobel, D. (2014). Learning to walk between the raindrops: The value of nature preschools and forest kindergartens. Children, Youth and Environments, 24(2), 228-238. U.S. Census (2018). Quick facts, Santa Cruz County. Retrieved from https://www.census.gov/quickfacts/santacruzcountycalifornia Waite, S., Bølling, M., & Bentsen, P. (2016). Comparing apples and pears? A conceptual framework for understanding forms of outdoor learning through comparison of English Forest Schools and Danish udeskole. Environmental Education Research, 22(6), 868-892. Wells, N. M. (2000). At home with nature: Effects of “greenness” on children’s cognitive functioning. Environment and Behavior, 32(6), 775-795. Wolch, J., Jerrett, M., Reynolds, K., McConnell, R., Chang, R., Dahmann, N., ... & Berhane, K. (2011). Childhood obesity and proximity to urban parks and recreational resources: A longitudinal cohort study. Health & Place, 17(1), 207-214. http://www.r-project.org/ http://www.r-project.org/ http://www.r-project.org/ https://www.census.gov/quickfacts/santacruzcountycalifornia