DOES PARTICIPATION IN ENRICHMENT ACTIVITIES AFFECT OVERALL PERFORMANCE OF ELEMENTARY AND MIDDLE SCHOOL STUDENTS?: AN EVALUATION OF THE RELATIONSHIP BETWEEN TCNJ BONNER CENTER AND HEDGEPETH-WILLIAMS K-8 SCHOOL Holly Malerba Diane C. Bates Department of Sociology The College of New Jersey January 2014 SUMMARY The purpose of this evaluation is to determine if Hedgepeth- participation versus non-participation in enrichment activities (notably the Academic Sports Academy afterschool program, or ASA) academic outcomes, and if the number of times a student participates in enrichment activities leads to a better outcomes. Based on records of participation in enrichment activities collected by the Bonner Center for Community Engagement (BCCE) and on grade reports, this study finds statistically significant higher science grades for students who participated in enrichment activities when compared to those who did not, and that more participation produced higher science grades in a linear fashion. There is some evidence to suggest that grades in language arts were also improved, but the data is much less conclusive. We were unable to find any significant differences for participants and non-participants in terms of overall grades, math grades, social studies grades, or technology grades; we also found no difference in reported absences and tardies. Nonetheless, significantly higher science grades denote an unequivocal and positive effect of participation in enrichment activities, which is particularly compelling given the low quality of data available (and thus the exclusion of many potential cases). Better record-keeping of participation in enrichment activities would greatly facilitate future evaluation efforts and is likely to demonstrate a stronger positive impact. METHODS Hedgepeth-Williams students participated in a wide range of activities sponsored and organized by the BCCE; these were recorded by compiling ASA attendance records, ASA registration rosters, other rosters, and student project artifacts (e.g., poems, essays, and artwork) from all known activities provided by the BCCE in paper and electronic archives. Student participation was then paired with the only complete Hedgepeth-Williams roster that was available to the researchers, from 2010-2011. Student names were then replaced by numbers and matched to grade reports from 2011-12. Because of the date of the grade reports, no data from Hedgepeth-Williams students or alumni who were currently in 8 th or higher grades were included in the final analysis, regardless of their previous participation in ASA or other enrichment activities. Subject grades represent arithmetic mean of four quarters of grades on a 100-point scale. The 100-point scale reflects either the original numeric scores assigned by Hedgepeth-Williams teachers, or a conversion of letter grades with the following scale: A+ = 97, A = 94, A- = 90, B+ = 87, B = 84, B- = 80, C+ = 77, C = 74, C- = 70, D+ = 67, D = 64, D- = 60, F, F- = 55. Letter grades of I, N, NI, O, S, and U were treated as missing data. One measure of academic outcomes involves absences and tardies. The number of absences and tardies for each student was recorded as part of the grade report. These were unchanged in the analysis. Other dependent variables examined include the average overall grades (TOTAVEN), language arts grades (LITAVEN), math grades (MATAVEN), science grades (SCIAVEN), social studies grades (SOCAVEN), and technology grades (TECHAVEN) of the students at Hedgepeth-Williams. The dependent variables are all represented as numerical variables ranging from 0 to 100. The average overall grades (TOTAVEN) is an average of the five original variables, average language arts grades (LITAVEN), average math grades (MATAVEN), average science grades (SCIAVEN), average social studies grades (SOCAVEN), and average technology grades (TECHAVEN). The means and standard deviations of the dependent variables are shown in Appendix 1. The distribution of the average overall grades (TOTAVEN) are shown in Appendix 2. The distribution of the average science grades (SCIAVEN) are shown in Appendix 3. The first independent variable is whether or not a student definitely participated in enrichment activities (TOTDBIN). The variable was coded so that students who had participated in enrichment activities were coded as 1, and students who had not participated in enrichment activities were coded as 0. The frequency distribution of definite participation versus non-participation in enrichment activities is shown in Appendix 4. The second independent variable is whether or not a student is suspected of participating in enrichment activities (TOTMBIN). Students who were matched based on incomplete names were included in this measure as participating. The variable was coded so that students who may have participated in ASA or TCNJ activities were coded as 1, and students who had not participated in ASA or TCNJ activities were coded as 0. The frequency distribution of suspected participation versus non-participation in enrichment activities is shown in Appendix 5. The third independent variable is the number of times the students definitely participated in enrichment activities (TOTDEF). This variable is a numerical variable ranging from 0 to 8. Students who participated in 6 or more activities were recoded into one category because there were very few students in this category. The frequency of participation in enrichment activities is shown in Appendix 6. The fourth independent variable is the number of times it is suspected the students participated in enrichment activities (TOTMAYBE). Students who were matched based on incomplete names were included in this measure as participating. This variable is also a numerical variable ranging from 0 to 8. Students who participated in 6 or more activities were recoded into one category because there were very few students in this category. The frequency off participation in enrichment activities is shown in Appendix 7. FINDINGS Students that participated in enrichment activities had higher average science grades than students that did not. There is a statistically significant relationship between average science grades (SCIAVEN), and whether students participated in enrichment activities or did not participate in enrichment activities (TOTDBIN, F=10.395, p=.001, and TOTMBIN, F=6.492, p=.011). The average science grades based on definite participation versus non-participation are shown in Appendix 8. The average science grades based on suspected participation versus non-participation are shown in Appendix 9. Whether students participated in enrichment activities or did not participate in enrichment activities and their average overall grades (TOTAVEN) was not statistically significant, regardless of whether TOTDBIN (F=.357, p=.551) or TOTMBIN (F=.626, p=.430) was used for whether or not the students participated in enrichment activities. There was not a statistically significant relationship between average language arts grades (LITAVEN, F=.525, p=.470), average math grades (MATAVEN, F=1.278, p=.259), average social studies grades (SOCAVEN, F=.341, p=.560), average technology grades (TECHAVEN, F=.497, p=.482), and whether students definitely participated or did not participate in enrichment activities (TOTDBIN). Furthermore, there was not a statistically significant relationship between average language arts grades (LITAVEN, F=1.117, p=.292), average math grades (MATAVEN, F=.195, p=.275), average social studies grades (SOCAVEN, F=.159, p=.690), average technology grades (TECHAVEN, F=.654, p=.420), and whether students are suspected of participating or not participating in enrichment activities (TOTMBIN). The more students participated in enrichment activities, the higher average science grades they had. There was a statistically significant relationship between average science grades (SCIAVEN) and the number of times students participated in enrichment activities (TOTDEF, F=3.572, p=.002, and TOTMAYBE, F=3.557, p=.002). The average science grades based on frequency of definite participation are shown in Appendix 10. The average science grades based on frequency of suspected participation are shown in Appendix 11. There is not a clear positive linear relationship between average language arts grades (LITAVEN) and the number of times we suspect students participated (TOTMAYBE), but it seems there is some improvement amongst students who participated in enrichment activities. There was a statistically significant relationship between average language arts grades (LITAVEN) and the number of times students participated in enrichment activities, but only when students we suspect participated were included (TOTMAYBE, F=3.020, p=.007). The average language arts grades based on frequency of suspected participation are shown in Appendix 12. The number of times Trenton elementary and middle school students participated in enrichment activities and their average overall grades (TOTAVEN) was not statistically significant, regardless of whether only the students we know for certain participated in the activities were included (TOTDEF, F=1.052, p=.395) or whether all the students we suspect participated in the enrichment activities a number of times based on an incomplete name were included also (TOTMAYBE, F=1.399, p=.221). There was not a statistically significant relationship between average language arts grades (LITAVEN, F=1.547, p=.163) average math grades (MATAVEN, F=1.293, p=.261), average social studies grades (SOCAVEN, F=1.652, p=.133), and average technology grades (TECHAVEN, F=1.134, p=.347), and the number of times students definitely participated in enrichment activities (TOTDEF). Furthermore, there was not a statistically significant relationship between average math grades (MATAVEN, F=.518, p=.794), average social studies grades (SOCAVEN, F=1.342, p=.239), and average technology grades (TECHAVEN, F=1.422, p=.212), and the number of times students are suspected of participating in enrichment activities (TOTMAYBE). Participation and suspected participation was also compared to the number of absences and tardies recorded on grade records. There were no significant relationships between participation or suspected participation and number of absences or tardies. DISCUSSION Trenton elementary and middle school students who participated in enrichment activities had statistically higher average science grades than students who did not, and greater participation in enrichment activities lead to statistically higher average science grades. However, it is difficult to conclude if there is a clear relationship between participation in the enrichment activities and average language arts grades of the students at the Trenton elementary and middle school because there was a statistically significant relationship only when we included students we suspect participated in enrichment activities, but are not certain. The findings suggest that there is no statistical difference in average overall grades between students who participated in enrichment activities versus students who did not participate in enrichment activities. Additionally, there is no statistical difference in average overall grades among students who participated in more enrichment activities. Further research should control for variables such as grade level, and classroom teacher to see if there are different results on average grades Research such as interviewing teachers that work with the students in order to be able to gain insight on other impacts of the enrichment activities such as increased participation levels and changes in behavior of students would be valuable. Another proposal for future research involves comparing HSPA scores of 11 th grade students who had participated in enrichment activities and students who had not participated in enrichment activities in elementary and middle school; this was not possible in this study because of missing data. Further research should also consider a longitudinal study, to see if participation in enrichment activities in elementary and middle school has an impact on students in high school. All future evaluation would be improved by better record-keeping for enrichment activities and by securing complete student rosters for each grade from the elementary school for each year. Storing such data in a single spreadsheet would greatly facilitate evaluation, although this would still need to be merged manually with grade reports and/or HSPA scores. APPENDIX 1: MEAN AND STANDARD DEVIATION OF DEPENDENT VARIABLES DEPENDENT VARIABLES MEAN STANDARD DEVIATION LANGUAGE ARTS GRADES 76.540 11.105 MATH GRADES 79.041 10.207 SCIENCE GRADES 80.616 9.765 SOCIAL STUDIES GRADES 77.577 11.629 TECHNOLOGY GRADES 76.320 17.719 AVERAGE OVERALL GRADES 78.292 10.081 APPENDIX 2: DISTRIBUTION AVERAGE OVERALL GRADES APPENDIX 3: DISTRIBUTION OF AVERAGE SCIENCE GRADES APPENDIX 4: FREQUENCY DISTRIBUTION OF DEFINITE PARTICIPATION VERSUS NON-PARTICIPATION IN ENRICHMENT ACTIVITIES VALUE FREQUENCY VALID PERCENT PARTICIPATED 134 49.8% DID NOT PARTICIPATE 135 50.2% TOTAL 269 100.0% APPENDIX 5: FREQUENCY DISTRIBUTION OF SUSPECTED PARTICIPATION VERSUS NON-PARTICIPATION IN ENRICHMENT ACTIVITIES VALUE FREQUENCY VALID PERCENT PARTICIPATED 152 56.5% DID NOT PARTICIPATE 117 43.5% TOTAL 269 100.0% APPENDIX 6: DEFINITE STUDENTS FREQUENCY OF PARTICIPATION IN ENRICHMENT ACTIVITIES APPENDIX 7: DEFINITE STUDENTS AND SUSPECTED STUDENTS FREQUENCY OF PARTICIPATION IN ENRICHMENT ACTIVITIES APPENDIX 8: AVERAGE SCIENCE GRADES BASED ON DEFINITE PARTICIPATION VERSUS NON-PARTICIPATION PARTICIPATION NUMBER OF STUDENTS MEAN 0 128 78.664 1 133 82.494 TOTAL 261 80.616 APPENDIX 9: AVERAGE SCIENCE GRADES BASED ON SUSPECTED PARTICIPATION VERSUS NON-PARTICIPATION PARTICIPATION NUMBER OF STUDENTS MEAN 0 111 78.844 1 150 81.927 TOTAL 261 80.616 APPENDIX 10: AVERAGE SCIENCE GRADES BASED ON FREQUENCY OF DEFINITE PARTICIPATION NUMBER OF TIMES STUDENT PARTICIPATED NUMBER OF STUDENTS MEAN 0 128 78.664 1 52 80.140 2 31 82.984 3 18 82.134 4 15 83.000 5 8 87.938 6 or more 9 89.435 TOTAL 261 80.616 APPENDIX 11: AVERAGE SCIENCE GRADES BASED ON FREQUENCY OF SUSPECTED PARTICIPATION NUMBER OF TIMES STUDENT PARTICIPATED NUMBER OF STUDENTS MEAN 0 110 78.844 1 55 79.817 2 36 81.178 3 20 84.988 4 15 78.883 5 13 85.942 6 or more 11 88.765 TOTAL 262 80.616 APPENDIX 12: AVERAGE LANGUAGE ARTS GRADES BASED ON FREQUENCY OF SUSPECTED PARTICIPATION NUMBER OF TIMES STUDENT PARTICIPATED NUMBER OF STUDENTS MEAN 0 110 75.691 1 54 77.859 2 36 78.144 3 20 76.704 4 15 66.950 5 13 81.904 6 or more 13 79.780 TOTAL 259 76.540 Undergraduate Journal of Service Learning and Community-Based Research, Volume 3, Fall 2014 Reflections on Community-Based Research Holly Malerba The College of New Jersey In Fall 2013, I took a Sociology class offered at The College of New Jersey (TCNJ) called Community-Based Research and Evaluation. The course featured a collaboration with the Bonner Center for Civic and Community Engagement (BCCE) at TCNJ to research and evaluate a variety of educational enrichment programs offered at a nearby elementary school. The programs were evaluated based on the academic outcomes for the elementary school students involved as well as in terms of providing learning opportunities for participating college students. This research project focused specifically on the academic outcomes for the elementary school students based on their participation in enrichment activities. The educational enrichment programs are offered to the students through a partnership between the elementary school, the BCCE, and a local foundation that runs the after-school program, ASA. It is essential that these educational enrichment programs have an impact on the students. According to the State of New Jersey School Performance Report (2012), the elementary school is one of the lowest performing elementary schools in the state of New Jersey and significantly lags in comparison to its peers in categories of academic achievement, college and career readiness, and student growth performance. Literature shows the immense value of school-community partnerships in improving student academic achievement. According to Bryan, “schools alone lack the necessary resources to address the large number of obstacles to learning that many minority and poor students in urban schools confront on a daily basis” (2005, 220). She articulates the importance of school- community partnerships by explaining that they are essential to developing “educational resilience” in students, or “the ability of children to succeed academically despite risk factors that make it difficult for them to succeed” (220). School-community partnerships can increase protective factors that foster resiliency in students, and extracurricular enrichment partnership programs have been successful in fostering resiliency and improving academic achievement (Bryan 2005). Sheldon (2003) found that school-family-community partnerships in low-income, urban schools contributed to improved scores among students on state tests. Epstein (1996) discusses how the job of educating and socializing children is a shared responsibility of schools, families, and communities. Therefore, family-school-community partnerships are necessary to best educate and socialize children. She theorizes that for families, communities, and schools to be most effective, they need to have some shared goals and missions, and work collaboratively. Bouillion and Gomez (2001) discuss how a disconnect between schools and communities can lead students to disengage in their education. They highlight the value of school-community partnerships in providing students the opportunity to see how their education in the classroom can provide them opportunities in the future. Additionally, they found school-community partnerships to have mutual benefits by bringing together a diverse range of people and knowledge. Although valuable, these partnerships are not without many challenges. Bryan argues that the social, cultural, and political factors faced by students in urban schools can often present “seemingly insurmountable barriers” to partnerships between schools, families, and the community (2005, 219). Epstein (1996) refers to some of the challenges in developing school- Malerba 2 Undergraduate Journal of Service Learning and Community-Based Research, Volume 3, Fall 2014 community partnerships, such as collaborative planning, and “sharing of time, space, staff responsibilities and budgets” (231). Sheldon (2003) discusses how teachers and families can be obstacles to school-family-community partnerships. According to him, higher quality teachers help schools better face the challenges of partnerships. Families can be obstacles to school- family-community partnerships when parents are unable to speak English, come to the school, or help their children at home. Bryan and Henry (2012) claim that one of the biggest challenges of partnerships is maintaining and sustaining the partnerships. Some of their suggestions to sustain partnerships include good planning and celebration of accomplishments. Evaluation of a partnership’s programs can indicate these accomplishments and can help stakeholders to plan better based on what aspects of a partnership are working well and what aspects can use improvement. When comparing the benefits and challenges of school-community partnerships, the positive impacts on students clearly outweigh the negative. In the elementary school, the existing educational enrichment programs have the potential to greatly impact student achievement. However, without an evaluation of the existing programs, determining whether they are making the impact on the students that they were intended to is difficult. Furthermore, an evaluation of the programs helps to identify more successful and less successful aspects of the enrichment programs to inform the BCCE what areas could use improvement (Bryan and Henry 2012). The educational enrichment programs offered to the students have existed for almost ten years. In this time, there had been no evaluation of the programs. In the semester-long course, our class identified existing data sources to assess the participation of students in enrichment activities and their academic outcomes. We obtained records of participation in enrichment activities from the BCCE and grade reports provided by the elementary school’s administration. Some of the challenges we encountered were incomplete record keeping about students’ participation and difficulty obtaining grade reports from the school’s administration. The incomplete records seemed to be due to a lack of focus on evaluation and a lack of resources. The BCCE and the after-school program both struggle to balance their many responsibilities due to a lack of staff, time, and funding. Due to the high demands that these organizations face, evaluation was not a high priority. The school’s administration, particularly, did not seem to prioritize the evaluation. We had to adjust the time frame of our research due to the delay in the receipt of the grade reports from the school’s administration. A community-based research approach seemed to be the most adequate method of evaluation because the community partner had asked us to carry out the evaluation. In order to best understand what the BCCE wanted out of the evaluation, including the organization in the research process made the most sense. Using a community-based research approach enabled us to have a deeper understanding of the history and mission of the educational enrichment programs and their relationship with the elementary school. The BCCE staff members were able to share their goals for the program and the challenges they had faced. Prior to this experience, I had taken a course on quantitative research methods. This project with the community broadened my understanding of the material I learned in my research methods course. I learned how to apply traditional research methods in less traditional settings. The project taught me the value of research in settings where the findings can be directly applied to make changes. In comparison to a traditional research approach, where the researcher decides the research question, our class collaborated with the BCCE to find out what issues were important to it. Based on its program goals, we decided what outcomes would be most important Malerba 3 Undergraduate Journal of Service Learning and Community-Based Research, Volume 3, Fall 2014 and relevant to evaluate (Hacker 2013). Another difference between traditional and community- based research is that researchers in a community-based research project have to have more flexible timelines, research designs, and research questions. Additionally, community-based researchers have to compromise some of their own goals to align with the goals of the community partners. Some of the benefits I found working with community partners were that we had greater access to data sources that the community partner could share with us, including the records of participation in enrichment activities. Additionally, we were able to gain a deeper understanding of the elementary school and the enrichment programs by talking with representatives from the BCCE. We were able to share our findings directly with the BCCE and the after-school program, ASA, which provides many of the enrichment activities, so that they were able to make use of the findings. We also were able to provide suggestions to the BCCE about improving record keeping to make future evaluations much easier (Hacker 2013). Some of the disadvantages I found working with community partners were conflicting views on the importance of evaluation between the BCCE, the elementary school, and the goals of our class. Particularly, the school’s administration did not give high priority to the evaluation. Due to this, we were unable to stick to our original time frame for the research because we did not receive the grade reports when we thought we would (Hacker 2013). Another disadvantage was the incompleteness of data we received from the community partners. Many of the attendance records of the afterschool program and of the day programs did not list all students in attendance or listed incomplete names. In some cases, we were able to match an incomplete name with a student. At times we also used projects from the after-school program or day programs to identify students who had participated when there was not an attendance list. Due to the incompleteness of the data, we were only able to include students in the evaluation we were certain had participated. Therefore, there were probably many more students that had participated in the enrichment programs but could not be included. This lack of data can be attributed to lack of staffing, time, resources, and priority. I was able to witness firsthand the challenges faced by the community partners to evaluate their own programs. With limited time and resources, they already struggled to carry out the existing programs. Finding extra time and resources to undergo an evaluation of these programs was even more difficult. I hope through our suggestions to the community partners they are able to understand small and quick changes they can make, such as record-keeping, that will significantly reduce the time and resources needed to carry out future evaluations. I also hope they will learn to give greater priority to evaluation because otherwise they are using time and resources without evidence that their programs are effective. I have learned the importance of flexibility, adaptability, and open communication in carrying out research with community partners. Flexibility was a key component to this research project because we had to be willing to compromise on a research question, methods, and use of the findings that the community partners agreed upon. We had to adapt our time frame for research and research methods when things did not happen as planned. Finally, open communication was essential to the success of the project. Open communication allowed us to inform the community partners of our needs and time frame. It also allowed for our community partners to share their needs for the research project and their insider knowledge pertaining to the project. We communicated with the BCCE by having a meeting with the members of our class and staff of the BCCE. In the meeting, we were able to ask questions of them and they were able to ask questions of us. This was the best form of communication because it involved the most Malerba 4 Undergraduate Journal of Service Learning and Community-Based Research, Volume 3, Fall 2014 participants in the project and it was the most direct. The biggest difficulty we had communicating with the community partners was time delays in the responses of the community partners. Since our class met twice a week, my classmates, professor, and I were very easily able to communicate about our progress on the project. There is no specific emphasis on science in the enrichment programs so it is surprising to find that scores in science showed improvement. It is possible that if we had more complete records, the students’ scores would have showed improvement in language arts and math. For example, there was a significant relationship between average language arts grades and the number of times students participated in enrichment activities, when students we suspect participated were included. I would suggest to the afterschool program that they incorporate science as a focus of their afterschool program. Since the afterschool program is already positively impacting students’ performance in science, they could capitalize on this finding to have even greater impacts on students’ performance in science. If these improvements were reflected in a future evaluation, it would put them in a better position to gain more funding to be able to build on and expand their programs. Since the evaluation, ASA and the BCCE have responded to the need for better record keeping. Although they have already faced many challenges at the start of the school year due to a reorganization of the school district, they plan to implement a system to keep better attendance records in a spreadsheet. Additionally, they were very surprised that the findings showed improvement in science scores. They suggested that this finding is possibly due to the fact that there are very good science teachers at the school, and the afterschool program might have simply reinforced good study habits and behaviors that were stemming from the science teachers. Due to the findings of the evaluation, the afterschool program is rethinking its curriculum to incorporate more science to complement its existing curriculum, which is focused on language arts. I found the experience working with community partners to carry out research to be at times frustrating, but very gratifying. Although we were not able to control the research as much as in other forms of more traditional research, I think the benefits outweighed the disadvantages. We were able to provide findings directly relevant to the community partner and build the community partner’s capacity to carry out further evaluation. As researchers, we were able to develop a greater understanding through our collaboration with the community partner. I am appreciative of my experience in this course for expanding and building upon my abilities as a researcher and I hope the community partners are able to benefit by applying the findings of the research. ~ I would like to thank my professor, Dr. Diane Bates, for her constant support, guidance, and encouragement throughout the course of this project. I also would like to thank the community partners for their collaboration and my classmates for their hard work that made it possible to carry out the research. Malerba 5 Undergraduate Journal of Service Learning and Community-Based Research, Volume 3, Fall 2014 References Bouillion, Lisa M. and Louis M. Gomez. 2001. “Connecting School and Community with Science Learning: Real World Problems and School-Community Partnerships as Contextual Scaffolds.” Journal of Research in Science Teaching 38 (8): 878-98. Bryan, Julia. 2005. “Fostering Educational Resilience and Achievement in Urban Schools Through School-Family-Community Partnerships.” American School Counselor Association 8 (3): 219-27. http://graingered.pbworks.com/f/Resilience-+School +%26+Family+Partnerships.pdf. Bryan, Julia and Lynette Henry. 2012. “A Model for Building School-Family-Community Partnerships: Principles and Process.” Journal of Counseling and Development 90 (4): 408-20. Epstein, Joyce. 1996. “Perspectives and Previews on Research and Policy for School, Family, and Community Partnerships.” In Family-School Links: How Do They Affect Educational Outcomes, edited by Alan Booth and Judith F. Dunn, 209-246. Mahwah, NJ: Lawrence Erlbaum Associates. Hacker, Kara. 2013. Community-Based Participatory Research. Thousand Oaks, CA: SAGE Publications. Sheldon, Steven B. 2003. “Linking School-Family-Community Partnerships in Urban Elementary Schools to Student Achievement on Tests.” The Urban Review 35 (2): 149- 165. State of New Jersey. 2012. NJ School Performance Report. http://www.state.nj.us/education/pr/ 2013/21/ 215210080.pdf. Holly Malerba Diane C Bates Research Holly Malerba Reflection