Microsoft Word - v21n79 fnl v3.doc Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 3/5/2013 Facebook: /EPAAA Revisions received: 6/21/2013 Twitter: @epaa_aape Accepted: 6/24/2013 education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 21 Number 79 October 7th, 2013 ISSN 1068-2341 STEM Club Participation and STEM Schooling Outcomes Michael A. Gottfried Loyola Marymount University & Darryl N. Williams Tufts University U.S.A Citation: Gottfried, M. A. & Williams, D. (2013). STEM Club Participation and STEM Schooling Outcomes. Education Policy Analysis Archives, 21 (79) Retrieved [date], from http://epaa.asu.edu/ojs/article/view/1361 Abstract: To develop a more robust understanding of the relationship between non-formal, school- based STEM activities and students’ success and persistence in STEM fields, this study evaluates how math club participation influences math GPA and how science club participation influences science GPA. Additionally, this study evaluates how math or science club participation associates with the probability of selecting a STEM major in college. Utilizing data from the National Longitudinal Study of Adolescent Health (Add Health) to examine these relationships, the results suggest that there is a STEM achievement gap in the success and persistence of students who do and do not participate in STEM-related extracurricular clubs. While, for the most part, the results were not differentiated by gender or race/ethnicity per se, they were in fact distinguishable by poverty status and the interaction between race and poverty status. Keywords: math club; science club; STEM Participación en clubes de Ciencias, Tecnologías, ingenierías y matemáticas (STEM) y Rendimiento escolar en STEM Resumen: Para llegar a una comprensión más sólida de la relación entre las actividades extraescolares en el área STEM realizadas las escuelas y el éxito y la persistencia de estudiantes en los epaa aape E ducation Policy A nalysis A rchives V ol. 21 N o. 79 2 campos de STEM, este estudio evalúa la participación en clubes de matemáticas influye en las calificaciones de matemáticas y cómo la participación en clubes de ciencias influye en las calificaciones de ciencias. Además, este estudio evalúa cómo la participación en clubes de matemáticas o la ciencia se asocia con la probabilidad de seleccionar un campo STEM en la universidad. Utilizando los datos del Estudio Longitudinal Nacional de Salud Adolescente (Add Health) para examinar estas relaciones, los resultados sugieren que existe una brecha en el rendimiento y la persistencia STEM entre estudiantes que participan y no participan en actividades extraescolares relacionados con STEM. Mientras que, en su mayor parte, los resultados no fueron diferenciadas por sexo o raza / etnia en sí, en realidad eran distinguibles por la situación de pobreza y de la interacción entre la raza y la condición de pobreza. Palabras clave: club de matemáticas, club de ciencias; STEM Participação nos clubes de ciência, tecnologia, engenharia e matemática (STEM) e desempenho escolar STEM Resumo: Para se ter uma compreensão mais sólida da relação entre as atividades extracurriculares realizadas nas áreas de STEM nas escolas e o sucesso e persistência dos alunos em áreas STEM, este estudo avalia si a participação em clubes de matemática influências as qualificações em matemática e como participação em clubes de ciência influenciar qualificações científicas. Além disso, este estudo avalia si a participação em clubes de matemática ou de ciências está associada com a probabilidade de selecionar um campo STEM na faculdade. Usando dados do Estudo Nacional Longitudinal de Saúde do Adolescente (Add Health) para examinar essas relações, os resultados sugerem que existe uma brecha no desempenho STEM e persistência entre os alunos participantes e não participantes em atividades extracurriculares relacionadas com STEM. Embora, na maioria das vezes, os resultados não foram diferenciados por sexo ou raça /etnia em si eram realmente distinguíveis pelos indicadores de pobreza e a interação entre raça e pobreza. Palavras-chave: clube, clube da matemática, ciência, STEM Introduction The United States relies on scientists and engineers to ensure our national security, to solve our most critical problems, to further our body of knowledge, and to increase our general standard of living (Hira, 2010). Despite the essential role this workforce plays, however, policy makers, educators, and business leaders are deeply concerned about declining quantity and quality of American youth in line to adopt these positions. In terms of quantity, the percentage of students pursuing degrees in engineering, physical science, and math have remained stagnant, and the percentage pursuing computer science has declined (National Science Board, 2010). In terms of quality, American youth are ill-prepared compared to many of the countries we would consider as counterparts. For example, on the Programme for International Student Assessment (PISA) exams, 15 year-olds in the United States rank 16 out of 26 countries in science literacy and 19 out of 26 countries in mathematical literacy (National Science Board, 2010). These statistics do not bode well for the rising generation of the U.S. workforce who will enter into a highly competitive, global economy. Due to these concerns regarding both quantity and quality, there has been increased pressure on the education system to take a more proactive approach in preparing youth for careers in STEM fields. Indeed, decades of research findings and reports disseminated by academia, government agencies, and industry support the alarming conclusion that there is a serious deficit in our education ST E M C lub P articipation and ST E M Schooling O utcom es 3 system (Rask, 2010); hence the need for an intervention particularly at the K-12 level is in order, such that the U.S. economy can sustain global competitiveness in the future (Augustine et al., 2010). In this context, there is an ever-growing research and policy interest geared at understanding how and why American youth develop, cultivate, and sustain an interest in science, technology, engineering, and mathematics (STEM)-related subjects. Moreover, data show that this national problem is not unique to any subgroup of the population: it is not localized to urban or suburban students, or to any particular gender, race, or socioeconomic group. Rather, the lack of high quality students in STEM is widespread among all youth in the U.S. (Augustine et al., 2010). Part of this pervasive problem can certainly be attributed to the lack of adequate formal preparation in K-12 grades (i.e., curriculum and classroom instruction). There are significant challenges that exist across the comprehensive K-12 education pipeline in STEM; a lack of appropriate resources, insufficient teacher training, and ineffective teacher pedagogy are often highlighted as K-12 factors inhibiting student preparation and performance in STEM (National Research Council, 2011). Prior research has called for a revamping of formal STEM education in order to more effectively prepare our youth. Along with formal preparation, non-formal exposure to STEM inside and outside of school (or lack thereof) has shown to be significant in a student’s trajectory into (or out of) a STEM-related career (Williams & Gottfried, 2010). Non-formal learning opportunities occur in planned, but highly adaptable ways in organizations or situations outside of formal or informal education experiences and often include activities such as academic clubs or career-related clubs (Butler & Serrell, 2001). Indeed, positive outcomes have been supported for those students participating in non-formal educational experiences: students’ achievement levels and social development improves based on students’ abilities to foster and cultivate intrinsic motivation and engagement in environments outside of the formal setting (Shernoff & Vandell, 2007). Other benefits of participation in non- formal activities have been reported, such as the development of positive social networks and leadership abilities (Broch, 2002; Lipscomb, 2007; Marsh, 1992). Furthermore, other benefits of participation in non-formal activities have been reported, such as the development of positive social networks and leadership abilities while facilitating involvement with new peer groups and adults, stimulating interest in a particular subject, and providing a richer experience than simply completing assignments in a formalized instructional setting (Broch, 2002; Lipscomb, 2007; Marsh, 1992). While limited, the empirical research in this area often focuses on the effects of non-formal in-school activities such as clubs. For instance, Eccles and Barber (1999) assessed the effects of participating in school-based academic clubs, such as foreign language club. Relying on a sample of approximately 1,300 predominantly White high school students in Michigan, the authors found that adolescents who indicated that they had participated in an academic club at school had higher than expected high school GPAs and were more likely to be enrolled in college at age 21 than were their peers who indicated that they did not participate in extracurricular clubs at school. It should be noted that the study relied on general measures of extracurricular activities and GPA and did not evaluate particular STEM fields in looking at the effect of non-formal participation. That said, however, other research has looked specifically at the effect of club participation on STEM outcomes under the premise that students’ involvement in school-based extracurricular activities develops their self-confidence, ability to persist in competitive situations, and focus among other characteristics – all of which may be related to improved interest and performance in STEM (Csikszentmihalyi, 1996; National Federation of State High School Associations, 2005; Vandell, Pierece, & Dadisman 2005;). In studying the effects of extracurricular club participation on STEM achievement, Lipscomb (2007) found that students at the secondary level who participated in athletics or extracurricular clubs exhibited a 2-percent increase in math and science test scores and 1- E ducation Policy A nalysis A rchives V ol. 21 N o. 79 4 percent increase in math test scores, respectively. The study utilized the National Education Longitudinal Study (NELS) of 1988 and included a base sample of 16,305 students. The nationally-representative findings in Lipscomb’s (2007) study built upon earlier work by Camp (1990), which also found a positive correlation between extracurricular student activity and academic achievement. In fact, Camp’s (1990) conclusions validated even earlier research as well: that participation in in-school student clubs had a positive prediction on academic achievement (Holland & Andre, 1987). It should be noted, however, that these studies utilized general measures of participation in extracurricular school clubs; none were domain specific, such as the influence of math club participation on math achievement. Although there have been several decades of research evaluating a broad relationship between extracurricular club participation and achievement, it is necessary to take a more domain-specific approach in examining the relationship between participation in math and/or science clubs and STEM outcomes. Doing so will allow for further clarity into the educational pipeline as to how exposure in specific STEM fields relates to success and persistence in those particular subject areas. Prior work has suggested that the positive outcomes related to participation in non-formal STEM experiences, such as in-school clubs, reflects combination of the skills students learn in these activities as well as the higher motivation of those who select into those clubs in the first place. Thus, it is important to separate the components of this combination in order to develop a more robust understanding of the true effects that non-formal, school-based STEM activities may have on building skills and promoting both success and persistence in STEM subjects. This study takes this next step by evaluating the relationship between participation in math or science club and STEM educational attainment in those respective, fields. Given these issues, there are three research questions: 1. Does in-school extracurricular math and science club participation have a positive association with math and science schooling success, respectively? 2. Does gender, race, or socioeconomic status moderate the relationship of of math or science club participation on STEM schooling success? 3. Does in-school extracurricular math and science club participation have a positive association with an increased probably of declaring a STEM major in college, respectively? In addressing these questions, this study makes three meaningful contributions. First, prior studies have examined the relationship between participation in extracurricular school activities and school success (e.g., Camp, 1990; Lipscomb, 2007) with some research examining math and science outcomes. However, no study has evaluated the relationship between a domain-specific extracurricular club on STEM achievement. That is, no study has examined the relationship between math club participation and math achievement or between science club participation and science achievement. This study does both. Additionally, this study evaluates if a long-term relationship persists between math or science club participation by looking not simply at high school STEM success but also at the differences in the probability of graduating with a STEM major in college for those students participating in math or science club. Second, this study asks these research questions while also employing a wider range of contextual variables than those that have been used in prior research, such as student academic attributes, official student transcript and course-taking data, and school characteristics. The analyses in this research study include these additional variables. Third, an additional gap in earlier research is that there is only little examination of the heterogeneity of these relationships by gender, race, or socioeconomic status (SES). However, there is a higher concentration among minority students who demonstrate a deeper lack of interest in STEM. According to Rowe (1977), the value of early and extensive exposure to science influences ST E M C lub P articipation and ST E M Schooling O utcom es 5 decisions that Black students make about science professions. This is, of course, not unique to just Black students and can be applied to many student populations where research shows that minority ethnic groups have similar attitudes and aspirations or lack thereof for STEM careers (Crisp et al., 2009). However, particular attention can be drawn to students of color given the fact that they lag behind their white counterparts in STEM across the board in terms of preparation and exposure to STEM (Simpson, 2001). Prior research suggests disparities in STEM fields by poverty versus non- poverty students, though these academic disparities were not necessarily limited to STEM (Donaldson et al., 2008). Hence, poverty students may have less involvement in STEM, lower achievements in these fields, and a lower rate of participation compared to non-poverty students. And looking at gender, there is evidence to suggest that differences in science-related experiences at the formal level extend into informal learning environments, where girls are less likely to engage in out-of-school activities than boys. Girls potentially miss out on the many kinds of skills developed in these non-formal activities (e.g. science fairs) where they explore, assemble, and tinker in a way that enhances interest and success in STEM (Greenfield, 1996; Jones, 1997; Rennie, 1987). If there truly is an association of math or science club participation with STEM outcomes, one might suspect that those of different demographic backgrounds would receive differential benefits from math or science club participation. Method Data As the purpose of this study was to determine the presence of a relationship between math or science club participation and math or science schooling success and on the probability of choosing a STEM major in college, this study utilized data from the National Longitudinal Study of Adolescent Health (‘Add Health’). Data collection was initiated in 1994 to examine how adolescents’ social contexts influence schooling and health outcomes. The dataset contained thousands of variables on adolescents’ families, schools, neighborhoods, and peers. The first wave, with in-school surveys, in-home surveys, parent surveys, and school administrator surveys, was conducted with 7th to 12th grade students between 1994 and 1995. The second wave, with in-home surveys of the students and school administrator surveys was conducted one-year later in 1996. The third wave, with an in-home survey for the students in the sample, was conducted in 2001-2002 when respondents were 18-26 years old. Additional information regarding the add health dataset is found in Harris et al. (2009). It is crucial to note that the research objective in this study was to ascertain whether a STEM experiences in high school associate with both high school and college outcomes. That is to say, the research questions were framed to have long-term implications, and so by design they required longitudinal data. To truly conduct a longitudinal study, it was important to observe youth over time. With these parameters in mind, Add Health was the most recently available national data to address the questions posed with the contextual variables that were required for the model. There was still much to be gleaned from these data that are of relevance to the current policy and research dialogue on these issues, even with information on students attending high school in the late 1990’s. Analytical Samples This study used Add Health data to create two samples. The first sample was utilized to generate the probability of participating in a math or science club in school, as it will be described below. This sample was based on having been a student in the in-home Wave I survey, and data must be available as to whether or not each student had participated in a math or science club at E ducation Policy A nalysis A rchives V ol. 21 N o. 79 6 school (N = 15,356). Math or science club was determined by a survey question in Wave I, in which a student was asked to indicate school club participation from a list of possibilities. Of the possibilities, “math club” and “science club” were both options. Thirty-one additional clubs were listed, though none could be classified as having a STEM focus (e.g., French club). Note that no additional detail was provided regarding math and science clubs. This is addressed in the limitations section of this article. The second sample was utilized to examine the effect of math or science club participation on math and science GPA or on the probability of choosing a STEM major in college. To be included in this sample, students must have had non-missing data on math and science GPA or on major selection in college in addition to the non-missing observations used to predict math or science club participation in the first sample. This final regression sample, as described in Table 1, included a total of N = 3,223 students. Within this sample, 202 students participated in science or math clubs in school, and the remainder did not. Variables Table 1 presents the means and standard deviation values of the dependent and independent variables used in this analysis, as broken out by whether or not a student participated in math or science club (i.e., STEM club in the table). A test of mean differences was employed to determine if the two student groups – participants versus non-participants – differed in a statistically meaningful way across the sample. Participation in a STEM club was binary. It indicates if a student had participated in a math or science club at school in the academic year in which the Wave I survey was administered. There were two sets of dependent variables in this analysis. The first were math and science high school GPAs. To supplement the educational data in Add Health, the Adolescent Health and Academic Achievement (AHAA) study began in 2001 with the collection of official high school transcripts. Researchers collected transcripts for 12,237 Add Health subjects based on the more than 1,200 schools they last attended. This investigation used students’ overall math and science GPAs as calculated by AHAA researchers based on transcript data. Table 1 suggests that there were statistically significant differences between the math and science GPAs of those students participating versus not participating in math and/or science clubs in school. These differences were tested at the p = 0.05 level. Hence, this difference of means served as a benchmark result of the influences of math or science club participation and merited further exploration with the use of more rigorous analytical techniques. The second dependent variable was having a STEM major in college. The variable employed here was binary: a 1 indicates if a student declared a STEM major in college, and a 0 indicates otherwise. Students could identify more than one major in college in the survey. Thus, if a student graduated with a STEM major in addition to a second major, he or she was also identified as having a STEM major. The determination of those college majors deemed as belonging to STEM was executed for this study based on the academic classification codes assigned to each major in the database. Majors that fell into science, technology, engineering, mathematics, or a hybrid of these four disciplines were considered to be STEM majors. A list of STEM majors included in the dataset is available upon request. For this dependent variable, Table 1 presents mean differences between those who participated in math or science club and those who did not. The results suggested that students in math or science clubs tended to have a greater chance of having a STEM major in college. ST E M C lub P articipation and ST E M Schooling O utcom es 7 Table 1 Descriptive Statistics Full Sample STEM Club Participants Non-Participants Mean Std Dev Mean Std Dev Mean Std Dev Outcomes High school math GPA 2.23 0.92 2.60(a) 0.97 2.21 0.92 High school science GPA 2.33 0.93 2.76(a) 0.94 2.30 0.95 Selection of STEM major in college 0.32 0.47 0.51(a) 0.50 0.30 0.46 Student demographic information Male 0.47 0.50 0.47 0.50 0.47 0.50 Minority 0.37 0.48 0.38 0.49 0.37 0.48 Asian 0.15 0.36 0.17 0.38 0.15 0.36 Middle school (versus high school) identifier 0.26 0.44 0.35 0.48 0.26 0.44 Household income (in thousands) 46.90 44.03 44.32 39.28 47.07 44.34 STEM preparation and motivation Highest math course taken in high school (1 = lowest, 9 = highest) 6.28 1.82 7.14(a) 1.75 6.22 1.81 Highest science course taken in high school (1 = lowest, 6 = highest) 4.48 1.26 4.95(a) 1.22 4.45 1.25 Percentage of failed math semesters in high school 0.11 0.19 0.08 0.17 0.11 0.19 Percentage of failed science semesters in high school 0.08 0.19 0.06 0.17 0.09 0.19 Peabody PVT test score 102.58 14.09 105.49(a) 13.40 102.38 14.11 Scale of desire to attend college (1 = lowest, 5 = highest) 4.55 0.91 4.88(a) 0.48 4.53 0.93 Parents are interested in child's grades (0 = lowest, 1 = highest) 0.92 0.27 0.91 0.29 0.92 0.27 School information Private 0.08 0.27 0.08 0.27 0.08 0.27 Magnet 0.37 0.48 0.40 0.49 0.37 0.48 Urban 0.30 0.46 0.39 0.49 0.29 0.45 Suburban 0.52 0.50 0.47 0.50 0.52 0.50 n 3,222 202 3,020 Note: (a) Statistically different from the non-participant sample at p<0.05 Sixteen independent variables were employed in this analysis. The covariates used as control variables in the regression models to predict math and science GPA or STEM major selection are the same covariates that were first used to estimate a student’s probability of participating in a math and/or science club, as described in the analytic strategy to follow. Of these independent variables, this study employed a standard set of demographic variables that has been used to predict high school GPA when using Add Health data (Gottfried & Polikoff, 2012) or STEM major selection E ducation Policy A nalysis A rchives V ol. 21 N o. 79 8 (Williams & Gottfried, 2010). In general, Table 1 did not show any statistically significant differences between student demographics between participants and non-participants in math and/or science clubs. A second set of variables fell into a category described by Williams & Gottfried (2010) as ‘preparation and motivation” for the pursuit and persistence in STEM. First, to account for preparation in STEM, this study evaluated the highest course completed while in high school in math and science. The scale of this measure was 1, being an introductory course, to 9 being an extremely advanced course for math and 1 to 6 for science. The subject categories of the math course sequence included: 1, Basic/Remedial Math; 2, General/Applied Math; 3, Pre-algebra; 4, Algebra I; 5, Geometry; 6, Algebra II; 7, Advanced Math (Algebra III, Finite Math, Statistics); 8, Pre- calculus (includes Trigonometry); and 9, Calculus. The subject categories for the science course sequence included: 1, Basic/Remedial Science; 2, General/Earth Science; 3, Biology I; 4, Chemistry; 5, Advanced Science (Biology II, Chemistry II); and 6, Physics. These categories reflected a hierarchy of courses ranging from less to more advanced. Note that students did not have to pass through each category of the sequence. For instance, students might have taken either Advanced Math or Pre calculus, but not both. Additionally, while most students’ course-taking patterns reflected a linear movement through the sequence, a minority of students may have had different patterns (i.e., Chemistry may not always precede Physics). Students who participated in math or science clubs tend to have taken courses higher in the sequence in these subjects, as presented in Table 1. A math and science failure index was also included in this analysis as the percentage of semesters failed in math or science over the course of middle or high school. Additional preparation in STEM is measured by individual scores on the Add Health Picture Vocabulary Test (PVT) (Williams & Gottfried, 2010). The test was an 87-item, multiple-choice, computerized, abridged version of the Peabody Picture Vocabulary Test administered to students at the start of the Wave I survey. Raw scores were standardized by age. Students in math or science clubs had statistically higher PVT scores than do students who did not participate. Motivation for STEM persistence (and general schoolwork) in this study, as consistent with the framework put forth by Williams & Gottfried (2010), was measured in two ways. First is a scale indicating a student’s desire to attend college was defined by his or her Wave I response to the question of how much he or she wanted to attend college on scale of 1 to 5, with 1 = low and 5 = high. Students who participated in math or science clubs had statistically higher desires to go to college, as presented in Table 1. Second was a student’s interpretation of parental interest in his or her school life: a binary variable indicts if a student felt that a parent was interested in his or her academic grades. Finally, the analysis included a set of school-level predictors. These included indicators for whether the school that a student attends was private, magnet, urban, or suburban. There were no statistically significant differences between the types of schools that participants in math or science clubs and non-participants attend. Analytic Strategy Baseline model. In the baseline empirical model of educational outcomes, student achievement was described using a linear relationship with a particular academic measure as the dependent variable and a vector of independent variables. In the described formulation of the model to follow, math or science GPA served as a dependent variable and STEM-club participation, student characteristics, and school characteristics serve as independent variables. Note that when STEM major selection was utilized as an outcome, the model became a probability model (i.e., logistic regression) rather than a linear model presented below. Regardless, the independent variables ST E M C lub P articipation and ST E M Schooling O utcom es 9 remained the same whether GPA or STEM major selection was chosen as an outcome in the specification. This baseline model is displayed in equation (1), Yigk = �0 + �1STEMit + �2Iigk + �2Fi + �3Sk + (1) where Y was either math or science GPA for student i in grade g in school k; STEM indicated if a student participated in a math or science club at school k (whether math or science club was selected depends on whether math GPA or science GPA was the outcome, respectively); I was a vector of individual-level characteristics; F were student-specific family characteristics; and S included school characteristics. Finally, was a random error capturing individual variations over time as well as a school-specific random component that is common to all members of the same school. The regression presented as equation (1) contained a large span of individual, family, and school characteristics, and it allowed for a baseline analysis as to whether there was a systematic relationship between math or science club participation at school and math or science GPA, respectively. However, estimates of �1, the coefficient on math or science club participation, may still have been biased under ordinary least squares methods. The reason was that there were current and past unobservable factors that could influence both math or science club participation (i.e., the independent variable) as well as math or science GPA (i.e., the dependent variable). For instance, student motivation might simultaneously be related to selecting into math club as well as a higher math GPA. As a consequence, the results from equation (1) served strictly as a baseline set of estimates; the results of a more rigorous methodology were compared to this first set of results. Strengthening the baseline model. A baseline model could test for whether there may be correlational evidence that students were acquiring non-formal STEM exposure from math or science club participation that was positively related to STEM schooling success or STEM major selection. However, it may have been possible that students who already had abilities correlated with greater STEM success were participating in math or science clubs, though this was not possible to determine with the secondary dataset utilized in this study. It would have been ideal to randomly assign students into math or science clubs at school. However, quasi-experimental methods provided an approximation in the absence of random assignment, and one widely accepted way in which quasi-experimental designs can emulate random assignment is through propensity score matching (Becker & Ichino, 2002; Schneider et al., 2007). This approach, which was adopted in this study, allowed for quasi-experimental contrasts between students in naturally occurring treatment and control groups, but who displayed similar likelihoods of experiencing the treatment based on their observed characteristics. In this study, propensity score matching was employed to contrast the math and science GPAs of students who did and did not participate in math or science clubs but who nonetheless displayed similar propensities to participate in them. Based on using a restricted comparison sample, the results from employing propensity score matching yielded an average treatment effect, which is considered a more robust finding than the average estimate derived from the entire sample in the baseline model (Heckman, 1996). In general, propensity score matching is a two-stage process. It was conducted twice in this study: once for math club when math GPA was an outcome, and once for science club when science GPA was an outcome. In the first stage, a propensity score, or probability, was calculated for selection into the treatment group. In this case, the treatment was a student’s participation in a math or science club at school. The calculation of a propensity score was based on Rosenbaum and Rubin (1983). The propensity score P(Z) was defined as the conditional probability that a student with a set of Z observable characteristics will be in math (or science) club (C=1), i.e., P(Z) = Pr(C=1�Z). The Z characteristics that are used to model the selection process into math or science club participation E ducation Policy A nalysis A rchives V ol. 21 N o. 79 10 were included based on prior literature examining what types of students typically succeed in STEM (e.g., Williams & Gottfried, 2010). The actual propensity score for each student was estimated using a logistic regression model (i.e., binary outcome model asking if a student did or did not participate in a STEM club) (Rosenbaum & Rubin, 1984): log[P(Z)/(1- P(Z))] = � + �f(Z), where � and � were parameters and f(Z) was the specified function. Again, this model was conducted twice – once for math club and once for science club. The propensities were evaluated separately depending on whether math GPA or science GPA is an outcome. To populate this function, this study employed 16 covariates to model a student’s propensity to participate in a math or science club - these were the same covariates and interaction terms that were used as control variables in the baseline model. Variables were kept in the model if they were statistically significant at the conventional level and/or if theory suggested a strong relationship between a particular variable and math or science club participation. Indeed, the covariates predicting math or science club participation were identical, and all 16 covariates were employed in both models. The second stage of propensity score matching entailed using the estimated propensity scores from the first stage to match students who were and were not in math or science clubs at school in order to derive the average treatment effect. Three different matching mechanisms are employed in this study as a way of conducting a sensitivity analysis (Luellen et al., 2005). The first was matching by blocks, in which treatment and control students are stratified into different groups by having extremely similar propensity scores. Within each block, the average treatment effect was calculated, and then they were averaged across all blocks to produce an overall average treatment effect – i.e., a more refined estimate of the relationship between math or science club participation and STEM outcomes than what equation (1) might have suggested. The second method for determining the average treatment effect was nearest-neighbor matching. Here, treatment students and control students were paired based on the close values of their propensity scores. Nearest-neighbor matching with replacement was employed, because it allowed for control students to be matched to more than one treatment student. This enabled for flexibility in that many control students might have matched to multiple treatment students. Finally, kernel matching was employed. Here, individual students in the treatment group (i.e., those who participated in STEM clubs) were matched to a weighted mean of control cases. The weights were determined by the distance of each propensity score to the propensity score of the treatment case to which each control was being matched. With kernel matching, the analysis was bootstrapped with 1,000 repetitions and a bandwidth of 0.01, as consistent with prior research (Pagan & Ullah, 1999). Overall, the estimated coefficient on math or science club participation represented the average treatment effect of participation on a student’s STEM schooling outcomes. If the estimated value of this coefficient were statistically significant and positive, this indicated that students who participated in math or science clubs tended to have higher schooling success than students who did not participate in math or science clubs. Relying on propensity score matching thus enabled this study to hypothesize on the direction of any bias from having relied strictly on a baseline regression model. Results Baseline Results ST E M C lub P articipation and ST E M Schooling O utcom es 11 Table 2 presents the results of the regressions for the relationship between math club participation on high school math GPA and also for the prediction of science club participation on science GPA. Under the first model of each subject, the prediction of math or science club was conducted without any other independent variable included in the model. Under the second model of each subject, the results are presented from employing a regression using the same covariates that were also used to estimate a student’s propensity to participate in math or science club in the matching algorithm. That is, the second model of each table displays the differences in GPA of students who did and did not participate in math and/or science clubs, after statistically controlling for the covariates used to predict a student’s propensity to participate in a club. Note that in both subjects, students have not yet been matched on their propensity to be in math or science clubs. The key variable in this study is presented in either the first or second row of the table, depending on the subject area of the outcome. The table indicates that participation in math club had a positive relationship with math GPA, and participation in science club had a positive relationship with science GPA. This was true whether participation is examined in the first or second models within each subject area. The inclusion of a full set of covariates, however, did reduce club participation’s positive effect in both subject areas from 0.47 to 0.25 GPA points in math and 0.42 points and 0.25 GPA points in science, though the coefficients remained significant at the p<0.001 level across all models. As such, though there was a reduction in the size of the coefficients from the inclusion of a full span of covariates, it did not alter this study’s fundamental premise that non-formal STEM exposure positively correlated with STEM schooling success. The use of a variant on the standardized beta coefficients is often implemented as the effect size when the independent variable of interest is binary (e.g., Datar, 2006). The effect sizes of participating in math or science club, as defined by the standardized regression coefficient , were 0.50� and 0.27� for models 1 and 2 in math, respectively, and 0.45� and 0.27� for models 1 and 2 in science. This implies that there existed a medium-sized (Cohen, 1988) standardized positive relationship between math or science club participation and respective GPA, though the relationship was tempered by the inclusion of a full set of covariates. E ducation Policy A nalysis A rchives V ol. 21 N o. 79 12 Table 2 Regression Results for Math and Science Cumulative GPA Predicting Math GPA Predicting Science GPA Model 1 Model 2 Model 1 Model 2 Coefficient Std Error Coefficient Std Error Coefficient Std Error Coefficient Std Error Participation in math club 0.469*** 0.088 0.228*** 0.057 0.415*** 0.084 0.250*** 0.053 Participation in science club Student demographic information Male -0.132*** 0.021 -0.142*** 0.021 Minority -0.216*** 0.024 -0.184*** 0.024 Asian 0.116*** 0.030 0.091** 0.030 Middle school (versus high school) identifier 0.021 0.024 0.046+ 0.024 Household income (in thousands) 0.000 0.000 0.000 0.000 STEM preparation and motivation Highest math course taken in high school (1 = lowest, 9 = highest) 0.124*** 0.008 0.135*** 0.008 Highest science course taken in high school (1 = lowest, 6 = highest) 0.050*** 0.011 0.048*** 0.011 Percentage of failed math semesters in high school -2.571*** 0.072 -0.666*** 0.072 Percentage of failed science semesters in high school -0.099 0.071 -2.107*** 0.070 Peabody PVT test score 0.003** 0.001 0.006*** 0.001 Scale of desire to attend college (1 = lowest, 5 = highest) 0.006 0.012 0.021+ 0.012 Parents are interested in child's grades (0 = lowest, 1 = highest) -0.022 0.038 -0.005 0.038 School information Private 0.122** 0.043 0.009+ 0.043 Magnet 0.006 0.023 0.041 0.023 Urban -0.047 0.034 0.041 0.034 Suburban -0.014 0.029 0.023 0.029 n 3,220 3,220 3,220 3,220 R2 0.01 0.59 0.01 0.61 Notes: **p<0.001; **p<0.01; *p<0.05; +p<0.10 ST E M C lub P articipation and ST E M Schooling O utcom es 13 Briefly turning to the set of control variables employed in this study, gender and race had negative relationships with high school math and science GPA, holding all else equal. This result was consistent across both subject areas and is also consistent with much of the literature (Argys, Rees, & Brewer, 1996; Caldas & Bankston III, 1997; Ogbu, 1989; Summers & Wolfe, 1977). The results for preparation and motivation were consistent with prior work in their relationships with STEM outcomes (e.g., Williams & Gottfried, 2010): taking increasingly more advanced courses in the sequence was related to higher GPAs, for instance. Finally, there was a general lack of significance in the relationships between school-level covariates and math and science GPA. Accounting for Unobserved Heterogeneity The results from the baseline model suggested that there may be evidence that students were acquiring non-formal STEM exposure from math or science club participation that was positively related to STEM schooling success. However, it might have been possible that students who already had abilities correlated with greater STEM success (i.e., through preparation, motivation, and exposure) might have participated in math or science clubs at school. Table 3 provides a more robust version of the analyses presented in Table 2: it displays the main results for math club participation on math GPA for both baseline analyses and propensity score matching models. The first two models were the results from the math models in Table 2. New to the table are the three latter models, which employed propensity score matching based on block matching, nearest neighbor matching, and kernel matching, as described in the Method section above. When matched on their propensity to participate in a math club, students in math club had higher math GPAs, as evidenced by both block matching and kernel matching. Even after controlling for a more rigorous regression model with the use of propensity score matching, participation in math club significantly related to math GPA. Table 3 Estimated Effects of Math Club Participation of Cumulative Math GPA OLS Regression (Table 2) Propensity Matching No covariates Full model Block Matching Nearest Neighbor Kernel Matching Math GPA 0.469*** 0.228*** 0.233* 0.136 0.426*** 0.088 0.057 0.104 0.093 0.089 Notes: ***p<0.001; **p<0.01; *p<0.05;+ p<0.10 Standard errors are in italics under the each estimate Table 4 presents similar findings for the relationship between science club participation and science GPA. Analogous to Table 3, the first two models present the results from the science models in Table 2. New to Table 4 are the three later models, which employed propensity score matching based on block matching, nearest neighbor matching, and kernel matching. When matched on their propensities to select into a science club, students in science club had higher science GPAs, as evidenced by significant coefficients in all three propensity score matching methods. In other words, the participation in science club is related to statistically significant differences in science GPA. E ducation Policy A nalysis A rchives V ol. 21 N o. 79 14 Table 4 Estimated Effects of Math Club Participation of Cumulative Science GPA OLS Regression (Table 2) Propensity Matching No covariates Full model Block Matching Nearest Neighbor Kernel Matching Science GPA 0.415*** 0.250*** 0.251* 0.212+ 0.396*** 0.084 0.053 0.088 0.090 0.080 Notes: ***p<0.001; **p<0.01; *p<0.05;+ p<0.10 Standard errors are in italics under the each estimate Results by Student Characteristic Tables 5 and 6 disaggregate the analyses in Tables 3 and 4, respectively, to determine if different demographic groups had different relationships to GPA by participation in math or science club. Each cell represents the coefficient on math club participation when math GPA was utilized as an outcome in Table 5 or science club participation when science GPA was utilized as an outcome in Table 6. Each cell represents the results from fully interacted models based on the demographic characteristic indicated in the leftmost model. Table 5 suggests that for math GPA, the relationship between math club participation and math GPA was approximately the same for males and females– the coefficients in each group tended to be fairly similar to one another across OLS regression and propensity score matching models. The same was true for students across different race/ethnicity categories. That is, examining across each racial/ethnic group shows that the results did not differ greatly whether a student was Black or Hispanic, Asian, or White. A major distinction, however, arose between poverty and non- poverty students (where poverty was previously determined in the dataset by being in a family that did or did not receive welfare assistance). There was an overall lack of statistical significance on the coefficients for poverty students. This was true for OLS regressions and propensity score models. On the other hand, non-poverty students who participated in math club also have higher math GPAs. This result warranted further investigation, and as such, poverty versus non-poverty status was delineated by race. Both Black or Hispanic and White non-poverty student subsamples tended to have higher math GPAs when participating in math club. On the other hand, participation in math club was not related to math GPA for poverty students across the same racial/ethnic groups. Asian students, regardless of poverty status, tended to have higher GPAs when participating in math clubs, however. It was noted that the high values of the effects in the Asian/poverty category may have arisen from the small sample size in this subgroup. Table 6, which presents the evaluation of the relationship between science club participation and science GPA, portrays a slightly different interpretation; this underscores the importance of examining club participation by subject area rather than as an aggregate measure. Generally speaking, the analysis for males and females looked fairly similar to what Table 5 presented for math club. Gender, once again, did not seem to moderate the association of science club participation on science GPA. Also as consistent with the analysis of math club participation in Table 5, there was no difference for White and Black or Hispanic students. However, for science club participation, there was no longer a statistically significant relationship for Asian students. ST E M C lub P articipation and ST E M Schooling O utcom es 15 Table 5 Estimated Effects of Math Club Participation of Cumulative Math GPA OLS Regression Propensity Matching No covariates Full model Block Matching Nearest Neighbor Kernel Matching Male 0.505*** 0.222** 0.336* 0.327* 0.441** 0.129 0.084 0.123 0.132 0.139 Female 0.434*** 0.239** 0.314* 0.328* 0.411** 0.117 0.078 0.118 0.118 0.123 Minority 0.556*** 0.327*** 0.444** 0.421** 0.530** 0.137 0.089 0.156 0.138 0.162 Asian 0.532** 0.314** 0.297 0.478* 0.421* 0.183 0.108 0.194 0.177 0.149 White 0.468*** 0.150+ 0.313** 0.548*** 0.439*** 0.118 0.079 0.100 0.116 0.093 Poverty 0.442 0.314+ 0.211 -0.069 0.375 0.299 0.183 0.416 0.286 0.369 Non-poverty 0.483*** 0.211*** 0.354** 0.404*** 0.442** 0.092 0.060 0.098 0.098 0.112 Minority & poverty 0.290 0.365 0.163 0.322 0.278 0.427 0.260 0.771 0.341 0.400 Minority & non-poverty 0.599*** 0.305** 0.493** 0.434*** 0.571*** 0.144 0.096 0.161 0.148 0.126 Asian & poverty 1.731* 0.462 1.281** 2.072*** 1.458*** 0.693 0.513 0.365 0.258 0.244 Asian & non-poverty 0.446** 0.294** 0.217 0.409* 0.346+ 0.188 0.113 0.206 0.152 0.198 White & poverty 0.283 0.192 0.357 0.532 0.538 0.456 0.273 0.645 0.340 0.509 White & non-poverty 0.418*** 0.130 0.319+ 0.531*** 0.443*** 0.131 0.083 0.110 0.127 0.109 Notes: ***p<0.001; **p<0.01; *p<0.05;+ p<0.10 Standard errors are in italics under the each estimate E ducation Policy A nalysis A rchives V ol. 21 N o. 79 16 Table 6 Estimated Effects of Math Club Participation of Cumulative Science GPA OLS Regression Propensity Matching No covariates Full model Block Matching Nearest Neighbor Kernel Matching Male 0.367** 0.224** 0.193* 0.177 0.347** 0.120 0.074 0.092 0.122 0.111 Female 0.482*** 0.285*** 0.321* 0.252* 0.464*** 0.116 0.076 0.113 0.125 0.112 Minority 0.500*** 0.328*** 0.364* 0.405** 0.488** 0.124 0.081 0.139 0.127 0.143 Asian 0.300 0.116 0.021 0.015 0.264 0.217 0.134 0.192 0.184 0.182 White 0.445*** 0.219** 0.259* 0.305* 0.426*** 0.107 0.069 0.116 0.115 0.104 Poverty 0.357 -0.079 0.275 0.469* 0.363 0.328 0.187 0.237 0.203 0.244 Non-poverty 0.419*** 0.277*** 0.252* 0.254* 0.400** 0.086 0.055 0.093 0.093 0.108 Minority & poverty 0.787 0.258 0.652 0.849*** 0.798* 0.483 0.255 0.411 0.191 0.278 Minority & non-poverty 0.465*** 0.342*** 0.335* 0.326* 0.453** 0.128 0.086 0.137 0.140 0.141 Asian & poverty 1.540 0.438 0.978** 1.971*** 1.521*** 0.902 0.659 0.269 0.490 0.151 Asian & non-poverty 0.229 0.089 -0.048 -0.130 0.192 0.222 0.139 0.191 0.191 0.183 White & poverty -0.039 -0.227 0.033 0.232 0.012 0.447 0.240 0.201 0.240 0.265 White & non-poverty 0.495*** 0.254*** 0.295* 0.333** 0.473*** 0.121 0.072 0.123 0.110 0.104 Notes: ***p<0.001; **p<0.01; *p<0.05;+ p<0.10 Standard errors are in italics under the each estimate Further, as consistent with the analysis of math club by subgroup, non-poverty students tended to have higher science GPAs when in science clubs, whereas the results generally suggested that poverty students did not. As with math results in Table 5, Black or Hispanic and White non- poverty subgroups had higher science GPAs when also in science clubs. However, the results were not significant for non-poverty Asians students, though they were for math club participation. Again, the results for Asian students by poverty status might have been influenced by the small sample sizes in each subcategory. Effects on STEM College Major Selection Table 7 displays the final set of results predicting the effect of math or science club participation in high school on having a STEM major in college. As in previous tables, the table is broken out first by OLS regression models as a way to examine baseline results. However, propensity score matching is subsequently presented. Regardless of OLS or propensity matching models, the coefficients are presented as odds ratios. ST E M C lub P articipation and ST E M Schooling O utcom es 17 Table 7 Estimated Effects of Club Participation on the Odds of Choosing a STEM Major in College OLS Regression Propensity Matching No covariates Full model Block Matching Nearest Neighbor Kernel Matching Math club 2.766** 2.187* 1.232* 1.404*** 1.260** 0.328 0.354 0.082 0.046 0.070 Science club 1.606 1.410 1.096 1.033 1.112 0.334 0.358 0.069 0.042 0.061 Notes: ***p<0.001; **p<0.01; *p<0.05;+ p<0.10 Standard errors are in italics under the each estimate Initially addressing the effects of math club participation, the first two models present the results from the baseline model, with the first model included no other covariates other than math or science club participation and the second model included the full span of covariates. The results indicated that participation in a math club had a positive relationship with having a STEM major in college. For math, the odds in the first model were almost 3-to-1 that a student who was in math club in high school chooses a STEM major in college. The inclusion of a full set of covariates, however, did temper the effect math or club participation’s positive association in both subject areas from approximately 3-to-1 odds to approximately 2-to-1 odds for math club. Nonetheless, the coefficients remained statistically significant. The latter three models in the table are the results from employing three different propensity- matching schemes: block matching, nearest neighbor matching, and kernel matching. When students were matched based on their propensities to select into math club, there remained a significant likelihood that students select a STEM major in college. This is evidenced by the statistical significance in all three matching methods in models predicting the relationship between math club and STEM major selection. The odds of selecting a STEM major decreased in the propensity matching models; this was most likely the case because the propensity matching models were accounting for many factors that may have been influencing baseline estimates of club participation on choosing a STEM major. That being said, however, the results nonetheless suggested a greater probability of having a STEM major in college as a result of math club participation. Table 7 suggests that the results starkly differ based on club subject area; hence, the distinction between the effects of math and science was a crucial one. The results for science club were not significant, neither in baseline OLS models nor in propensity matching. The relationship between club participation and STEM major selection was only prevalent in math. Discussion This study has contributed a unique perspective on issues pertaining to non-formal experiences in STEM by evaluating the association between math and science club participation and STEM schooling success. This relationship was assessed using a nationally-representative sample of adolescent students; the dataset was a unique fit to this study because it contained student attributes, official school transcripts, and school data in addition to containing precise information about in which clubs each student had participated. Additionally, it was longitudinal in nature, thereby allowing this present study to evaluate concurrent STEM school success as well as longitudinal outcomes. E ducation Policy A nalysis A rchives V ol. 21 N o. 79 18 Overall, this study relied on two approaches. The first approach evaluated the relationship between having participated in a math or science club and subsequent high school (official) math or science GPAs. This was done by employing the data in two ways. First, a linear regression model was specified and was used to provide baseline results. Second, this study employed propensity score matching. The analyses in this first approach provided two major findings. First, while the use of propensity score matching did temper the estimates in comparison to the baseline estimates, it was evidenced that math club participation was related to higher cumulative high school math GPA. In other words, controlling for all else, the achievement gap widened in math performance between two identical students in which one participated in an extracurricular in-school math club and one did not. Second, the results indicated that science club participation was related to higher cumulative high school science GPA – and there was evidence presented in this study of a domain-specific effect. These two findings held true even after controlling for covariates previously included in studies on the effects of extracurricular in-school clubs (e.g., prior performance measures and demographic characteristics) and after controlling for new characteristics (e.g., courses taken in STEM and school attributes). A second approach provided further clarity on these two relationships. It did so by examining if the relationship between STEM clubs and STEM outcomes was longitudinal – i.e., how math or science club participation was related to the probability of having a STEM major in college. The baseline results suggested odds ratios that were almost 3-to-1 that students who participated in math or science clubs would select STEM majors in college. After employing propensity score matching, the odds were tempered. Nonetheless, the results still identified positive, statistically significant relationships between math club participation and STEM major selection. The results in this second approach did not find any statistically significant relationships between science club participation and STEM major selection. However, this is consistent with prior research. Mathematics preparation is often used as the litmus test for measuring academic interests in STEM due to the fact that mathematics skills are prerequisites to participation in a large array of scientific and technology-based fields, such as economics, physics, and engineering (Hackett & Betz, 1981). The same may not be true about preparation in science, as the reach of exposure in biology, say, may be more specialized and less generalizable to other fields in contrast to preparation in calculus. In tandem, these two approaches suggested that there was a STEM achievement gap in the success and persistence of students who do and do not participate in STEM-related extracurricular clubs. It arose from the fact that students who participate in math and science clubs tend to have higher math or science GPAs, respectively, during that same time-period of schooling. It also arose from the fact that students who participate in math or science clubs experienced a sustained persistence, even after high school. Therefore, math and science club participants had a distinguishable, positive relationship with both concurrent and future STEM outcomes. Indeed, these conclusions align with previous studies that highlighted factors associated with cultivating students' interest in math and science (Thomas, 1986) as well as with those studies that provided evidence of a relationship between extracurricular club participation and STEM success (Lipscomb, 2007). Additional exploration, however, revealed that these results were not equal among all students. While, for the most part, the results were not differentiated by gender alone or race/ethnicity alone, they were in fact distinguishable by poverty status (i.e., whether or not the student’s family received a form of welfare payment) and the interaction between race and poverty status. This held true in both math and science regression analyses and propensity score models. ST E M C lub P articipation and ST E M Schooling O utcom es 19 Indeed, the major distinction that arises between poverty and non-poverty students requires the need for further explanation. The results indicated that there is an overall lack of statistical significance on the coefficients for poverty students, and that this was true for both baseline regressions and propensity score models. This conclusion supports research findings that suggest that students from low SES backgrounds are less likely to have high academic achievement regardless of ethnicity (Sirin, 2005). On the other hand, students from more affluent backgrounds have many more support systems in place to ensure that they will succeed in school (Simpkins et al., 2005). For example, in terms of mathematics achievement, research suggests that the home environment often predicts students’ attitudes and success in this domain (Sheldon & Epstein, 2005). This is defined by how involved the parents are in terms of engaging their students in activities and conversation as well as showing concern for academic engagement in and out of school. High poverty students may experience less parental involvement compared to non-poverty students (Sampson & Laub, 1994). This may shed light on the mechanisms driving the results in Tables 5 and 6. Although the effects of poverty as it relates to negative academic performance can be seen across racial/ethnic groups, the impact is significantly greater for students in STEM minority groups (i.e., Black and Hispanic). Minorities are more likely to live in low-income households or in single parent families; their parents are likely to have less education; and they often attend under-funded schools (National Commission on Children, 1991). Noguera (2003) surmises that the range of choices available to an individual are profoundly constrained and shaped by external forces. Therefore, looking at the findings from this study, one might conclude that students within poverty may have competing external forces (i.e. resource-constrained families and schools) that prevent them from experiencing the full benefits of participation in academic clubs. That is, low-income communities and schools may have fewer activities available and fewer high-quality resources (i.e., teachers) to support these activities even if they were present. Moreover, youth in low income families may have more restraints on their free time because of caregiver and household responsibilities that constrain the time they can spend away from home (Simpkins et al., 2005). From a policy perspective, there are implications from our findings that should be highlighted. For the general student sample, the data showed a statistically significant association between STEM club participation and STEM achievement and between STEM club participation and subsequent selection of a STEM major in college. This can be viewed as a form of exposure that reinforces students’ interests and also supports their academic self-efficacy. It also further validates the relationship between preparation, motivation, and exposure by documenting how academic club participation influences academic performance in STEM (Williams & Gottfried, 2010). On the other hand, given the fact that there appeared to be little association between STEM club participation and STEM outcomes for low SES students, it would be important for further research to examine why it is in particular that these relationships do not hold for low SES students. On the one hand, it could be an institutional issue: schools serving these students might be exposed to lower quality teaching, fewer best practices, higher class sizes, and less access to opportunities in higher education – all which may inhibit students from enrolling in STEM clubs or being successful in STEM. Alternatively, it might be the case that low SES students are indeed interested in STEM but there may be external factors inhibiting successful STEM experiences, such as afterschool employment or family obligations. If this latter situation were the case, schools serving low SES students may need to be more innovative in their approach to reaching those students around non- formal STEM experiences. In thinking about ways for schools to involve low SES students, it is important to recognize that schools in high-poverty communities are often challenged financially which limits the available E ducation Policy A nalysis A rchives V ol. 21 N o. 79 20 resources that would support such activities. A way to combat this would be to provide targeted resources and supports to boost exposure to STEM-specific activities; doing so may lift the achievement and persistence in math and science for those youth who are at risk for educational failure (Sirin, 2005). This also points to the need for these schools to leverage existing resources by partnering with community based organizations, local STEM industries, and neighboring colleges and universities. These outlets of non-formal STEM exposure would build community relationships and create more avenues for exposure to authentic, contextual experiences in STEM for underserved students rather than having to rely exclusively on formal instruction or resource- constrained non-formal in-school activities. If these opportunities are provided in a club-like setting, students might then feel more compelled to participate in an ongoing basis which could in- turn have an impact on current schooling success, future college STEM participation, and ultimately entry into the STEM workforce. Limitations and Recommendations for Further Study There are several limitations in this study that provide points for future investigation. One particular limitation is that the analysis does not take into consideration the format or how the academic clubs are organized. While it is certain that these math or science clubs were school-based activities, it was not clear from the dataset whether the clubs were conducted during the school day (i.e., lunchtime) or after school. It was also not clear who is sponsoring the club (i.e., teacher, parent, administrator), nor are the qualifications of these sponsors delineated in the dataset. These details are important for future research: if clubs were after school rather than during school or sponsored by different types of individuals for low- versus high-poverty students, this may explain the differences that arose when evaluating the models for students by SES. A second limitation is the lack of specificity of the duration and consistency for which the students actually participated in the academic clubs. This is important to consider, as the number of contact hours might directly influence how well students do in their general coursework. This would relate back to why the data showed little association between STEM club participation and STEM outcomes for students affected by poverty because of the time restraints and other competing forces that prevent full and consistent participation. With more detail, future quantitative or qualitative research can focus on these precise differences between clubs in high- and low-poverty schools, which is not possible using the present dataset. A third limitation is that the study is not specific about the characteristics of the academic clubs, such as robotics or career interest (e.g. Junior Engineering and Technology Society). There is a need for further investigation of domain-specific academic clubs and activities to better understand how they engage students in STEM learning. To date, there is a limited body of knowledge as it relates to this particular subject, and it would benefit the education and policy communities to know how best to leverage these non-formal activities to enhance the academic achievement of all students. For example, robotics has played a major role in student engagement for many decades. But what role do robotics clubs and experiences play in STEM learning, and how do they motivate students to persist in STEM thus leading to STEM related academic and career pursuits? Learning from such activities that have established connections to both formal and non-formal settings would further enable a more clear understanding of how effective these activities are at substantially increasing STEM preparation, persistence, and success. ST E M C lub P articipation and ST E M Schooling O utcom es 21 References Augustine, N., Barrett, C., Cassel, G., Grasmick, N., Holliday, C., Jackson, S. A., Jones, A. K. , Levin, R., Mote, C. D., Murray, C., O’Donnell, P., Raymond, L. R., Richardson, R. C., Vagelos, P., Vest, C. M. , Whitesides, G. M. , & Zare, R. N. (2010) Rising above the gathering storm revisited: Rapidly approaching category 5. Washington D. C.: The National Academies Press. Argys, L. M., Rees, D. I., & Brewer, D. J. (1996). Detracking America’s schools: Equity at zero cost? Journal of Policy Analysis and Management, 15, 623-645. Becker, S.O., & Ichino, A. (2002). Estimation of average treatment effects based on propensity scores. Stata Journal, 2, 358-377. Broch, B. (2002). Linking extracurricular programming to academic achievement: Who benefits and why? Sociology of Education, 75, 69-91. Butler, B., & Serrell, D. (2001) Informal learning in public gardens: the process and the product. Public Garden (autumn), 4-9. Caldas, S. J., & Bankston III, C. (1997). Effect of school population socioeconomic status on individual academic achievement. Journal of Educational Research, 90, 269-277. Camp, W. (1990). Participation in student activities and achievement: A covariance structural analysis. Journal of Educational Research, 83, 272-278. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Crisp, G., Amaury, N., Taggart, A. (2009). Student characteristics, pre-college, college and environmental factors as predictors of majoring in and earning a STEM degree: An analysis of students attending a Hispanic serving institution. American Educational Research Journal, 46, 924-942. Csikszentmihalyi, M (1996). Creativity: flow and the psychology of discovery and invention. New York, NY: Haper Collins. Datar, A. (2006). Does delaying kindergarten entrance given children a head start? Economics of Education Review, 25, 43-62. Donaldson, K., Lichtenstein, G., & Sheppard, S. (2008). Socioeconomic status and the undergraduate engineering experience: Preliminary findings from four American universities. Paper presented at the Annual Conference of the American Society of Engineering Education, Pittsburgh, PA. Eccles, J., & Barber, B.(1999). Student council, volunteering, basketball, or marching band: what kinds of extracurricular involvement matters? Journal of Adolescent Research, 14, 10-43. Eccles, J., & Gootman, J.A. (2002). Community programs to promote youth development. Washington, D.C.: National Academy Press. Gottfried, M.A., & Polikoff, M.S. (2012). A religious experience? Personal, parental, and peer religiosity and the academic success of sexual-minority youth. Journal of LGBT Youth, 9, 183- 199. Greenfield, T.A. (1996) Gender, ethnicity, science achievement and attitudes. Journal of Research in Science Teaching, 33, 901-933. Hackett, G. (1985). The role mathematics self efficacy in the choice of mathematics-related majors of college women and men: A path analysis. Journal of Counseling Psychology, 32, 47-56. Hackett, G., & Betz, N.E. (1981). An exploration of the mathematics self-efficacy/mathematics performance correspondence. Journal of Research in Mathematics Education, 20, 261-273. E ducation Policy A nalysis A rchives V ol. 21 N o. 79 22 Harris, K.M., Halpern, C.T., Whitsel, E., Hussey, J., Tabor, J., Entzel, P., & Udry, J.R. (2009). The National Longitudinal Study of Adolescent Health: Research Design. Retrieved at: http://www.cpc.unc.edu/projects/addhealth/design. Heckman, J.J. (1996). Identification of causal effects using instrumental variables: Comment. Journal of the American Statistical Association, 91, 459-462. Holland, A., & Andre, T. (1987). Participation in extracurricular activities in secondary school: What is known, what needs to be known? Review of Education Research, 57, 437-466. Jones, L.S. (1997). Opening doors with informal science: exposure and access for our underserved students. Science Education, 81, 663-667. Lent, W.L., Lopez, F.G., & Bieschke, K.J. (1991). Mathematics self efficacy: Sources and relations to science-based career choice. Journal of Counseling Psychology, 38, 424-430. Lipscomb, S. (2007). Secondary school extracurricular involvement and academic achievement: A fixed effects approach. Economics of Education Review, 26, 463-472. Luellen, J.K., Shadish, W.R., & Clark, M.H. (2005). Propensity scores: An introduction and experimental test. Evaluation Review, 29, 530-558. Marsh, H.W. (1992). The effects of participation in sport during the last two years of high school. Sociology of Sport Journal, 10, 18-43. National Commission on Children (1991). Speaking of kids. Washington, D.C.: Author. National Federation of State High School Association. (2005). 2004-2005 Participation summary. Indianapolis, IN: Author. National Research Council (2011). Successful K-12 STEM education: Identifying effective approaches in science, technology, engineering, and mathematics. Washington, D.C.: The National Academies Press. Noguera, P.A. (2003). The trouble with black boys: The role and influence of environmental and cultural factors on the academic performance of African American males. Urban Education, 38, 431-459. Ogbu, J.U. (1989). The individual in collective adaptation: A framework for focusing on academic underperformance and dropping out among involuntary minorities. In L. Weis, E. Farrar, & H. G. Petrie (Eds.), Dropouts from school: Issues, dilemmas, and solutions. Albany, NY: State University of New York Press Pagan, A., & Ullah, A. (1999). Nonparametric econometrics. Cambridge, U.K.: Cambridge University Press. Rask, K. (2010). Attrition in STEM fields at a liberal arts college: The importance of grades and pre- collegiate preferences. Economics of Education Review, 29, 892-900. Rennie, L. (1987). Out-of-school science: Are gender differences related to subsequent attitudes and achievement in science? In J. Daniels & J. Kahle (Eds.), Contributions to the Fourth Girls and Science and Technology Conference (pp. 8–15). Ann Arbor, MI: University of Michigan. Rosenbaum. P.R., & Rubin, D. (1983). The central role of the propensity score in observational studies of causal effects. Biometrika, 70, 51-55. Rowe, M.B. (1977). Why don’t blacks pick science? The Science Teacher, 44, 34-35. Sampson, R.J., & Laub, J.H. (1994). Urban poverty and the family context of delinquency: A new look at structure and process in a classic study. Child Development, 65, 523-540. Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W. H., & Shavelson, R. J. (2007). Estimating causal effects using experimental and observational designs. Washington, DC: American Educational Research Association. Sheldon, S.B. & Epstein, J.L. (2005). Involvement counts: Family and community partnerships and mathematics achievement, Journal of Educational Research, 98, 196-206. ST E M C lub P articipation and ST E M Schooling O utcom es 23 Shernoff, D.J., & Vandell, D.L. (2007). Engagement in after school program activities: Quality of experience from the perspective of participants. Journal of Youth Adolescence, 36, 891-903. Simpkins, S.D., Davis-Kean, P.E., & Eccles, J.S. (2006). Mathematics and science motivation: A longitudinal examination of the links between choices and beliefs. Developmental Psychology, 42, 70-83. Simpson, J.C. (2001). Segregated by subject: Racial differences in the factors influencing academic major between European Americans, Asian Americans, and African, Hispanic, and Native Americans. The Journal of Higher Education, 72, 63-100. Sirin, S. (2005). Socio-economic status and academic achievement: A meta-analytic review of research. Review of Education Research, 75, 417-453. Summers, A. A., & Wolfe, B. I. (1977). Do schools make a difference? American Economic Review, 67, 639-652. Thomas, G. (1986). Cultivating the interest of women and minorities in high school mathematics and science. Science Education, 70, 31-43. Vandell, D.L., Pierce, K.M., & Dadisman, K. (2005). Out-of-school settings as a developmental context for children and youth. In R.V. Kail (Ed.), Advances in child development and behavior, 33, 43-77. New York, NY: Academic. Williams, D.N., & Gottfried, M.A. (2010). Who chooses the E in STEM? Proceedings of Engineering Education, RDP 34, 1-20. E ducation Policy A nalysis A rchives V ol. 21 N o. 79 24 About the Authors Michael A. Gottfried Loyola Marymount University mgottfri@lmu.edu Michael A. Gottfried, PhD, is an assistant professor in the School of Education at Loyola Marymount University. His current research agenda pertains to the effect of schooling contexts, peer effects, outcomes for underrepresented student populations, and STEM learning, coursework, and non-formal experiences. Darryl N. Williams Tufts University Darryl.Williams@tufts.edu Darryl N. Williams, PhD, is Associate Dean of Recruitment, Retention, and Community Engagement for Tufts University's School of Engineering. He holds a secondary appointment as a research associate professor in chemical and biological engineering focusing on nanobiotechology research. Serving as the director of the Center for STEM Diversity, he is deeply involved in engineering education research with emphasis on learning environments that foster and sustain student-driven innovation and the recruitment, retention, and engagement of students from underrepresented groups in the science, technology, engineering and math fields. education policy analysis archives Volume 21 Number 79 October 7th, 2013 ISSN 1068-2341 Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non- commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or EPAA. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizona State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A2 (Brazil), SCImago Journal Rank; SCOPUS, SOCOLAR (China). Please contribute commentaries at http://epaa.info/wordpress/ and send errata notes to Gustavo E. Fischman fischman@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. ST E M C lub P articipation and ST E M Schooling O utcom es 25 education policy analysis archives editorial board Editor Gustavo E. Fischman (Arizona State University) Associate Editors: David R. Garcia (Arizona State University), Stephen Lawton (Arizona State University) Rick Mintrop, (University of California, Berkeley) Jeanne M. Powers (Arizona State University) Jessica Allen University of Colorado, Boulder Christopher Lubienski University of Illinois, Urbana- Champaign Gary Anderson New York University Sarah Lubienski University of Illinois, Urbana- Champaign Michael W. Apple University of Wisconsin, Madison Samuel R. Lucas University of California, Berkeley Angela Arzubiaga Arizona State University Maria Martinez-Coslo University of Texas, Arlington David C. Berliner Arizona State University William Mathis University of Colorado, Boulder Robert Bickel Marshall University Tristan McCowan Institute of Education, London Henry Braun Boston College Heinrich Mintrop University of California, Berkeley Eric Camburn University of Wisconsin, Madison Michele S. Moses University of Colorado, Boulder Wendy C. Chi* University of Colorado, Boulder Julianne Moss University of Melbourne Casey Cobb University of Connecticut Sharon Nichols University of Texas, San Antonio Arnold Danzig Arizona State University Noga O'Connor University of Iowa Antonia Darder University of Illinois, Urbana- Champaign João Paraskveva University of Massachusetts, Dartmouth Linda Darling-Hammond Stanford University Laurence Parker University of Illinois, Urbana- Champaign Chad d'Entremont Strategies for Children Susan L. Robertson Bristol University John Diamond Harvard University John Rogers University of California, Los Angeles Tara Donahue Learning Point Associates A. G. Rud Purdue University Sherman Dorn University of South Florida Felicia C. Sanders The Pennsylvania State University Christopher Joseph Frey Bowling Green State University Janelle Scott University of California, Berkeley Melissa Lynn Freeman* Adams State College Kimberly Scott Arizona State University Amy Garrett Dikkers University of Minnesota Dorothy Shipps Baruch College/CUNY Gene V Glass Arizona State University Maria Teresa Tatto Michigan State University Ronald Glass University of California, Santa Cruz Larisa Warhol University of Connecticut Harvey Goldstein Bristol University Cally Waite Social Science Research Council Jacob P. K. Gross Indiana University John Weathers University of Colorado, Colorado Springs Eric M. Haas WestEd Kevin Welner University of Colorado, Boulder Kimberly Joy Howard* University of Southern California Ed Wiley University of Colorado, Boulder Aimee Howley Ohio University Terrence G. Wiley Arizona State University Craig Howley Ohio University John Willinsky Stanford University Steve Klees University of Maryland Kyo Yamashiro University of California, Los Angeles Jaekyung Lee SUNY Buffalo * Members of the New Scholars Board E ducation Policy A nalysis A rchives V ol. 21 N o. 79 26 archivos analíticos de políticas educativas consejo editorial Editor: Gustavo E. Fischman (Arizona State University) Editores. Asociados Alejandro Canales (UNAM) y Jesús Romero Morante (Universidad de Cantabria) Armando Alcántara Santuario Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Fanni Muñoz Pontificia Universidad Católica de Perú Claudio Almonacid Universidad Metropolitana de Ciencias de la Educación, Chile Imanol Ordorika Instituto de Investigaciones Economicas – UNAM, México Pilar Arnaiz Sánchez Universidad de Murcia, España Maria Cristina Parra Sandoval Universidad de Zulia, Venezuela Xavier Besalú Costa Universitat de Girona, España Miguel A. Pereyra Universidad de Granada, España Jose Joaquin Brunner Universidad Diego Portales, Chile Monica Pini Universidad Nacional de San Martín, Argentina Damián Canales Sánchez Instituto Nacional para la Evaluación de la Educación, México Paula Razquin UNESCO, Francia María Caridad García Universidad Católica del Norte, Chile Ignacio Rivas Flores Universidad de Málaga, España Raimundo Cuesta Fernández IES Fray Luis de León, España Daniel Schugurensky Arizona State University Marco Antonio Delgado Fuentes Universidad Iberoamericana, México Orlando Pulido Chaves Universidad Pedagógica Nacional, Colombia Inés Dussel FLACSO, Argentina José Gregorio Rodríguez Universidad Nacional de Colombia Rafael Feito Alonso Universidad Complutense de Madrid, España Miriam Rodríguez Vargas Universidad Autónoma de Tamaulipas, México Pedro Flores Crespo Universidad Iberoamericana, México Mario Rueda Beltrán Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Verónica García Martínez Universidad Juárez Autónoma de Tabasco, México José Luis San Fabián Maroto Universidad de Oviedo, España Francisco F. García Pérez Universidad de Sevilla, España Yengny Marisol Silva Laya Universidad Iberoamericana, México Edna Luna Serrano Universidad Autónoma de Baja California, México Aida Terrón Bañuelos Universidad de Oviedo, España Alma Maldonado Departamento de Investigaciones Educativas, Centro de Investigación y de Estudios Avanzados, México Jurjo Torres Santomé Universidad de la Coruña, España Alejandro Márquez Jiménez Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Antoni Verger Planells University of Amsterdam, Holanda José Felipe Martínez Fernández University of California Los Angeles, USA Mario Yapu Universidad Para la Investigación Estratégica, Bolivia   ST E M C lub P articipation and ST E M Schooling O utcom es 27 arquivos analíticos de políticas educativas conselho editorial Editor: Gustavo E. Fischman (Arizona State University) Editores Associados: Rosa Maria Bueno Fisher e Luis A. Gandin (Universidade Federal do Rio Grande do Sul) Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Brasil Jefferson Mainardes Universidade Estadual de Ponta Grossa, Brasil Paulo Carrano Universidade Federal Fluminense, Brasil Luciano Mendes de Faria Filho Universidade Federal de Minas Gerais, Brasil Alicia Maria Catalano de Bonamino Pontificia Universidade Católica-Rio, Brasil Lia Raquel Moreira Oliveira Universidade do Minho, Portugal Fabiana de Amorim Marcello Universidade Luterana do Brasil, Canoas, Brasil Belmira Oliveira Bueno Universidade de São Paulo, Brasil Alexandre Fernandez Vaz Universidade Federal de Santa Catarina, Brasil António Teodoro Universidade Lusófona, Portugal Gaudêncio Frigotto Universidade do Estado do Rio de Janeiro, Brasil Pia L. Wong California State University Sacramento, U.S.A Alfredo M Gomes Universidade Federal de Pernambuco, Brasil Sandra Regina Sales Universidade Federal Rural do Rio de Janeiro, Brasil Petronilha Beatriz Gonçalves e Silva Universidade Federal de São Carlos, Brasil Elba Siqueira Sá Barreto Fundação Carlos Chagas, Brasil Nadja Herman Pontificia Universidade Católica –Rio Grande do Sul, Brasil Manuela Terrasêca Universidade do Porto, Portugal José Machado Pais Instituto de Ciências Sociais da Universidade de Lisboa, Portugal Robert Verhine Universidade Federal da Bahia, Brasil Wenceslao Machado de Oliveira Jr. Universidade Estadual de Campinas, Brasil Antônio A. S. Zuin Universidade Federal de São Carlos, Brasil