Microsoft Word - 699-2799-6-LE.docx 1 Volume 14, Number 2 May 28, 2011 ISSN 1099-839X Do Schools Make a Difference? A Study of High School Effects and First Year College Success Wade Smith Jason Droddy Louisiana State University A. J. Guarino Massachusetts General Institute of Health Schools across America are being ranked for their effectiveness on a number of student criteria, among them preparation of students for a successful college experience. This study investigates the relationship between graduating seniors, their successful first year retention in college and several personal and school related factors. The study also explores the relationships between school accountability rankings and the demographics of the respective schools in an effort to assess school effect sizes for student growth, a mandate of No Child Left Behind (NCLB). Nearly 11,000 high school seniors from more than 280 schools were included in the analyses. When taken collectively, results from the study indicate that school accountability rankings provide little predictive value when assessing successful first year college retention. Keywords: accountability, school effects, college retention “Success in college is predicated upon success in high school.” Intuitively this just makes sense, and as a result, students who perform well in high school are rewarded with scholarships and are courted by colleges seeking their admission. Many states in an effort to bolster the high school experience are now offering concurrent or dual enrollment courses. In these courses, students are able to receive both Carnegie units and college credits simultaneously and the interest in these opportunities is growing (U.S. Department. of Education, 2010). These efforts reflect the belief that the high school experience contributes to subsequent success in college. Beyond the high school to college expectations, No Child Left Behind applies pressure to states and school districts to develop and implement accountability measures with the express purpose of improving student learning and achievement. In Louisiana, the site of this study, the accountability system has a goal of documented improvement in student achievement and it is reasonable to conclude that one of the desired outcomes is students that are well-prepared for success in higher education. This conclusion is supported by a state constitutional requirement that requires the Louisiana Board of Elementary and Secondary Education (BESE) to meet semi-annually with Louisiana’s higher education coordinating board to coordinate programs (Louisiana Constitution,1974). Although the data base is unique to Louisiana, efforts to affect human and social capital as well as individual productivity via schools are by no means unique C u r r e n t I s s u e s in E d u c a tio n V o l. 1 4 N o . 2 2 to this particular state. Historically, America has held a strong belief that schools provide students of all social classes with the means to achieve (Traub, 2000) and school accountability models are expected to provide confirmation of their successes or failures. A core goal of accountability models in general, and in the particular case of Louisiana’s accountability model, is the measurement of a school’s contribution to students’ academic success. Therefore, schools can be viewed as the provider of environmental stimuli that are expected to shape and facilitate productive student behaviors. Or, said another way, Louisiana’s accountability model attempts to categorize the extent individual schools are preparing their students. These efforts borrow heavily from tenets of environmental determinism, a view most notably associated with B. F. Skinner (e.g., 1971, 1974) and can be summarized succinctly by the phrase “the school makes the student.” No one would likely argue the statement to be literally true. However, the underlying basis for the statement is indeed a powerful contributor to school accountability efforts throughout the nation. Accountability models are, therefore, efforts to document school level effects as they pertain to student achievement. Environmental solutions provide a practical means of attempting to facilitate and document student learning. The solutions can be applied in a rather universal fashion, providing an efficient means of attempting to improve school and student productivity. This contrasts markedly with efforts that attempt to change the individual belief systems of students. Processes such as these would, by necessity, tend to be idiosyncratic and time consuming. Furthermore, how exactly would an accountability system accurately recognize and reward the school for these internal shifts in motivation or perseverance? For reasons such as these, it is understandable why accountability models such as Louisiana’s tend to focus on measuring school effects. That said, it is important to remember that school effects are mediated by the collective and individual results of environmental conditioning being imported to the school by its student body, not to mention the role that strongly held internal beliefs of students, faculty, and indeed all stakeholders will ultimately play in determining just how strong the school level effects actually are. For these reasons there is compelling justification to analyze how the two contexts might interact. In other words,” To what extent do the students make the school or the schools make the students?” is a valid question. Research conducted by Payne and Biddle (1999) and Berliner and Biddle (1995) attest to longstanding efforts towards these ends. Louisiana’s school accountability model categorizes schools into five categories and these are designated by stars. Five star schools are the highest ranking school and 1 star the lowest. Accountability rankings are a combination of attendance, standardized test scores, and graduation rates (Louisiana Department of Education, 2009). Because a 5 Star school has documented success for the accountability parameters, it is reasonable to assume these effects will translate into the success of the students who have enjoyed the collective benefits of the opportunities afforded them. Conversely, poor performing schools have strict guidelines they must follow in order to work towards improving their school scores. If these efforts do not produce results a school is designated as” academically unacceptable.” If progress is not forthcoming the school is designated as a “failed school” and is taken over by the Recovery School District (Louisiana Department of Education, 2010). If a school’s success is the order of the day for documenting school accountability, it is reasonable to infer that a school’s accountability ranking is a valid indicator of success in delivering appropriate instruction. From this perspective, it is also reasonable to infer that student success in college should be influenced by the level of success schools have been able to generate regarding these effects. It is also sensible to posit that other variables contributing to a school’s accountability score may also influence a student’s success in their first year of college (Cohn, Cohn, Balch, & Bradley, 2004; DeBeard, Spielmans, & Julka, 2004; Tinto, 1993). With that in mind, the study included variables that would indicate a student’s academic preparation, socioeconomic status (SES), and the SES of the student’s school. These variables were included to provide means of assessing effects that might be primarily attributed to a school along with those that would likely be outside the scope of the school’s primary influence. The variables were operationalized through widely recognized indicators of these factors (National Association for College Admissions Counseling, 2008), such as high school grade point average on a college preparatory curriculum (CoreGPA) and eligibility status for the state’s merit-based scholarship called the Taylor Opportunity Program for Students (TOPS). Additionally, in an effort to determine whether school effects might be extending to disadvantaged students Pell Grant status was also included as a study variable. Finally, with the question of school level effects being raised, we believe it is essential to investigate the extent (if any) that school accountability rankings might be a function of a school’s attendant student demographics. This study has three primary objectives. First, this study investigates the predictability of school accountability status (STAR) upon first year college retention for students. A second concern is the nature of the relationships of successful first-year college retention to Pell Grant status, Core GPA, TOPS eligibility, and proportion of students on Free/Reduced Lunch. Finally, the study investigates the relationships between school accountability rankings and attendant student populations. Collectively, these analyses Do Schools Make a Difference? A Study of High School Effects and First Year College Success 3 Table 1 Number and Percent of Students in STAR Status Schools, TOPS, and PELL Eligibility Variable Status n Percent 1 3229 29.7% 2 4969 45.8% 3 2308 21.3% 4 354 3.3% STARS Total 10860 100.0% No 3715 34.2% Yes 7145 65.8% TOPS Total 10860 100.0% No 6799 62.6% Yes 4061 37.4% PELL Total 10860 100.0% Table 2 At-Risk Students by STAR Mean Percents, SD, and N STARS Mean Std. Deviation N 1 62.7788 15.34221 3250 2 43.1878 12.18475 4988 3 31.2507 15.17589 2324 4 20.0289 10.73974 355 Total 45.7258 18.60767 10917 Table 3 Mean Core GPA and Percent of At-Risk Students in Sample n Minimum Maximum Mean Std. Deviation Core GPA 10860 .33 4.00 3.1004 .61332 Covariate ATRISK 10860 2.09 100.00 45.6965 18.57601 are belived to provide a picture of the interactions between school level effects and subsequent student success in the first year of college. Methods The Louisiana Board of Regents, the state’s higher education coordinating body, provided a dataset of 17,123 student-level records of students who graduated from a Louisiana high school in 2006 and enrolled in a public four- year university the same year. The sample was refined to 10,935 due to obvious data entry errors, missing data, and graduates of private and parochial schools that are not subject to the state’s accountability system. Retention is defined as a 2006 Louisiana public or non-public high school graduate who began at public four- year university in fall 2006 and enrolled at any public postsecondary institution, two- or four-year. Students who were retained in the public postsecondary system were identified with a “1”, and students were not retained or transferred out of the public system were assigned a “0”. C u r r e n t I s s u e s in E d u c a tio n V o l. 1 4 N o . 2 4 The Louisiana School Accountability System assigns a baseline school performance score (SPS) comprised of data from three statewide student assessments, two subject area tests, attendance, drop-out rates, and graduation data (Louisiana Department of Education, 2009). Schools with higher baseline SPS earn a higher rating from the state, and we assigned a value (STAR) equivalent to the number of stars each school achieved. A baseline SPS of 140 or greater is equivalent to five stars, four stars to 120-39 points, three stars to 100-119 points, two stars to 80-99, and one star to an SPS of 60-79. Schools scoring below 60 are considered “academically unacceptable,” and we coded those as “0” (Louisiana Department of Education, 2009). Given the small number of schools designated as 4 and 5 star schools, it was decided to collapse these schools into one category (STAR4). Unacceptable schools (O stars) and 1 star schools were also collapsed into one category for the same reason, providing the categories of STAR1, STAR2, STAR3, and STAR4 respectively. Noting NCLB’s stated interest in improving student performance across all sub-populations, it was deemed necessary to use a variable indicative of community wealth which could also imply social capital. Heller and Rasmussen (2002, August) created a “school SES” variable using the proportion of students at each school who qualified for free- or reduced-lunch and they used the term “at risk.” An ATRISK variable was included to provide a means of analyzing relationships between a school’s demographics and its accountability ranking. At-risk is the proportion of students in each school who qualify for the free and reduced lunch program. The mean at-risk proportion in this sample is 45.73%, indicating that at the average Louisiana public high school; nearly half of the student population qualify for free or reduced lunch (Louisiana Department of Education, 2006). Results Data were analyzed using PASW (18) employing the following statistical tests, (a) Multi-level Modeling, (b) Generalized Estimating Equation (GEE) with ordinal logistic regression, (c) GEE with binary logistic regression, and (d) Trend analysis with simple contrasts. Prior to analyses, descriptive statistics for the sample were collected. Of interest were the disproportionate levels of students in 3 star or lower schools (96.7%) and the small number of students attending four or five star schools (3.3%). Approximately 65% of the students in the sample were TOPS eligible while over 37% were Pell eligible, suggesting that many of the students who are TOPS eligible could also be categorized as “at-risk.” It is also worth noting that the mean core GPA for the group was 3.10. This information is provided in Tables 1-3. Analysis 1: Predictors of First Year Success in College To assess retention rates among the school districts, an unconditional Multi-Level Model analysis revealed that there was significant variability among the districts in retention rates, indicating that a conditional model could explain additional variability. The conditional model included the following predictors, (a) Pell Grant (1= Yes, 0 =No), (b) TOPS (1 = Yes, 0 =No), and (c) STAR Status (4 = Highest Level to 1 = Lowest Level) with Free/Reduced Lunch and Core GPA as covariates. Results indicated that holding all other predictor variables constant; students attending STAR4 schools were more likely to persist compared to STARS1 or 2 classified schools with odds ratios of 1.38 and 1.62 respectively. There were no statistically significant differences in retention between STAR 3 and 4 schools. Pell Grant recipients were less likely to persist with an odds ratio of 0.87. Students achieving TOPS demonstrated no statistically significant differences in retention in this model, regardless of their school’s accountability ranking. This conditional model demonstrated a substantial improvement in model fit by reducing the Quasi Likelihood under the Independence Model Criterion (QIC) value from 48521.73 to 37609.37. Analysis 2: Predicting TOPS Recipients by School Accountability Status Generalized Estimating Equation (GEE) with ordinal logistic regression was conducted to assess the effects TOPS status (1=yes, 0=no), Pell Grant Status (1=Yes, 0= No), Core GPA, and ATRISK values on STAR classification (1=poor to 4=highest rating). Although previous analyses indicated no significant difference for the retention rates of TOPS students from high or low performing schools, the data indicate there is a significant relationship between a school’s accountability status and the likelihood of students receiving TOPS. Data analysis indicate STAR 4 schools compared to STAR 1, 2, and 3 schools were more likely to produce TOPS recipients 1.66, 1.26, and 1.14 times respectively. Analysis 3: Predicting Pell Recipients by School Accountability Status Results suggest that STAR 1 schools compared with STAR 2, 3, and 4 schools (while holding all other predictors constant), were more likely to produce Pell recipients 1.81, 2.62, and 2.40 times respectively. Analysis 4: Is there a relationship between At- Risk students and STAR Classification? Results indicated a statistically significant trend, F(3, 10917) = 2969.22, p < .001, eta-squared = .45. Simple contrasts supported the linear trend that lower rated schools Do Schools Make a Difference? A Study of High School Effects and First Year College Success 5 reported higher levels of Free/Reduced lunches. These results are represented in Figure 1. Discussion Taken collectively, our analyses provide little evidence to suggest that any of our findings confirm a substantive school level effect is translating into subsequent student success in the first year of college. In fact, the totality of our analyses suggests that neither school accountability models nor any of the other attendant variables in this study are accurate predictors of first-year college success for students. We did find that a school’s accountability ranking was predictive of the likelihood of astudent receiving TOPS, but one must wonder what the importance of this finding is when faced with data that indicate students achieving TOPS demonstrated no statistically significant differences for first year college success. Furthermore, retention rates for students receiving TOPS from a one STAR school demonstrated no significant difference for retention rates than students from high performing schools. One could reasonably infer that a school’s higher ranking would indicate the likelihood of a school level effect providing some advantage to their students, but our analyses failed to document this possibility. With regard to predicting first year college success via a school’s accountability ranking, the analyses again demonstrated a statistically significant effect. Students from higher performing schools do indeed tend to be successfully retained at a higher rate than those who graduate from 1 or 2 STAR schools. However, this too must be tempered with the findings that nearly 45% of a school’s accountability ranking can be accounted for by the level of ATRISK students that attend. It seems possible (likely?) that the findings for school accountability rankings are operating as proxy measures for levels of human and social capital present in a school, at least as it pertains to first-yea college success. This possibility seems to be supported by the findings for Pell grants that indicate Pell recipients are less likely to persist when compared to those who do not receive the grant. In short, our initial analyses provided little evidence to advance an argument that a school’s accountability status has significant predictive value for their students’ first year college retention. We are not altogether surprised by this finding. There is a year of distance between the last direct influence a high school has upon a senior and the culmination of the freshmen year in college. A tremendous amount of variables beyond those included in this study can influence first year success in college, and those that are part of the first-year college experience seem to be far more important. Additionally, the small sample size for high performing schools may also limit our conclusions. It should be noted that none of our data provide any personal profiles so we do not know how motivated these students were to attend college or the degree to which they are inclined to persevere. Finally, and perhaps most importantly, we do not know the levels of attention colleges are providing to their incoming freshmen in an effort to maximize first-year retention. Figure 1. Simple Trend Analysis of At-Risk Populations vs Star Classification C u r r e n t I s s u e s in E d u c a tio n V o l. 1 4 N o . 2 6 It seems to us however that a broader issue emerges regarding school accountability. School accountability models are expected to discriminate between levels of school performance and the information about these schools is expected to inform public opinion regarding the quality of education received from a school. Linn (2000) recognizes this reality and notes that the ready accessibility of accountability rankings is a major factor in their popularity. Because of this ready access, innumerable parents have received accountability results and scrutinized high schools searching for the best possible educational experience with a belief that this decision will have a large bearing on subsequent college success for their child. In these cases parents are behaving rationally, using information provided to them via a state accountability model to maximize their child’s future educational opportunities. However, these decisions do not seem to be supported by the real world evidence. We are not arguing that choice of school decisions are without merit. There are obvious differences in instructional efficacy in Louisiana schools subject to the state accountability model. To argue otherwise would be absurd. We return however to a foundational issue at this point: accountability models represent a concerted effort to document school level effects regarding student success. They posit that “schools make the student.” If that position is held regarding first-year success of graduating students it will be difficult to defend, given the results of this study. Finally, No Child Left Behind requires states to develop school accountability models. NCLB also pays close attention to disadvantaged populations, requiring that data be disaggregated for this specific sub-population. Our findings suggest that something far different may be happening from the outcomes NCLB anticipates with at- risk populations. In spite of the considerable attention being paid to this sub-population in school accountability models there remains compelling empirical evidence indicating much work is left to be done. We do not want to overreach with our conclusions. However, this we know: In our study, nearly 280 schools with an average at-risk population of nearly 46% have individually tackled the problem of affecting substantial educational progress for their at-risk populations. Yet, after all of these efforts we are still left with compelling collective evidence suggesting that at-risk students are dramatically less likely to succeed in their first year of college, regardless of how their school is rated in the accountability model. That this trend is evident so clearly after nearly a decade of NCLB should give one pause when seeking solutions to the problem of elevating the educational advantages attained by students at-risk. Our data also indicate that nearly 45% of the variance in school accountability scores can be explained by the percentage of at-risk population found at a particular school. This finding should be tracked over time to see if this trend changes or if it is reinforced. Also, similar analyses should be conducted in different states to assess whether results generalize. As noted previously, at-risk students demonstrated no significant differences in retention rates regardless of their high school’s accountability ranking. Essentially, there was nothing in the data to suggest that at-risk students from high performing schools were more likely to succeed in their first year of college. There is, however, compelling evidence that higher performing schools tend to have lower percentages of at-risk students. But, when we looked for any evidence that there was a school level effect facilitating this phenomenon none was demonstrated in our analysis. This finding supports Traub’s conclusion that, in general, schools are not (cannot?) providing effects that effectively mediate for the myriad of external issues associated with at- risk populations. This implication, if it is borne out by further investigation, is significant. Some states, Louisiana included, are now designing or utilizing accountability systems that take performance scores from students as the primary component of a school accountability assessment (2 The Advocate, 2010). In essence, the level of analysis is being extended from the school level to the teacher level. And, at least in the case of Louisiana, the weight being assigned to variables other than assessment scores will “barely register in the assessment model,” as stated by the designer of Louisiana’s model in the aforementioned newspaper article. If this is the case, it will be interesting to see if the trend analysis conducted for this study varies significantly. Perhaps more important will be the retrospection that occurs regarding whether these efforts to document school level effects are, in fact, translating into subsequent student success. References 2 The Advocate (2010). New law changes teacher evaluation. Retrieved from http://www.2theadvocate.com/new/100953239.ht ml Berliner, D. C., & Biddle, B. J. 1995. The manufactured crisis: Myths, fraud, and the attack on America’s public schools. Reading, MA: Perseus Books. Cohn, E., Cohn, S., Balch, D., & Bradley, J. (2004). Determinants of undergraduate GPAs: SAT scores, high-school GPA and high- school rank. Economics of Education Review, 23(6), 577-586. DeBerard, M. S., Spielmans, G. I., & Julka, D. L. (2004). Predictors of academic achievement and retention among college freshmen: A longitudinal study. College Student Journal, 38(1), 66-80. Do Schools Make a Difference? A Study of High School Effects and First Year College Success 7 Heller, D. E., & Rasmussen, C. (2002, August). Merit scholarships and college access: Evidence from Florida and Michigan. In d. E. Heller & P. Marin (Eds.), Who Should We Help? The Negative Social Consequences of Merit Aid Scholarships. Linn, R. L. (2000). Assessments and accountability. Educational Researcher, 29(2), 4-16. Louisiana Constitution. (1974). Article VIII, Section 5(D). Retrieved April 4, 2010 from http://www.legis.state.lsu.us/lss/lss.asp?doc=2065 76 Louisiana Department of Education (2006). Multiple Statistics By Site Code For MFP and Other Public School Funded Student Membership - May 2006. May DB: May 01, 2006 Multi-Stats - By Site-4 (Funded Membership). Retrieved from http://www.louisianaschools.net/lde/pair/2289.html Louisiana Department of Education (2009). 2006-2007 accountability summary results: A user guide for the state and district table. Retrieved from http://www.louisianaschools.net/lde/uploads/1147 5.pdf Louisiana Department of Education (2010). Accountability page. Retrieved August, 18, 2010 from http://www.doe.state.la.us/lde/portals/accountabilit y.html Payne, K. J., & Biddle, B. J. August-September 1999. Poor school funding, child poverty and mathematics achievement. Educational Researcher 28(6), 4-13. Skinner, B. F. (1971). Beyond freedom and dignity. New York: Knopf. Skinner, B. F. (1974). About behaviorism. New York: Knopf. Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago: The University of Chicago Press. Traub, J. (Jan. 16, 2000). Schools are not the answer. The New York Times Magazine. Cambridge, MA: Harvard University. U.S. Department of Education (2010). High School Students Using Dual Enrollment Programs to Earn College Credits, New Reports Say. Retrieved from http://www2.ed.gov/news/pressreleases/2005/04/0 4062005a.html C u r r e n t I s s u e s in E d u c a tio n V o l. 1 4 N o . 2 8 Article Citation Smith, W., Droddy, J., & Guarino, A.J. (2011). Do schools make a difference? A study of high school effects and first year college success. Current Issues in Education, 14(2). Retrieved [date], from http://cie.asu.edu/ojs/index.php/cieatasu/article/view/699 Author Notes Wade Smith, PhD Louisiana State University 14255 Dalrymple Drive Baton Rouge, LA 70803 smithwa@lsu.edu Wade Smith is the Superintendent of the Louisiana State University Laboratory School. He maintains an active research agenda related to school level effects on student learning and social cognitive theory. Jason Droddy, PhD Louisiana State University 156 Thomas Boyd Hall Baton Rouge, LA 70803 jdroddy@lsu.edu Jason Droddy serves as Director in the Chancellor’s Office at LSU, specializing in external relations. His research interests include higher education funding and successful retention of college students. A. J. Guarino, PhD Massachusetts General Institute of Health 36 First Avenue, Charlestown Navy Yard Boston, MA 01880 ajguarino@mghihp.edu A. J. Guarino is a professor of inter-disciplinary studies and nursing at the Massachusetts General Institute of Health. He has co-authored the following statistics textbooks: Applied Multivariate Research Design and Interpretation (2006), Analysis of Variance Designs with SPSS and SAS (2008), and Data Analysis Using SAS Enterprise Guide (2009). Do Schools Make a Difference? A Study of High School Effects and First Year College Success 9 Volume 14, Number 2 May 28, 2011 ISSN 1099-839X Authors hold the copyright to articles published in Current Issues in Education. Requests to reprint CIE articles in other journals should be addressed to the author. Reprints should credit CIE as the original publisher and include the URL of the CIE publication. Permission is hereby granted to copy any article, provided CIE is credited and copies are not sold. Editorial Team Executive Editor Lori Ellingford Assistant Executive Editors Krista Adams Melinda Hollis Layout Editor Elizabeth Reyes Recruitment Editor Rory Schmitt Copy Editor/Proofreader Lucinda Watson Section Editors Hillary Andrelchik Meg Burke Douglas Deiss Elizabeth Frias Ayfer Gokalp Angela Hines Younsu Kim Seong Hee Kim Lisa Lacy Angeles Maldonado Carol Masser John Michael Tapati Sen Jennifer Shea Kara Sujansky Alaya Swann Melissa Tarango Andrew Tesoro Jill Wendt Faculty Advisors Dr. Gustavo Fischman Dr. Jeanne Powers