Higher Standards and Lower Achievement? Higher Standards and Lower Achievement? An Evaluation of Pennsylvania’s Keystone Exams ADAM McGLYNN East Stroudsburg University of Pennsylvania As the requirement that students pass three Keystone Exams to graduate from high school was set to take effect, a new round of criticisms arose over the adop- tion of this assessment system. While the Keystone Exams better align with the new Pennsylvania Core Standards as compared to their predecessor, the Pennsylvania System of Student Achievement, low passage rates especially among low-income and minority students have been cited as a reason to delay implementation of the graduation requirement. This work uses OLS regression analysis to explain which factors are most predictive of school-level performance on the Algebra I, Biology, and Literature Keystone Exams. It finds that race, socioeconomic status, and a school’s English Language Learner and special edu- cation populations drive performance on the exams. The work concludes by discussing possible policy interventions for the Keystone Exam program going forward. The impetus for much of the educational reforms that have been enact- ed over the last 30 years was the need for tougher standards. Quite simply, it was believed that reports such as A Nation at Risk (1983) and the results of international assessments such as the Trends in International Math and Science Study (TIMMS) and the Programme for International Stu- dent Assessment (PISA) were due to U.S. students not being pushed to achieve COMMONWEALTH, Volume 18, Issue 1 (June 2016). © 2016 The Pennsylvania Political Science Association. ISSN 2469-7672 (online). http://dx.doi.org/10.15367/cjppp.v18i1.83. All rights reserved. 32 ADAM McGLynn more. Policymakers believed that if more were required of students, then they would rise to the occasion, and student achievement would increase. Of course, standards are relatively useless unless there is a mechanism in place to assess their achievement. This work focuses on the assessment of the achieve- ment of higher standards in Pennsylvania by examining the recently imple- mented Keystone Exams, which replaced the Pennsylvania System of Student Achievement (PSSA) in high schools throughout the Commonwealth. Specifi- cally, the question this work intends to address is which factors most influence proficiency rates on the Keystone Exams in Pennsylvania high schools. There are multiple reasons for addressing this question, but the most important and timely reason is that in February of 2016, Governor Wolf signed legislation delaying the implementation of the requirement that stu- dents pass three Keystone Exams in Literature, Algebra I, and Biology in order to graduate from high school. This requirement was planned to take effect during the 2016–2017 school year with the new legislation delaying that requirement by two years until 2018–2019. The reason for the delay was to respond to complaints from school districts over the “high-stakes nature” of the exams and how to help students who have failed the exams complete alternative assessments (Murphy 2016). Therefore, this work will examine demographic, staffing, and programmatic variables in public high schools to identify which factors drive proficiency rates, which in turn could spur a discussion of targeted interventions that would aid high schools in the goal of increasing their proficiency rates on the Keystone Exams. Background The PSSA tests mirrored many other high stakes tests in the nation at the high school level where they assessed, for the most part, the minimum level of skill and knowledge believed to be required of a high school graduate. In many cases, the content of the high school PSSA exam was reading and mathemat- ics material that students had learned in eighth and ninth grades, despite the exams being conducted while students were in the 11th grade (Murphy 2015). This means that whether intentional or not, the PSSAs had elements similar to other state assessments and were not necessarily assessing material that students planning to attend college would be exposed to later in high school (Bishop et al. 2000). The Keystone Exam program thus emanates from a desire to assess stu- dent achievement of higher standards adopted in the form of the Pennsylvania Core Standards. The Pennsylvania Core Standards, with which the Keystone Exams are aligned, were derived from the national Common Core State Stan- Higher Standards and Lower Achievement? 33 dards (CCSS) with the goal of creating “standards aligned with expectations for success in college and the workplace” (Pennsylvania Department of Edu- cation 2014, ix). In 2008, the Commonwealth developed the Keystone Exams to create an assessment program “To provide a system that is aligned, focused, standards-based, accurate, universally applicable, and publicly accessible” (Pennsylvania Department of Education 2014, ix). The Keystone Exams thus serve as “final course exams” (Pennsylvania Department of Education 2014, ix), similar to what one would find in a col- lege-level course. The program was originally charged with creating assess- ments in 10 subject areas: Algebra I, Algebra II, Biology, Chemistry, Civics and Government, English Composition, Geometry, Literature, U.S. History, and World History. Each exam would account for at least one-third of a student’s final course grades, and students who did not score at the level of proficiency or advanced after two attempts would be able to complete a project-based assessment to meet the requirements. However, as of this writing, Pennsyl- vania has only implemented three exams: Algebra I, Biology, and Literature. These tests are being used to assess high schools as a part of the Pennsylvania School Performance Profile system, which was created under the waiver from the requirements of No Child Left Behind granted to the Commonwealth by the U.S. Department of Education. Considering the emphasis that has been placed on college attendance in the last 20 years, requiring students to pass final exams that are aligned with a recently completed curriculum makes sense in preparing students for the rigors of undergraduate education. In theory, an end-of-course exam (EOC) would increase standards as it motivates all students to prepare for the test, whereas a minimum-competency test (MCT) will be unlikely to pose a challenge to better performing students (Bishop et al. 2000). Most states adopted MCTs (if they were not already in place) after the adoption of No Child Left Behind in 2002. As NCLB evaluated schools based on a standard of proficiency, mini- mum-competency tests created an incentive to focus resources on the lowest performing students in a school and a district (Bishop et al. 2000). Achieving high levels of proficiency could ensure that a school and/or a district avoided the punitive measures associated with the failure to achieve adequate yearly progress as required under NCLB, but for the reasons stated above, it does not guarantee a movement toward higher standards and college readiness. Despite these perceived benefits of EOCs compared to MCTs, the United States has little experience with subject-area testing in the assessment of K–12 schools. As Isaacs (2014) notes, SAT subject tests, Advanced Placement (AP) courses and exams, as well as the subject tests conducted by the National Assessment of Educational Progress (U.S. Department of Education 2015) are the most 34 ADAM McGLynn prominent national examples of subject testing such as occurs in the new Key- stone Exam program. It should be noted that none of these assessments form the basis of education policy decisions at the national or state level, meaning that education policy nationally and locally is driven by the results of MCTs. Therefore, Pennsylvania is limited in the examples it has to draw from in the creation and implementation of its end-of-course exam program. One program that could serve as a guide for the use of EOCs is the New York State Regents Exam program, which has existed for over 100 years. However, for much of the program’s existence students were not required to take Regents exams. Students planning to go to a college or university would often com- plete the Regents track, which required passing five EOCs (Regents exams) in Global History and Geography, English, U.S. History, Mathematics, and Science (Isaacs 2014). The math and science exams are subject specific (i.e., Algebra, Geometry, Chemistry, Biology), but students are only required to pass one from each field. Successfully passing the five exams with a score of 65 or above would earn one a Regents Diploma or a Regents Diploma with honors if a student had an average pass rate of 85 or above. Students not planning to go to college could opt for the Regents Competency Tests (RCTs), which were less rigorous, and could still earn a high school diploma in the process (Isaacs 2014). Less than half of New York State students were graduating with Regents diplomas in the mid-1990s, which led then New York State Commissioner of Education Richard Mills to move to mandate the exams for all students in order to achieve a high school diploma (DeBray 2004). The expansion of the Regents exam program created many challenges, including concerns over low-income and minority student achievement and the validity of the tests in assessing the state’s learning standards, which led to proposed delays in the program’s implementation similar to what we currently see in Pennsylvania (Isaacs 2014). Further, there was a concern that standards would need to be lowered to ensure students could pass the exams, which of course in turn could weaken the value of the Regents exam program. This was problematic given that the New York State Education Department released a report that found that state colleges and universities believed only Regents exam scores of 85 or above, what the state viewed as graduation with dis- tinction, were indicative of readiness for higher education (Isaacs 2014; New York State Department of Education 2011). Teachers, while seeing benefits to changes in the testing program, also raised concerns regarding the implemen- tation of testing and corresponding curriculum changes without significant input from teachers (Grant 2000). Isaacs concluded her work by emphasizing that the strength of the Regents program is its alignment with the curriculum, which has only recently come into vogue in other states because of Common Higher Standards and Lower Achievement? 35 Core. However, this brief discussion of the New York State Regents exams shows that a comprehensive EOC system worked best when it was optional and helped raise standards for above average students; once this system was expanded to all students, significant challenges and growing pains emerged as is the case now in Pennsylvania. In theory EOC exams are a better assessment tool than an MCT, so why then in an era where there has been greater demand for accountability and increased standards would the Keystone Exams, which seemingly accom- plished both, be met with resistance? The reasons are numerous but start with an overall movement against increased testing in schools. In a letter to State Senator Andrew Dinniman, who stands in ardent opposition to the exams, superintendents from more than 50 schools in Southeastern Pennsylvania explained that the exams would place a tremendous burden on high school students and could lead to students spending as many as three or four weeks taking tests when the Keystones are added to AP and local course exams. Additionally, they explained that many eighth graders enrolled in Algebra I would be required to take both the eighth-grade PSSA exams and the Algebra I Keystone Exam (Coalition of Pennsylvania Public School Superintendents and Intermediate Unit Executive Directors 2013). Just as significant to the superintendents was the cost of preparing students for the exam and provid- ing remediation in the form of the project-based assessments that students could complete in lieu of passing a Keystone Exam. In a position paper posted to his website, Senator Dinniman argued that the costs associated with pre- paring students for the exams, especially those taking it more than once, in addition to the cost of the project-based assessments was an estimated $300 million. He argued that the result is essentially an unfunded mandate placed on school districts by the state, which could ultimately lead to higher property taxes (Dinniman 2013). Concern over the Keystone Exams is not limited to the southeastern part of the state. Superintendents in the Lehigh Valley also expressed concern about the exams specifically over the project-based assessments. Overall, school leaders in the region were concerned that the project-based assess- ments have not been clearly explained and would take longer than the eight to ten hours suggested by the state. This would require students to miss class time, staff to be reassigned, and possibly the need for additional staff at a time when local residents are opposed to property tax increases and when state funding has been stagnant (Palochko 2015). Of course, it is unlikely that the Keystone Exams would be met with such vitriol if students were performing admirably on the assessments. However, when the exams were first piloted widely in 2011, only 49.9% of students in the Commonwealth scored at or 36 ADAM McGLynn above the proficient level on the Literature exam, with 38.6% and 35.7% of stu- dents achieving proficiency on the Algebra I and Biology exams, respectively. This has led to a concern from many groups, including the NAACP (Penn- Live Editorial Board 2009) and school superintendents, that “an inevitable outcome of tying Keystone Exams to graduation rates is an increased drop- out rate that will affect a disproportionate amount of low-income and at-risk students than their peers in more affluent communities whose parents and schools have more resources to focus on remediation” (Coalition of Penn- sylvania Public School Superintendents and Intermediate Unit Executive Directors 2013). The Keystone Exams are now beyond the pilot phase, so the practicality of the assessments must be evaluated as we approach full imple- mentation. One of the factors that will be examined in this work is whether socioeconomic status and racial and ethnic minority populations drive Key- stone Exam proficiency rates as feared by these school and community leaders. These concerns are significant as some members of the Pennsylvania General Assembly, while supporting the purpose of the exams, have called for remov- ing the requirement that the exams be passed in order to graduate from high school (Pennsylvania House of Representatives Education Committee 2015). As mentioned in the introduction, the Keystone Exams graduation requirement has been delayed until the 2018–2019 school year, which serves as a significant impetus for this research. The delay appears to be an example of where politics is conflicting with what policy analysts and government agencies believe is the best path forward. In response to the proposed two- year delay, the State Board of Education Chairman Larry Wittig, who helped develop the Keystone Exam system, was skeptical. In comments to Penn Live reporter Jan Murphy (2015), Wittig cited the use of the tests as an evaluation measure for teachers as a reason to oppose the delay, but also said, “This was a very well-crafted system. I don’t think people critical of it fully under- stand the ramifications of the meltdown. . . . In the bottom of my heart, I don’t feel that by eliminating this requirement, it’s doing a positive thing for students” (Murphy 2015). However he also noted that, “If they can come up with something better that achieves the same desired result, I’m all in” (Murphy 2015). It is interesting to note that under the newly passed Every Student Succeeds Act (ESSA), states and schools are no longer mandated to assess teachers using test scores. (The U.S. Department of Education had been granting states waivers from this requirement of No Child Left Behind, in any event.) Therefore, this provides an additional opportunity for poli- cymakers to walk back from the Keystone Exam program if they choose to do so, as they will not have to worry about the pressure to satisfy a federal teacher-evaluation mandate. However, some type of testing must remain— Higher Standards and Lower Achievement? 37 whether it’s the Keystones or something else—as the testing mandate for high school has remained a part of the ESSA. The move to delay the implementation of the graduation requirement and changes to education policy at the federal level provide an opportunity for policymakers to re-evaluate the program to assess not only the appropriate- ness of the learning standards and the exams designed to assess them, but the feasibility of future student success in the Keystone Exam program as currently constructed. An educational-input regression model was estimated to assess the factors that appear most influential in predicting school-level success on the Keystone Exams with the goal of helping to identify how poli- cymakers can allocate resources going forward to achieve the goals of the Keystone Exam program. Data, Methods, and Hypotheses In November of 2015, the Pennsylvania Department of Education released the results of the Keystone Exam administration for the 2014–2015 Academic Year, representing the most recent available data to assess school performance on the exams. These data provide the dependent variables in this study. Over- all, six dependent variables are assessed using Ordinary Least Squares (OLS) regression analysis. Each of the six regression models was estimated using the Stata software program with robust standard errors to correct for the pos- sibility of heteroscedasticity impacting the validity of the regression coeffi- cient estimates. The six dependent variables are the percentage of students achieving proficiency (i.e. passing) the Keystone Exams in Algebra I, Biol- ogy, and Literature in each public high school in the Commonwealth and the percentage of Historically Underperforming (HU) students achieving profi- ciency on the same three exams in each high school in the Commonwealth.1 As explained by the Pennsylvania Department of Education (PDE) (2015), “The Historically Underperforming group consists of students who are: (1) economically disadvantaged, (2) English Language Learners, or (3) have an Individualized Education Plan. If a student is in more than one of those cat- egories, that student is only counted one time.” As the focus of federal educa- tion programs has been to improve the achievement of these HU students, it is important to look at Keystone Exam results for both groups. The use of these data provide an advantage over previous analyses of the Keystone Exam program given that student performance improves over time regardless of student ability as students learn strategies specific to a given assessment. As the Keystone Exams were first field tested in 2009 and 2010, and implemented widely in 2011, students and teachers should have familiarity with the assess- 38 ADAM McGLynn ments at this point. As such, student performance in 2014–2015 should be higher than previous years. The data for the independent variables were obtained from the PDE Fast Facts data set, which provides information on everything from school addresses to demographic data. For this work the following independent variables are included in the models: school type (charter school or tradi- tional public school), the percentage of teachers in a school who are highly qualified, the average number of years of experience of teachers in the school, the percentage of students in a school who are female, economically disad- vantaged, Hispanic, African-American, English Language Learners, identi- fied as gifted, and have an Individualized Education Plan (IEP). Finally, the number of students enrolled in the school and the number of AP courses offered by the school are also included as independent variables.2 The only public schools omitted from the models are those with missing data and the career, vocational, and technical schools in the Commonwealth often referred to as Career and Technical Centers (CTC) or Area Vocational and Techni- cal Schools (AVTS). These schools’ student populations are often composed of students from multiple districts, with many only attending part-time and being assessed by their home districts. This meant that many of these schools had very low testing populations. For these two reasons, they were excluded from the analysis due to concerns that their unique nature could confound the results. After accounting for missing data, 634 high schools in the Com- monwealth formed the basis for this analysis with just under 10% of those schools being charter schools. The purpose of this work is to assess achievement on the Keystone Exams and to help policymakers identify areas of intervention that could help make the program successful. As an alternative, should the results show that achievement gaps based on wealth, race, and ethnicity persist under the Key- stone Exam system, policymakers will need to ask whether the program actu- ally benefits students and provides them with an opportunity to be successful. With this goal in mind, this work will test the following hypotheses: H1: Race, Ethnicity, English Proficiency, Special Education Popula- tions and Socioeconomic Status will impact Keystone Exam pro- ficiency rates negatively in all three subject areas consistent with national performance data on standardized tests (U.S. Depart- ment of Education 2015). H2: Individual school/building level factors such as the gender com- position of the student population, the percentage of gifted stu- dents, teacher experience, and the number of AP courses offered Higher Standards and Lower Achievement? 39 will not impact school-level performance on the Keystone Exams. One would expect that having experienced teachers, many gifted students, and more AP courses offered would likely be an indica- tor of higher proficiency rates on the Keystone Exams; however, there is little variation on the teacher experience and gifted stu- dent variables, and many schools offer AP exams as an indication of rigor, but that does not necessarily mean students are earning AP credit, which is why none of these three variables are hypoth- esized to have an impact. H3: Schools with lower enrollments will have higher proficiency rates on the Keystone Exams given research that has demonstrated that small schools can aid in overcoming the impacts of socioeconom- ic status in educational achievement. (Bickel et al. 2001) Results Table 1 presents the descriptive statistics for the variables that are included in the regression models. These figures demonstrate a wide disparity in student achievement and the demographics of the student population. We see that while in most schools a majority of students are proficient on all three Key- stone Exams, we also see that a majority of the historically underperforming student populations in Pennsylvania high schools do not achieve proficiency in Algebra I and Biology. Also of note is that most schools have highly quali- fied and experienced teachers, which means those factors are unlikely to be demonstrated to be impactful in explaining school-level performance on the Keystone Exams. Finally, unlike the national trend, African-Americans com- prise a much greater proportion of the Pennsylvania student population, with the average high school having an African-American student population of almost 15%, which is more than double the average high school’s Hispanic student population. A few key findings are evident from the regression models in Tables 2 and 3. First, all of the models aid in the explanation of school-level performance on the Keystone Exams although the HU models explain less of the variance in proficiency rates as evidenced by their lower r-squared values.3 Overall, there is mixed support for all three hypotheses.4 The most striking finding is that in all six models, the percentage of students in special education programs, the percentage of African-American students, and the percentage of English Language Learners (ELL) had a negative impact on proficiency rates. These significant variables exhibited the lowest p-values, demonstrating the high likelihood that the relationship between test scores and a school’s special edu- 40 ADAM McGLynn cation population is not due to chance. Not only was the level of significance impressive, but also the magnitude of their impact, especially with the ELL and special education student variables. On average across the six models, a percentage point increase in a school’s special education population would yield a percentage point decrease in proficiency rates, while percentage point increases in the ELL population would on average decrease proficiency rates by approximately seven-tenths of a point. Comparatively, although consistently statistically significant, a percentage point increase in the African-American student population would decrease proficiency rates by two-tenths of a point in high schools in the Commonwealth. Thus while race matters in explaining Keystone Exam proficiency rates, the steepest declines will be observed with increases in a high school’s ELL and Special Education populations. Also significant across multiple models was the percentage of students who were economically disadvantaged, which was found to reduce proficien- cy rates by an average of two-tenths of a point in the total student population models. In an unexpected finding, the percentage of economically disad- Table 1. Descriptive statistics Variable statistic Minimum Maximum AVG Proficiency Rate—Literature 72.12 7.58 100 AVG Proficiency Rate—Algebra I 63.70 3.88 100 AVG Proficiency Rate—Biology 57.50 0 100 AVG Proficiency Rate—Literature (HU Students) 58.18 7.69 100 AVG Proficiency Rate—Algebra I (HU Students) 48.47 4.05 100 AVG Proficiency Rate—Biology (HU Students) 42.64 0 100 Number of Charter Schools & PCT of Sample 63 (9.94%) — — Percent African-American 14.65 0 100 Percent Hispanic 6.75 0 93.59 Percent Economically Disadvantaged 43.81 3.09 100 Percent English Language Learners 1.80 0 46.44 Percent Female 49.00 0 100 Number of AP Courses Offered 8.16 0 34 Percent Gifted Students 4.1 0 51.86 AVG Number of Students Enrolled 892.18 97 9344 Percent Special Education Students 14.96 0 44.67 AVG Number of Years of Teaching Exp. 13.04 1.83 20.42 Percent Highly Qualified Teachers 97.64 0 100 Note: Statistics based on a sample size of 634, except for the HU proficiency rates where N = 620. Higher Standards and Lower Achievement? 41 Table 2. Keystone Exam Regression Model for Total High school student Populations Variable Literature (N = 634) Algebra I (N = 634) Biology (N = 634) Coefficient (S.E.) Coefficient (S.E.) Coefficient (S.E.) Charter School –2.40 (2.94) –8.20 (2.80)** –6.86 (2.48)** Percent African- American –0.15 (0.036)*** –0.218 (0.036)*** –0.255 (0.035)*** Percent Economically Disadvantaged –0.181(0.039)*** –0.205 (0.044)*** –0.342 (0.042)*** Percent English Language Learners –0.903 (0.129)*** –0.643 (0.152)*** –0.641 (0.150)*** Percent Female 0.339 (0.113)** 0.211 (0.113) 0.152 (0.108) Percent Hispanic –0.003 (0.067) –0.054 (0.066) –0.059 (0.063) Number of AP Courses Offered 0.364 (0.076)*** 0.420 (0.086)*** 0.313 (0.083)*** Percent Gifted Students 0.380 (0.107)*** 0.409 (0.128)** 0.299 (0.158) Number of Enrolled Students –0.0007 (0.0006) –0.0015 (0.0006)* –0.001 (0.001) Percent Special Education Students –0.990 (0.125)*** –0.997 (0.108)*** –0.695 (0.093)*** Average Number of Years of Teaching Exp. –0.222 (0.201) –0.286 (0.219) –0.321 (0.216) Percent Highly Qualified Teachers 0.138 (0.074) 0.159 (0.082) 0.087 (0.079) Constant 67.80 (10.32)*** 67.25 (10.45)*** 73.86 (9.89)*** R-squared value 0.656 0.669 0.738 *p < 0.05 **p < 0.01 ***p < 0.001 vantaged students had a small positive impact on the Algebra I proficiency rate but only in the Historically Underperforming student model. This is an anomaly that is difficult to explain but could possibly be attributed to the use of school-level data for this subset of students as discussed previously. Overall, the analysis demonstrates as expected in the first hypothesis that race, socioeconomic status, English language proficiency, and special edu- cation status negatively impact student performance. Also worth noting is that there was an independent impact for race and socioeconomic status. One could posit that the disproportionate poverty African-Americans live in compared to Caucasians (Macartney, et al. 2013) could mitigate the impact of race, and only show an impact for socioeconomic status, but here it is appar- 42 ADAM McGLynn ent that African-American students are struggling to achieve proficiency on the Keystone Exams even with the economically disadvantaged variable in the model. Additionally, the number of AP courses offered in the high school had a significant and positive impact on proficiency levels in all three models for the total student population and for the Literature and Algebra I models among Historically Underperforming students, contradicting the second hypothesis. Each additional AP course offered is estimated to increase proficiency rates by approximately 0.35 points. Further, the percentage of gifted students was found to increase proficiency rates on the Literature and Algebra I Keystone Table 3. Keystone Exam Regression Model for Historically Underperforming student Populations Variable Literature (N = 620) Algebra I (N = 620) Biology (N = 620) Coefficient (S.E.) Coefficient (S.E.) Coefficient (S.E.) Charter School 3.03 (3.27) –3.45 (3.38) –3.10 (3.05) Percent African- American –0.126 (0.039)** –0.194(0.046)*** –0.244 (0.037)*** Percent Economically Disadvantaged 0.069 (0.040) 0.104 (0.051)* –0.051 (0.050) Percent English Language Learners –0.813 (0.139)*** –0.603 (0.140)*** –0.593 (0.137)*** Percent Female 0.414 (0.115) 0.332 (0.119)** 0.168 (0.115) Percent Hispanic –0.026 (0.072) –0.079 (0.066) –0.092 (0.065) Number of AP Courses Offered 0.332 (0.102)** 0.354 (0.113)** 0.179 (0.109) Percent Gifted Students 0.610 (0.125)*** 0.574 (0.292)*** 0.390 (0.248) Number of Students Enrolled –0.001 (0.0007)* –0.002 (0.0001)** –0.001 (0.001) Percent Special Education Students –1.14 (0.124)*** –1.10 (0.127)*** –0.832 (0.108)*** Average Number of Years of Teaching Exp. –0.281 (0.252) –0.247 (0.284) –0.443 (0.271) Percent Highly Qualified Teachers 0.120 (0.075) 0.142 (0.084) 0.093 (0.074) Constant 43.05 (10.30)*** 34.65 (11.57)** 49.24 (10.09)*** R-squared value 0.400 0.391 0.476 p < 0.05 **p < 0.01 ***p < 0.001 Higher Standards and Lower Achievement? 43 Exams: A percentage point increase in the gifted population increases pro- ficiency rates by approximately one-half of a percentage point. The findings related to AP course offerings and the gifted student population contradict the second hypothesis and will be examined in the discussion section. The impact of charter schools was limited. Only in the Algebra I and Biology models for all students in a school did being a charter school have a statisti- cally significant impact, and in those instances charter schools had profi- ciency rates that were eight points lower in Algebra I and seven points lower in Biology. The results also show that percentage point increases in the female population of a high school would on average increase proficiency rates on the Literature Keystone Exam (total student population model) and Algebra I Keystone exam (HU student model) by three-tenths of a point. And finally, smaller schools are found to have a positive impact on proficiency rates for the Algebra I exam in both models, and the Literature exam in the total stu- dent population model, supporting the third hypothesis. However, it should be noted that the magnitude of the impact of smaller schools is low. Just as important as the findings are the non-findings. In all six models, Hispanic ethnicity, teaching experience, and the percentage of teachers who were highly qualified proved statistically insignificant. The insignificance of the teacher variables was expected and supports the second hypothesis, but the findings related to ethnicity are contrary to hypothesis one. Therefore, despite national test results showing ethnicity negatively impacting student achievement, the results here do not support those findings. While it is dif- ficult to offer any definitive explanations, the smaller overall size of the His- panic population in Commonwealth high schools could be the cause of the observed result. Discussion The results presented here do provide insight into how the Commonwealth can save the Keystone Exam program and achieve its original goals of ensur- ing that students are achieving higher academic standards. However, this work does not allow for a specific diagnosis of the cause of the achievement gap that is confirmed by this work. There are myriad possible reasons for these findings, including test bias, inadequate funding in minority and low-income schools, lower standards and expectations in the classrooms of historically underperforming students, and concerns that out of school experiences can negate what goes on in the classroom. These reasons are all plausible. Howev- er, school finances should be the first issue addressed by policymakers. First, the literature on test bias is controversial, with psychometricians appearing 44 ADAM McGLynn to believe that mean differences in group test scores are unlikely due to bias (Warne et al. 2014). Also, the adoption of the Common Core standards and recent changes to teacher evaluation systems in the Commonwealth (the result of the waiver Pennsylvania received from the requirements of NCLB) show an attempt to address the possibility that teachers expect less from historically underperforming groups, which a 2014 report from the Center for American Progress found to be the case. Finally, there is a large body of research explain- ing the impact of outside factors on educational achievement, especially pov- erty (Jensen 2009), but policymakers are limited in their ability to address such problems due to both budgetary and public opinion constraints. These issues deserve much greater discussion than provided here, but for the reasons stated above and because a recent study by the Education Trust (2015) found Pennsylvania to have the third worst funding gap between low poverty and high poverty districts in the country, school finances will be the focus as one possible solution to the Keystone Exam achievement gap. First and foremost, the Commonwealth must find ways to provide more services and resources to its special education and English Language Learner student populations. For most schools, the special education population will be of greater concern given its size relative to the ELL population. Just as its predecessor No Child Left Behind did, the new Every Student Succeeds Act requires schools to assess 95% of its student population, and because most schools have 15% of their student population or more requiring special education services, at least two-thirds of that group will need to be tested to meet the requirement. In recent years Pennsylvania revised its special educa- tion funding formula, which now accounts for not only student headcounts but also employs weighted cost categories to ensure more funding goes to districts with students who cost the most to educate (Commonwealth of Pennsylvania General Assembly 2013). However, schools continue to struggle with the costs of special education services as school funding has remained relatively flat in Pennsylvania, and the revised formula is being applied only to small increments of additional annual funding. As well, it has been found that school districts end up overpaying charter schools to educate special education students who live in their district based on current law. The Penn- sylvania Association of School Business Officials (PASBO) estimates this overpayment to be $200 million a year (PASBO 2014). This problem is likely to worsen given the rising number of students that are classified as being on the Autism spectrum, which entitles them to special education services. Thus of fundamental importance to raising Keystone Exam scores will be a comprehensive restructuring of how special education students are funded and serviced in the Commonwealth. Higher Standards and Lower Achievement? 45 While it is easier to diagnose and formulate plans regarding special edu- cation funding and services, policies that could successfully address racial and economic disparities remain elusive. Many reform advocates have campaigned for more charter schools to address racial and socioeconomic achievement gaps, and while several are operating within the Commonwealth in areas with low-income, minority populations, the results here show that charter schools do not perform better than traditional public schools; in the case of the Algebra I and Biology exams, they perform significantly worse. Thus, the Commonwealth’s main objective should be to continue inter- ventions in the communities and schools with large low-income and racial minority populations. This starts with greater resources for programs such as early childhood education, which has been one of the few interventions to demonstrate lasting positive results (Karoly et al. 2005). While it is not fea- sible to attempt to place more gifted students in high schools that lack such students, it is possible to help increase standards by providing the funds to offer more AP courses. At the start of any such initiative in schools with low proficiency rates, few students are likely to earn college credit through these courses. However, exposure to this advanced material will better prepare stu- dents for higher education, even if it does not immediately result in college credit. Lastly, it sends the message to students in these high schools that fac- ulty and administrators believe they are capable of achieving more. Districts could also consider having students spend more time on their schooling with after-school and possibly weekend enrichment programs. Parents and com- munities should also be incorporated more into the education process. These are interventions that have also proven successful (UCLA Center for Men- tal Health in Schools 2007) as the amount of time students spend outside of school dwarfs the time spent inside the classroom. Out-of-classroom time sometimes undoes what was achieved in the classroom. The Pennsylvania General Assembly and the governor are currently at odds over the Commonwealth budget; an impasse that has lasted over eight months as of this writing. A principal area of disagreement is revenue genera- tion to be used for increasing funding to public schools. Based on the results of this paper, while one can posit that money is not everything, supplying more resources for Pennsylvania schools is vital at this time. In theory, the Keystone Exams are a worthwhile program as they help to ensure higher standards are achieved while better aligning curriculum with assessment, something that was missing at the high school level under the PSSAs. However, to close with an analogy, if one can high jump five feet, one can’t raise the bar six inches and expect increased effort to immediately result in the achievement of the new standard. Just as an athlete would need to train to meet the new standard, 46 ADAM McGLynn schools and students, especially in areas with large low-income, ELL, special education, and minority student populations, need the time and resources to be able to better prepare for these assessments. As discussed in the introduc- tion, the Keystone Exam program has worthwhile benefits, however, like the recently ended No Child Left Behind legislation, those left with implementing the program are struggling without the resources needed to achieve its goals, and without significant changes in Harrisburg, the program is likely to see achievement gaps persist as evidenced in this work. NOTES 1. Individual student data would be ideal for the purposes of this paper, but at present no individual-level data have been published by the Pennsylvania Department of Educa- tion on its website, which is the reason that school-level data are used. 2. One obvious omission here is the absence of a variable measuring per-pupil expen- ditures. Data for this variable are only available at the district/Local Education Agency Level. While there is likely to be a correlation between overall spending and spending at the high school, there are several factors that could impact per-pupil spending in each school such as the technology available in and the special education populations of each school. As such, the decision was made to only include variables measured at the school/ building level. 3. The HU population models continue to use data at the school level for the indepen- dent variables while only examining proficiency for a subset of the school’s population. This could account for the less robust results for the HU models. This is not ideal, however given that different variables could be found to drive proficiency rates among the full school and the HU subset, the HU models were included. 4. Please note that while there are correlations between some of the variables, e.g., the percentage of economically disadvantaged students and the percentage of African- American students, an examination of Variance Inflation Factors (VIF) did not show evidence of multicollinearity. REFERENCES Bickel, Robert, Craig Howley, Tony Williams, and Catherine H. Glascock. 2001. “High School Size, Achievement Equity, and Cost: Robust Interaction Effects and Tentative Results.” Education Policy Analysis Archives 9 (40). http://epaa.asu.edu/ojs/article/ view/369 (March 13, 2016). Bishop, John. H., Ferran Mane, Michael Bishop, and Joan Moriarty. 2000. “The Role of End-of-Course Exams and Minimum Competency Exams in Standards-Based Re- forms.” July 1. (CAHRS Working Paper #00-9). Ithaca, NY: Cornell University, School of Industrial and Labor Relations, Center for Advanced Human Resource Studies. http://digitalcommons.ilr.cornell.edu/cahrswp/88 (September 15, 2015). Center for American Progress. 2014. “RELEASE: Teachers Have Lower Expectations for Students of Color and Students from Low-Income Backgrounds.” October 6. https:// www.americanprogress.org/press/release/2014/10/06/98317/release-teachers-have -lower-expectations-for-students-of-color-and-students-from-low-income-back grounds/ (February 25, 2016). Higher Standards and Lower Achievement? 47 Coalition of Pennsylvania Superintendents and Intermediate Unit Executive Direc- tors. 2013. “Letter to Senator Andrew Dinniman.” November 5. http://www.senator dinniman.com/wp-content/uploads/2013/11/1KeystonesPositionPaper.pdf (Novem- ber 5, 2015). Commonwealth of Pennsylvania. 2013. Pennsylvania General Assembly. “Special Edu- cation Funding Commission Report.” December 11. http://archive.pasbo.org/Spe cial%20Ed%20Commission%20FINAL%20REPORT.pdf (December 8, 2015). ———. 2015. House of Representatives Education Committee. Presentation on PA State Assessments. July 29. DeBray, Elizabeth. 2004. “Richard Mills and the New York State Board of Regents, 1995– 2001.” Journal of Cases in Educational Leadership 7 (2): 27–43. Dinniman, Andrew E. 2013. “8 Reasons Why We Oppose the Keystone Exams. Minor- ity Chair, Senate Education Committee.” http://www.senatordinniman.com/wp-con tent/uploads/2013/11/EightReasonsAgainstKeystones.pdf (November 5, 2015). Education Trust. 2015. “Funding Gaps 2015.” March 25. https://edtrust.org/resource /funding-gaps-2015/ (February 25, 2016). Grant, S.G. 2000. “Teachers and Tests: Exploring Teachers’ Perceptions of Changes in the New York State Testing Program.” Education Policy Analysis Archives 8 (14). http:// epaa.asu.edu/ojs/article/view/405/528 (November 5, 2015). Isaacs, Tina. 2014. “150 Years of Statewide Assessment in New York: Are Regents Exams Still Fit for a Purpose?” Assessment in Education: Principles, Policy and Practice 21 (3): 344–357. Jensen, Eric. 2009. Teaching with Poverty in Mind: What Being Poor Does to Kids’ Brains and What Schools Can Do about It. Alexandria, VA: Association for Supervision and Curriculum Development. Karoly, Lynn, M., Rebecca Kilburn, and Jill S. Cannon. 2005. “Proven Benefits of Early Childhood Interventions.” RAND Corporation. http://www.rand.org/pubs/research_ briefs/RB9145.html (December 1, 2015). Macartney, Suzanne, Alemayehu Bishaw, and Kayla Fontenot. 2013. “Poverty Rates for Se- lected Detailed Race and Hispanic Groups by State and Place: 2007–2011.” American Community Survey Briefs. Washington, DC: U.S. Census Bureau. https://www.census .gov/prod/2013pubs/acsbr11-17.pdf (February 29, 2016). Murphy, Jan. 2015. “Graduation Testing Requirement Delay Wins House Passage.” Penn- Live, November 24. www.pennlive.com/politics/index.ssf/2015/11/graduation_test ing_requirement_1.html (November 24, 2015). ———. 2016. “Wolf Signs Bill to Suspend Use of Keystone Exams as a Graduation Require- ment.” PennLive, February 3. http://www.pennlive.com/politics/index.ssf/2016/02 /wolf_signs_bill_to_suspend_use.html (February 26, 2016). National Commission on Excellence in Education. 1983. “A Nation at Risk: The Impera- tive for Education Reform.” April. http://www2.ed.gov/pubs/NatAtRisk/index.html (October 29, 2015). New York State Education Department. Board of Regents. 2011. Potential Revision of High School Graduation Requirements. March 11. Albany, NY: NYSED. http://www .regents.nysed.gov/common/regents/files/documents/meetings/2011Meetings/March 2011/311p12ccrgradreqpresentation.pdf (November 5, 2015). Palochko, Jacqueline. 2015. “Keystone Exams One Big Headache for Lehigh Valley School Districts.” The Morning Call, May 10. http://www.mcall.com/news/local/mc-lehigh -valley-keystone-project-based-assessments-20150509-story.html (November 5, 2015). PennLive Editorial Board. 2009. “Important Questions Remain about Keystone Exams.” 48 ADAM McGLynn PennLive, August 30. http://www.pennlive.com/editorials/index.ssf/2009/08/impor tant_questions_remain_abo.html (October 29, 2015). Pennsylvania Association of School Business Officials. 2014. “Misleading Claims by Charter School Advocates Must Not Be Permitted to Stall Needed Special Education Reform.” May 13. http://archive.pasbo.org/Special%20Education%20Funding%20 Press%20Release.pdf (December 8, 2015). Pennsylvania Department of Education. 2014. Keystone Exams: Technical Report. Harris- burg, PA: Department of Education. http://www.education.pa.gov/Documents/K-12 /Assessment%20and%20Accountability/Keystone%20Exams/Technical%20 Reports/2014%20Keystone%20Technical%20Report.pdf (October 1, 2015). ———. 2015. 2015 Keystone Exam School Level Data. Harrisburg, PA: Department of Edu- cation. http://paschoolperformance.org/Downloads (November 18, 2015). UCLA Center for Mental Health in Schools. 2007. “After-School Programs and Address- ing Barriers to Learning.” http://www.isbe.net/learningsupports/pdfs/after-school -prog.pdf (December 10, 2015). U.S. Department of Education. 2015. National Assessment of Educational Progress (NAEP). Washington, DC: Department of Education. http://nces.ed.gov/nationsreportcard/ (November 29, 2015). Warne, Russell T., Myeongsun Yoon, and Chris J. Price. 2014. “Exploring the Various Interpretations of ‘Test Bias.’” Cultural Diversity and Ethnic Minority Psychology 20 (4): 570–582.