CIE Template Current Issues in Education, 18(3) 1 Volume 18, Issue 3 October 6, 2015 ISSN 1099-839X Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Gary Storie and Peter R. Denner Idaho State University Abstract: In the fall of 2012, Idaho implemented a plan to award bonus pay to schools whose students demonstrated academic growth based on the Betebenner (2008) method. This study examined the relationship of the amount of bonus paid to a school, the percentage of students from low income families associated with a school, and the location of the school (urban, suburban, town, or rural). Using hierarchical set regression, a statistically significant negative relationship was found between the percentage of students eligible for subsidized meals and per pupil school bonus pay. When school location was added as a predictor, rural school location emerged as a positive predictor of the amount of per pupil bonus money received by a school. The percentage of students eligible for subsidized meals also predicted whether a school received any bonus money. Keywords: Merit pay, pay for performance, accountability APA-Style Citation: Storie, G., & Denner, P. R. (2015). Using student test scores to award merit pay: A look at the 2012 pay-for- performance program for Idaho schools. Current Issues in Education, 18(3). Retrieved from http://cie.asu.edu/ojs/index.php/cieatasu/article/view/1373 Accepted: July 15 th , 2015 Can the solitary influence of a teacher or school on student learning truly be measured? The answer appears to be “yes,” according to the American Recovery and Reinvestment Act and President Obama’s Race-To-The-Top initiative. The affirmative answer is so compelling that guidelines for states to apply for federal money required teachers and schools to be accountable for student learning, and to have that accountability measured by evaluation models that in whole, or in part, use student test results to judge whether or not teachers and schools are effective (U. S. Department of Education, 2010). Idaho initiated such a plan in 2012 and provided bonus pay to schools that demonstrated growth in student achievement scores. The concept of paying teachers or schools more based on student test scores is highly contested in educational research with few studies touting the value of it. Certainly the idea is to motivate teachers to work harder and ensure their students achieve higher scores on tests, but researchers are finding a significant number of flaws regarding the practice (Alicias, 2005; Amrein-Beardsley, 2014; Berliner, 2014; Kupermintz, 2003). Among the criticisms, the http://cie.asu.edu/ojs/index.php/cieatasu/article/view/1373 Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 2 strongest may be the inability to control for all the factors that contribute to student learning besides what the teacher does. With this in mind, we conducted this study to consider the amount of the bonus pay Idaho schools received in 2012 in relation to some of these questionable variables, specifically socio-economic status and school location. Students from homes with low family income generally achieve at a lower rate when compared to students from more affluent backgrounds (Coleman et al., 1966; Sum & Fogg, 1991). We looked specifically at this connection and asked the basic question: Was there a relationship between the amount of bonus pay a school received and the percentage of students from low-income families who attended that school? Other research has shown school location has an effect on student learning (Hopkins 2005; Plecki, 1991; Zoda, Combs, & Slate, 2011). This factor was considered as well. The biggest challenge to establishing merit pay for teachers or schools based on student test scores has been accounting for all the variables that affect student learning. Generally, the simplistic concept of merit pay involves whether or not students improve their scores from one year to the next, and if they do, then bonuses ought to be awarded. However, to define this academic growth and the factors that may, or may not, have contributed to it has proven to be elusive. Value-Added Models (VAM) and Student Growth Models (SGM) State and federal education policy makers who advocate providing bonus pay for higher test scores consider student learning to be significantly influenced by the teacher or a collective of teachers in a school setting. This sentiment may be drawn from research sponsored by The New Teacher Project (TNTP) – an analysis of survey data that suggest current teacher evaluation models are inconsequential or irrelevant, do not address teaching deficiencies, nor recognize and reward instructional effectiveness; there was no tangible incentive for good teaching or reasons to retain them, especially in urban settings (The New Teacher Project, 2010). A 2009 study found current performance assessments of teachers fail to distinguish the good from the bad, 99% of teachers receive a satisfactory rating from their supervisors, and poor teachers were not removed from the classrooms (Weisberg, Sexton, Mulhern, & Keeling, 2009). The need for objective measures of teacher and school performance in the light of student learning became a central focus of TNTP and has been embraced by education officials in high levels of both state and federal offices. As a result, the establishment of teacher evaluation models that can to be tied to achievement based on improved student learning is a condition for states to receive additional dollars under Obama’s Race-to-the-Top program (U.S. Department of Education, 2010). As a consequence, and not surprisingly, the concept has garnered favor and/or practice in 44 states (Amrein-Beardsley, 2014). Though research has shown the competency and relational skills of a classroom teacher have a significant relationship with student achievement (Heck, 2009; The New Teacher Project, 2012; Zhou, 2012), there are many other variables associated with learning that are well out of the teacher’s or school’s control, some of which may be serious impediments for even the most able educator to overcome. This fact brings into question the equitable practice of evaluating teachers and schools on the basis of student test scores and using the scores to make decisions about teacher tenure, compensation, and school performance rating and labeling. To address questions about fairness, statisticians sought to isolate the teacher or school effect on learning from other variables, devising what are known as value-added models (VAM) of teacher evaluation. These models existed in the past yet were sparsely used. Berk (1984) reported such policies were in place in nine school districts in seven states during the early Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 3 1980s. Now the majority of states have one form or another of teacher evaluation programs that can be considered value-added models. The Tennessee Value Added Assessment System (TVAAS) gained prominence in the late 1990s, and then gave rise to the Education Value-Added Assessment System (EVAAS) a decade later. The EVAAS was originally used in Texas and subsequently spread to other states. TVAAS developer, William Sanders, a statistician from the University of Tennessee, reported that his model met its objectives; he claimed test score gains by students were attributable to the teacher and not to the students’ abilities or achievement levels (Sanders & Horn, 1994). The Tennessee State Board of Education was so convinced of the model it now relies on the results from its statewide testing program to make consequential decisions about teachers based on whether or not their students make gains in test scores. Teachers in Tennessee may be paid bonuses if student test scores improve over time, or they might lose their licenses if scores decline (Hardy & Sher, 2013; Tennessee State Board of Education, 2013). Both TVASS and EVAAS and value added models in general have been criticized in recent years. Kupermintz (2003) claimed he found flaws in Sanders’ approach and cautioned education policy makers about relying too heavily on TVAAS or similar value-added teacher evaluation models; he doubted the statistical ability to calculate a pure teacher effect on learning. Alicias (2005) challenged the TVAAS and other value-added models on similar grounds, claiming it is impossible to eliminate all the possible interactions and effects of other stimuli that influence a student’s performance on a test. Amrein-Beardsley (2014) questioned the validity of such instruments, asserting that other measures of teacher performance (i.e., classroom observation, student and parent surveys) do not correlate with VAMs in general. She also found gross instabilities in individual teacher’s VAM scores on EVAAS over a course of four years, which challenged the reliability of using this method to discern a true teacher effect on learning (Amrein-Beardsley, 2014). Berliner (2014) claimed various classroom, school, and home characteristics and the interactions between them are likely among the strongest influences on student learning yet not always accounted for in value added assessment models. A student growth model (SGM) is another method that relies on student test scores to evaluate teacher and school performance. Although not claimed to be VAMs per se, in actuality SGMs seek to do something very similar – attribute the growth of student learning, or lack of it, to the teacher or school. Betebenner (2008, 2011) developed a student growth model he believed was an alternative approach to VAMs and growth to standard school models of accountability. His approach employed a “growth percentile methodology" to capture growth over time in student achievement (Betebenner, 2008, p. 2). Growth percentile methodology looked at learning growth normatively using "quantile regression techniques" (p. 12). Regarding such growth calculations, Betebenner (2008) wrote, If the student’s current year score exceeded the scores of most of their academic peers, in a normative sense they have done well. If the student’s current year score was less than the scores of their peers, in a normative sense they have not done well. (p. 4) In essence, by comparing students’ growth results on tests to the growth results of students with similar profiles, the Betebenner (2008) model could “side step many of the thorny questions of causal attributions, and instead provide descriptions of student growth that have the ability to inform discussion about assessment outcomes and their relation to education quality” (p. 2). Betebenner (2008) said his approach to gauging student academic growth was not a value-added model because it did not specifically try to isolate the teacher or school effects on Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 4 learning from other variables. However, by aggregating data suitable for the model at the school or classroom level, inferences could still be made about student growth pertinent to that setting. In other words, a teacher or school could be judged by whether or not students demonstrated growth comparable to the growth of similar students elsewhere. The 2012 pay-for-performance program in Idaho used Betebenner’s (2008, 2011) growth percentile methodology to determine growth in student test scores compared to that of peers. Quartiles were established, and bonuses were awarded to schools whose scores fell above the first quartile (25 th percentile). Bonuses were paid to schools rather than to individual teachers, but school boards and administrators had discretion on how to disperse the money to teachers and staff. About 75% of the schools in Idaho received bonus pay money. Other Factors that Influence Student Learning Education research is rife with studies on the dynamics of student learning and factors that influence it. Among the most noteworthy of these factors is the socio-economic status of students. This fact became especially important after a study by Samuel Coleman and others in 1966. He found that students from economically disadvantaged homes were likely to be less academically proficient than their more affluent peers (Coleman et al., 1966). Sum and Fogg (1991) found that students from low-income homes generally ranked in the 19 th percentile on achievement tests. Sirin (2005) conducted a meta-analysis of the research published between 1990 and 2000 on the topic of the relationship between students’ socio-economic status and achievement. He examined the differences between the studies including the specific measures for achievement and poverty, statistical methodologies, and data sources. Among all the studies he discerned correlations ranging from .005 to .77, with a mean of .29. Sirin (2005) reported “a medium level of association between SES and academic achievement at the student level and a large degree of association at the school level” (p. 438). The stronger relationship found at the school level is especially pertinent to this study. Other studies suggest the effect of poverty on student learning is somewhat mitigated in smaller schools. After a review of the literature on school size and student achievement, Zoda et al. (2011) reported that poor students in small schools do better than poor students in larger school settings. Howley and Bickel (1999) found smaller schools in Ohio, Georgia, and Texas seem better able to accommodate the unique needs of disadvantaged and impoverished students, thereby enabling them to achieve a degree of academic success. Plecki (1991) found a negative correlation between school size and achievement in a study pertaining to elementary students in California. The mitigating effect of school size for students from poverty seemed to vanish in schools larger than 760 students (Borland & Howsen, 2003). Since many small schools exist in remote rural locations, there may be a relationship between student learning and the place in which a school is located. Alspaugh (1992) examined fifth grade reading and math scores for rural and urban schools; although no statistical difference was noted among aggregate scores, when controlling for low family income (as indicated by whether or not student qualified for government subsidized meals), he found location was the best predictor for urban students doing poorly in reading. Fan and Chen (1999) collected data on 24,500 students from across the country over a period of six years and found that students in rural schools did as well in the subjects tested as their peers in other settings. However, family income was not a factor in their analysis. When taking family income into account, Hopkins (2005) discovered poor students in rural Tennessee schools outperformed their counterparts in Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 5 large cities and in other non-rural locations on a subset of the Mathematics ACT, and on test scores from the Tennessee Comprehensive Assessment Program (TCAP). Idaho Bonus Pay Study Concerned that VAMs and SGMs may not always account for variables that contribute to student learning (other than the effects of the teacher and school), this study looked at the relationship between the amount of bonus pay received by schools in Idaho and the following: low family income (as indicated by eligibility for free and reduced priced school meals), school size, and school location. We believed these variables were well established in the literature as influences that may be beyond the control of statistical calculations embedded within current teacher performance programs that attempt to measure teacher or school effectiveness based solely on student test scores. Though VAMs have received considerable scrutiny in the literature recently, SGMs, such as Betebenner’s (2012) model used in Idaho in 2012, have not been as well examined (Amrein-Beardsley, 2014). Our study looked at the fairness of the school level bonus pay distribution model applied based on Betebenner (2012) methodology for evaluating student achievement. This study examined whether distribution of the bonus money to schools was affected by variables over and beyond the effect the school had on student learning. These variables included the percentage of students from low-income families, as indicated by their eligibility for subsidized meals, who attended the school, and school location. Because the bonus distribution model was expected to distribute more bonus pay to larger schools with more students, the effect of school size was controlled by looking at the amount of bonus money distributed per pupil. If either the percentage of students from low income families or the location of the school accounted for differences in the per pupil amount of bonus money to the schools, even if the differences were small, then the amount of bonus money awarded could not be attributed entirely to the performance of the teachers and the school. Research Questions The basic research questions for this study were the following:  Was the per pupil amount of bonus pay distributed to schools in Idaho during the fall of 2012 related to the percentage of students from low income families in the school buildings as measured by the percentage of students eligible for subsidized meals?  Was there any additional effect on the amount of per pupil bonus pay money awarded to a school because of the school location (urban, suburban, town or rural), when controlling for the effect of the percentage of students eligible for subsidized meals?  Did the percentage of students eligible for subsidized meals predict whether schools received any bonus money?  Did school location predict whether schools received any bonus money? Method All the data used in this study were obtained from the Idaho State Department of Education (2013) website and were available in the public domain. Data collection included the amount of 2012 bonus pay received by each school in the state, the enrollment of each school, and the percentage of students receiving subsidized meals. Schools were assigned a location code based on the definition established by the Institute of Education Sciences National Center for Education Statistics. For equivalent comparisons between large and small schools, the amount of bonus pay each building received was divided by its enrollment to yield a per pupil amount of bonus pay. In the 2012 school year, there were 704 schools in Idaho (Idaho State Department of Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 6 Education, 2013). Information was available for 627 (89%) schools. Of the 627 schools, 449 (72%) received some amount of bonus pay, 178 (28%) schools received no bonus pay. For this study, we used hierarchical set regression to examine the relationship between two factors, the percentage of students receiving subsidized meals and school location, and the amount of per pupil bonus money awarded to schools. The percentage of students receiving subsidized meals was the predictor variable entered in the first set. Controlling for the percentage of students receiving subsidized meals, school location variables (urban, town, and rural) were added as predictors in the second set using dummy coding. The second set was analyzed using stepwise selection. To avoid multicollinearity, the category of suburban school served as the reference category. The suburban school category was chosen as the reference category because it contained the fewest number of schools in Idaho. Separate binary logistic regression analyses were conducted to determine whether either the percentage of students receiving subsidized meals or school location were related to whether the schools received any bonus pay at all (0 = no bonus pay money, 1 = bonus pay money). All tests for statistical significance were conducted at  = .05. Results We examined the effects of the percentage of students eligible for subsidized meals and school location on the amount of per pupil bonus pay money received by schools in Idaho in 2012 based on student achievement levels. The mean per pupil amount of bonus pay distributed to the schools was M = $163.14 (SD = $133.78). Table 1 presents the means and standard deviations for school location category, percentage of students receiving subsidized meals, and the amount of per pupil bonus money paid to schools. Table 1 also displays the correlations of the location categories with percentage of students receiving subsidized meals and the amount of per pupil bonus money as well as the correlation of the percent of students receiving subsidized meals with the amount of per pupil bonus money paid to schools. As can be seen in Table 1, there was a statistically significant relationship found between the percent of students receiving subsidized meals and the amount of per pupil bonus money paid to the schools. Table 1 Means, Standard Deviations, and Selected Correlations (N = 627) for School Location Categories, Percentage of Students Receiving Subsidized Meals, and the Per Pupil Bonus Pay Money Awarded to Schools Measure M SD % Students Receiving Subsidized Meals Per Pupil Bonus Money Urban Location 0.21 0.41 .05 .01 Suburban Location 0.11 0.32 -.07 -.09 * Town Location 0.21 0.41 .09 * -.04 Rural Location 0.47 0.50 .01 .08 * % Students Receiving Subsidized Meals 54.48 18.63 - -.25 * Per Pupil Bonus Money $163.14 133.78 -.25 * - * p < .05 Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 7 Per Pupil Bonus Money Paid to Schools We examined the influence of the percentage of students receiving subsidized meals and school location on the per pupil amount of bonus money paid to schools using hierarchical regression. Although preliminary inspection indicated the presence of outliers, as might be expected given the fact that some schools received no bonus money, inspection of a plot of the residual errors revealed the errors had an approximate normal distribution. Table 2 presents the results of the hierarchical set regression. At step 1, the percentage of students receiving subsidized meals was found to be a statistically significant negative predictor of the amount of bonus pay received per pupil by the schools, b = -1.81, t = -6.51, SE = .28, p < .001. The correlation was R = .25, indicating the percentage of students receiving subsidized meals accounted for about 6% of the variance in the per pupil amount of bonus pay received by the schools. The results show that for every 1% increase in the percentage of students in the school receiving subsidized meals the amount of per pupil bonus pay declined by $1.80 per pupil. At step 2 of the hierarchical set regression that controlled for the effect of the percentage of students receiving subsidized meals, school location codes (urban, town, and rural) were entered as a second set of predictors using stepwise selection. At this second step, rural location was found to be the only additional statistically significant predictor of per pupil bonus pay when controlling for the percentage of the students receiving subsidized meals, b = 21.50, t(624) = - 2.08, SE = 10.34, p = .038. Rural school location was found to be a positive predictor when the percentage of students receiving subsidized meals was held constant. Although statistically significant, as can be seen in Table 2, the final multiple correlation was not much larger, R = .26 (R 2 = .07). Table 2 The relationship between percentage of students receiving subsidized meals in a school and the amount of bonus pay money it received Predictor R  B 95% CI for B Step 1 .25 * Percent of Students with Subsidized Meals -0.25 * -1.80 * [-2.35, -1.25] Step 2 .26 * Percent of Students with Subsidized Meals -0.25 * -1.81 * [-2.35, -1.26] Rural School Location 0.08 * 21.50 * [1.19, 41.81] Note: N = 627. CI = confidence interval. * p < .05. The final model accounted for 7% of the total variance in the amount of per pupil bonus pay received by the schools. Nevertheless, both the percentage of the students receiving subsidized meals and rural school location were shown to have a significant influence on the distribution of per pupil bonus money to the schools. The regression coefficient revealed that rural location increased the amount of bonus pay distributed to rural schools by $21.50 per pupil Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 8 when controlling for the percentage of students receiving subsidized meals. As shown in Table 1, urban and town locations were not significantly correlated with the amount of per pupil bonus pay, and they did not emerge as statistically significant predictors of the amount of per pupil bonus money when controlling for the percentage of students receiving subsidized meals. Table 3 Subsidized Meals as a Predictor of School Receiving any Bonus Pay Money Model B SE-B Wald df Exp(B) 95% CI Exp(B) Intercept 3.40 0.37 83.04 * 1 Percentage of Students Receiving Subsidized Meals -.04 0.01 41.26 * 1 0.96 [0.95, 0.97] Note. Receiving bonus pay money was the target outcome group. * p < .05. Table 4 Probability of a school receiving any amount of bonus pay based on the percentage of students receiving subsidized meals Percentage of students receiving subsidized meals Predicted probability of receiving any amount of bonus money 10 .95 20 .93 30 .90 40 .87 50 .82 60 .75 70 .68 80 .58 90 .49 Any Bonus Money Paid to Schools Binary logistic regression was used to determine the probability of whether or not a school would get any amount of bonus pay at all based on its percentage of students receiving subsidized meals. The binary logistic regression was statistically significant, -2 Log Likelihood = 631.01,  2 (1, N = 627) = 47.15, p < .001. The Cox and Snell pseudo R 2 = .07, indicating the model accounted for 7% of the total variance in the awarding of bonus pay to schools. This result was similar to the finding for the per pupil amount of bonus money. Table 3 presents the binary logistic regression coefficients, the Wald test, the odds ratio, and 95% confidence interval for the Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 9 odds ratio. Based on a classification criterion for success of .50 or higher for predicted probabilities, the correct prediction rate was 99.2% for schools receiving bonus pay, but only 4.8% for schools not receiving bonus pay. Table 4 shows the predicted probability of a school receiving bonus pay increased as its percentage of students receiving subsidized meals declined. A separate Binary logistic regression was used to determine the probability of whether or not a school would get any amount of bonus pay at all based on its location. The overall binary logistic regression was not statistically significant, -2 Log Likelihood = 671.61,  2 (3, N = 627) = 6.55, p = .09, Cox and Snell pseudo R 2 = .01. This means school location was not a factor in predicting whether or not a school received any bonus pay. Discussion Our study looked at whether the 2012 Idaho performance pay model that distributed bonus money to schools based on student achievment was related to the percentage of students who were eligible for subsidized meals and the school location. The Idaho State Department of Education used Betebenner’s (2008, 2011) growth percentile methodology for determining student achievement growth and subsequently the amount of bonus pay awarded to schools. Betebenner (2008, 2011) does not describe his model as value-added, a concept that purports to attribute student learning to only the teacher. Rather, his model uses a normative student growth percentile methodology that sorts students into stratified groups with similar characteristics and profiles, thereby mitigating concern about other variables associated with student achievement. Thus, schools were awarded bonus pay money based on the extent to which their students exceeded the normative expectations for their respective groups. The results of the study indicated the 2012 distribution of bonus money to schools in Idaho was not entirely successful because it was somewhat related to variables beyond the control of the schools. The results of the study also add to the current literature regarding problems with statistical formulas that attempt to discern the teacher or school effect on learning, specifically with Betebenner’s (2008, 2011) student growth model (SGM). Though Value Added Models have received notable investigations recently, SGM’s have not (Amrein- Beardsley, 2014). Our findings showed the percentage of students eligible for subsidized meals influenced the odds of a school receiving bonus money and was a significant negative predictor of the amount of per pupil bonus pay money received by a school. The higher the percentage of students from low-income families, the smaller was the likelihood of a school receiving bonus money. With regard to the per pupil amount of bonus pay received by schools in Idaho, this study showed about 6% of the variance was due to the percentage of students from low-income families in the school building. One might argue that this is a small enough difference to be unimportant. In this light, perhaps the distribution of bonus pay based on Betebenner’s (2008, 2011) methodology worked. However, there was an unexpected predictive relationship between the percentage of students from low-income families and the amount of per pupil bonus pay awarded to a school. This finding suggests that the bonus pay distribution methodology did not entirely side step student achievement causal attribution issues. The results of our study also showed that when controlling for the percentage of students eligible for subsidized meals, the distribution of bonus money favored rural schools. Rural location did contribute to the distribution of bonus money when it was not expected to be an influential factor at all, given that achievement growth was supposedly benchmarked against like students. Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 10 Together the results suggest the distribution of bonus pay money was influenced partly by the percentage of students on subsidized meals and school location, and was not entirely due to the effects of the teachers and administrators in the schools on student achievement. In this respect, the results corroborated earlier findings (Alicias, 2005; Amrein-Beardsley, 2014; Berliner, 2014; Kupermintz, 2003) that bonus pay models sometimes neglect factors like the percentage of students from low-income families and school location that are known to be associated with student achievement test scores. Thus, the question remains as to whether or not it is fair to provide teacher compensation based on factors over which a school or teachers have little or no control. This study found that variables associated with learning that were beyond the control of teachers and school administrators continue to affect bonus pay distribution models. The results also corroborated the findings of studies (Alspaugh, 1992; Hopkins, 2005) that indicated students tend to achieve better in rural schools when controlling for family income level. The New Teacher Project brought to light important information regarding ineffective and invalid teacher evaluation practices, and the money available from Race-To-The-Top induced states to ameliorate how their schools appraise teacher practices. At the core, these aspects of education reform are not entirely misguided. However, the problem arises when statisticians and policy makers collaborate to include student test scores as part of the teacher evaluation process. Not only is the test taken during a single week or on a given day affected by numerous unquantifiable influences acting on the student, but variables such as the student’s socio-economic status and the location in which the student attends school also affect test scores. Student learning is far too complex a process to ascribe to a single variable such as a teacher or school. This study was an examination of the relationship between the percentage of students from low-income families and school location, and the amount of bonus pay distributed using Betebenner's (2008) growth percentile methodology for calculating achievement growth. Relationships between bonus pay and these known variables influencing student learning were apparent, a notion that should be considered as education policy makers craft ways to provide financial support to schools as well as to compensate effective teachers. References Alicias, E. R. (2005). Toward an objective evaluation of teacher performance: The use of variance partitioning analysis, VPA. Education Policy Analysis Archives, 13(30). doi: 10.14507/epaa.v13n30.2005 Alspaugh, J. W. (1992). Socioeconomic measures and achievement: Urban vs. rural. The Rural Educator, 13(3), 2-7. Amrein-Beardsley, A. (2014). Rethinking value-added models in education: Critical perspectives on tests and assessment-based accountability. New York, NY: Routledge. Berk, R. A. (1984, March). The use of student achievement test scores as criteria for allocation of teacher merit pay. Paper presented at the National Conference on Merit Pay for Teachers, Sarasota, FL. Retrieved from http://dlxs.lib.wayne.edu/aft/archive/files/varwwwaftprep3480083_c09a8e5b11.pdf Berliner, D. C. (2014). Exogenous variables and value-added assessments: A fatal flaw. Teachers College Record, 116(1), 1-31 Betebenner, D. W. (2008). Norm- and criterion-referenced student growth. Dover, NH: National Center for the Improvement of Educational Assessment. Retrieved from http://www.nciea.org/publications/normative_criterion_growth_DB08.pdf http://dlxs.lib.wayne.edu/aft/archive/files/varwwwaftprep3480083_c09a8e5b11.pdf Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 11 Betebenner, D. W. (2011). A technical overview of the student growth percentile methodology: Student growth percentiles and percentile growth projections/trajectories. Dover, NH: National Center for the Improvement of Educational Assessment. Retrieved from http://www.nj.gov/education/njsmart/performance/SGP_Technical_Overview.pdf Borland, M. V., & Howsen, R. M. (2003). An examination of the effect of elementary school size on student academic achievement. International Review of Education, 49(5), 463- 474. doi: 10.1023/a:1026348922511 Coleman, J. S., Campbell, E. Q., Hobson C. J., McPartland, J., Mood, A. M., Weinfeld, F. D., & York, R. (1966). Equality of educational opportunity. Washington, DC: U.S. Government Printing Office. Fan, X., & Chen, M. J. (1999). Academic achievement of rural school students: A multi-year comparison with their peers in suburban and urban schools. Journal of Research in Rural Education, 15(1), 31-46. Heck, R. H. (2009). Teacher effectiveness and student achievement: Investigating a multi-level cross-classified model. Journal of Educational Research, 47(2), 227-249. doi: 10.1108/09578230910941066 Hopkins, T. M. (2005). If you are poor, it is better to be rural: A study of mathematics achievement in Tennessee. The Rural Educator, 27(1), 21-28. Howley, C. B., & Bickel, R. (1999). The Matthew Project: National report. Retrieved from ERIC (ED433174). Idaho State Department of Education. (2013). Bonus pay distributed to Idaho schools in 2012, school enrollment in 2012, and the percentage of students receiving subsidized meals by school in 2012. Boise, ID: Author. Kupermintz, H. (2003). Teacher effects and teacher effectiveness: A validity investigation of the Tennessee value added assessment system. Educational Evaluation and Policy Analysis, 25(3), 287-298. doi: 10.3102/01623737025003287 Plecki, M. (1991, April). The relationship between elementary school size and student achievement. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL. Retrieved from ERIC (396861). Sanders, W. L., & Horn, S. P. (1994). The Tennessee value-added assessment system (TVAAS): Mixed model methodology in educational assessment. Journal of Personnel Evaluation in Education, 8(3), 299-311. doi: 10.1007/bf00973726 Sher, A., & Hardy, K. (2013, June 22). Teacher pay plan OK’d by Tennessee State Board of Education. Chattanooga Times Free Press. Retrieved from http://www.timesfreepress.com/news/2013/jun/22/teacher-pay-plan-okd-by-board/ Sirin, S. R. (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75(3), 417-453. doi: 10.3102/00346543075003417 Sum, A. M., & Fogg, W. N. (1991). The adolescent poor and the transition to early adulthood. In P. B. Edelman & J. Ladner (Eds.), Adolescence and poverty: Challenge for the 1990s (pp. 37-109). Washington, DC: Center for National Policy. The New Teacher Project. (2010). How federal education policy can reverse the widget effect: Transforming ESEA Title II to improve teacher effectiveness and student outcomes. Brooklyn, NY: Author. Retrieved from http://tntp.org/assets/documents/TNTP_TitleII_Brief.pdf http://www.nj.gov/education/njsmart/performance/SGP_Technical_Overview.pdf http://www.timesfreepress.com/news/2013/jun/22/teacher-pay-plan-okd-by-board/ http://tntp.org/assets/documents/TNTP_TitleII_Brief.pdf Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 12 The New Teacher Project. (2012). Keeping irreplaceables in D. C. public schools: Lessons in smart teacher retention. Brooklyn, NY: Author. Retrieved from http://tntp.org/assets/documents/TNTP_DCIrreplaceables_2012.pdf Tennessee State Board of Education. (2013). Minutes. Retrieved from http://www.tn.gov/assets/entities/sbe/attachments/7-26-13-June_2013_Minutes.pdf U.S. Department of Education. (2010). Race to the top Fund. Washington, DC: Author. Retrieved from http://www2.ed.gov/programs/racetothetop/legislation.html Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D., (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. Education Digest, 75(2), 31-35. Zhou, J. (2012). The effects of reciprocal imitation on teacher-student relationships and student learning outcomes. Mind, Brain, and Education, 6(2), 66-73. doi: 10.1111/j.1751- 228x.2012.01140.x Zoda, P., Combs, J. P., & Slate, J. R. (2011). Elementary school size and student performance: A conceptual analysis. International Journal of Educational Leadership Preparation, 6(4), 1-20. Retrieved from http://cnx.org/contents/78ba1624-a1e6-4360-804d- 0a1773cc3670@2/Elementary-School-Size-and-Stu . http://tntp.org/assets/documents/TNTP_DCIrreplaceables_2012.pdf http://www.tn.gov/assets/entities/sbe/attachments/7-26-13-June_2013_Minutes.pdf http://www2.ed.gov/programs/racetothetop/legislation.html http://cnx.org/contents/78ba1624-a1e6-4360-804d-0a1773cc3670@2/Elementary-School-Size-and-Stu http://cnx.org/contents/78ba1624-a1e6-4360-804d-0a1773cc3670@2/Elementary-School-Size-and-Stu Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 13 Author Notes Dr. Gary Storie Department of School Psychology and Educational Leadership Idaho State University Campus Stop 8059 Pocatello, ID 83209 storgary@isu.edu Dr. Gary L. Storie is Assistant Professor in the Department of School Psychology and Educational Leadership of the College of Education at Idaho State University. He is a former school superintendent and currently teaches courses for the preparation of school administrators. Dr. Peter Denner Department of School Psychology and Educational Leadership Idaho State University Dr. Peter Denner is Professor of Educational Psychology in the Department of School Psychology and Educational Leadership at Idaho State University. He currently teaches courses in statistics and provides collaborative support to the research of colleagues and graduate students in the areas of research design and statistical analysis. An earlier version of this manuscript was presented at the 106 th annual conference and research symposium of the National Rural Education Association, San Antonio, TX, October 17-18, 2014. mailto:storgary@isu.edu Storie & Denner: Using Student Test Scores to Award Merit Pay: A Look at the 2012 Pay-for-Performance Program for Idaho Schools Current Issues in Education, 18(3) 14 Volume 18, Issue 3 October 6, 2015 ISSN 1099-839X Authors hold the copyright to articles published in Current Issues in Education. Requests to reprint CIE articles in other journals should be addressed to the author. Reprints should credit CIE as the original publisher and include the URL of the CIE publication. Permission is hereby granted to copy any article, provided CIE is credited and copies are not sold. Editorial Team Executive Editor Constantin Schreiber Assistant Executive Editors Anna Montana Cirell Niels Piepgrass Authentications Editor Tray J. Geiger Layout Editor Constantin Schreiber Copy Editor Lucinda Watson Section Editors Earl Aguilera Evelyn Concepcion Baca James Cunningham Darlene Michelle Gonzales Megan Hoelting Colin Kavanagh Laura Beth Kelly Tomé Martinez Terrence McTier Priyanka Parekh Lydia Ross Olivia Stewart Joseph Sweet Lori Talarico Faculty Advisors Dr. Gustavo E. Fischman Dr. Jeanne M. Powers