Relationship among merit pay policies, teacher mobility and attrition: A study in the Rio de Janeiro public schools Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 8/20/2021 Facebook: /EPAAA Revisions received: 12/18/2021 Twitter: @epaa_aape Accepted: 12/18/2021 education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 30 Number 109 July 26, 2022 ISSN 1068-2341 Relationship among Merit Pay Policies, Teacher Mobility and Attrition: A Study in the Rio de Janeiro Public Schools1 Karina Carrasqueira Pontifícia Universidade Católica do Rio de Janeiro Brazil & Mariane C. Koslinski Universidade Federal do Rio de Janeiro Brazil Citation: Carrasqueira, K., & Koslinski, M. C. (2022). Relationship among merit pay policies, teacher mobility and attrition: A study in the Rio de Janeiro public schools. Education Policy Analysis Archives, 30(109). https://doi.org/10.14507/epaa.30.7136 Abstract: The objective of this work was to verify if there is an association between a merit pay policy practiced by the Rio de Janeiro municipal local administration in the patterns of teachers’ mobility and attrition. The research followed the movement of teachers who took posts in the Rio de Janeiro municipal system between 2009 and 2011 during the eight years the policy was in effect (2009-2016). Bivariate analyses and comparisons with tests of differences in means were performed. The main result was that the number of times the school reached the goals set by the policy and, consequently, the staff received performance pay presented a correlation with teacher mobility and attrition. 1 This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) - Brasil [Finance Code 001] and by Fundação de Amparo à Pesquisa do Estado do Rio de Janeiro (Faperj). http://epaa.asu.edu/ojs/ https://doi.org/10.14507/epaa.30.7136 Education Policy Analysis Archives Vol. 30 No. 109 2 Thus, the greater the frequency of school reached the performance target, the higher the probability of retaining teachers. Keywords: teacher mobility; teacher attrition; school accountability; merit pay policy Relación entre la política de responsabilidad, la movilidad y el abandono de los profesores: Un estudio sobre la red municipal de Río de Janeiro Resumen: El objetivo de este trabajo fue verificar si existe una asociación entre la política de remuneración por mérito practicada por la red municipal de Río de Janeiro en los patrones de movilidad y abandono de los profesores. La investigación siguió el movimiento de los profesores que ocuparon puestos en la red municipal de Río de Janeiro entre 2009 y 2011 durante los 8 años en que la política estuvo en vigor (2009 -2016). Se realizaron análisis bivariados y comparaciones con pruebas de diferencias de medias. El principal resultado fue que el número de veces que la escuela alcanzó las metas rendimiento fijadas por la política y, en consecuencia, el personal recibió bonifi caciones por desempeño presentó una correlación tanto con la movilidad como con el abandono del profesorado. Así, cuanto mayor era la frecuencia con la que la escuela alcanzaba la meta, mayor era la probabilidad de retener a los profesores. Palabras-clave: movilidad de profesores; abandono de profesores; política de responsabilidad; política de bonificación Relação entre política de responsabilização, mobilidade e abandono docente: Um estudo sobre a rede municipal do Rio de Janeiro Resumo: O objetivo deste trabalho foi verificar se existe uma associação entre a política de responsabilização escolar praticada pela rede municipal do Rio de Janeiro nos padrões de mobilidade e abandono docente. A pesquisa acompanhou a movimentação de docentes que tomaram posse na rede municipal do Rio de Janeiro entre 2009 e 2011 durante os 8 anos em que a política esteve em vigor (2009-2016). Foram realizadas análises bivaridas e comparações com testes de diferenças de médias. O principal resultado foi que o número de vezes que a escola alcançou as metas de desempenho estabelecidas pela política e, consequentemente, que a equipe ganhou bonificação apresentou correlação tanto com a mobilidade, quanto com o abandono docente. Assim, quanto mais vezes a escola alcançou sua meta, maior sua probabilidade de reter os docentes. Palavras-chave: mobilidade docente; abandono docente; política de responsabilização; política de bonificação Relationship among Merit Pay Policies, Teacher Mobility and Attrition: A Study in the Rio de Janeiro Public Schools Teacher mobility and attrition are relevant issues in the discussion about educational opportunities. The way teachers are distributed among schools has an impact on the propensity of a system to reduce or increase inequality (Akiba et al., 2007; Luschei et al., 2013; Rao & Jani, 2011), since the evidence indicates that the teacher is the school factor that most impacts student learning (Darling-Hammond, 2000; Muijs et al., 2014). Even though teacher turnover is, to a certain extent, normal and beneficial, as it guarantees staff renewal, when very high it has impacts on the educational system, for both schools and students (Atteberry et al., 2017; Burkhauser, 2017; Rivkin et al., 2005; Ronfeldt et al., 2011). As Allensworth and colleagues (2009) pointed out, teacher turnover can generate organizational Relationship among merit pay policies, teacher mobility and attrition 3 problems for the school, and can even harm the pedagogic work due to a lack of consolidation of the teaching staff during the school year. If schools that serve a specific sector of the public have higher teacher turnover than others, this variation can lead to inequalities among schools in the system (Clotfelter et al., 2010). Indeed, studies has shown that schools with highly vulnerable students have more difficulty in retaining teachers, either because they drop out of the school system or because they seek to change schools, even while remaining in the same system (Boyd et al., 2008; Hanushek et al., 2005; West & Chingos, 2009). It is also due to this evidence that developed countries, and subsequently those developing, have aimed their efforts towards the qualification of teachers, and introduced educational accountability policies to promote more effective teaching practices. However, currently, school accountability policies are identified as a factor that can lead to increased teacher attrition arising from the high level of stress generated (Ryan et al., 2017), and the migration of teachers from schools with poor results to schools with better performance. In this way, it would contribute to the increase of inequality between these schools (Ladd, 2001). Although there are many studies on the impact of accountability policies on student performance, there are only a few that seek to investigate the relationship between these policies and teacher turnover, especially in contexts that opt for models with low stakes. In Brazil, this type of policy has gained prominence since the 2000s, but there are no studies in the Brazilian context that investigate whether or how accountability policies - in particular merit pay policies, the model most implemented in the country - impacted the dynamics of teachers’ mobility and attrition. Contribute do fill that gap is the propose of this study. The Present Study In 2009, the Rio de Janeiro municipality implemented a policy of school accountability that included its own education assessment, imposition of goals to each school and a bonus pay for all tenure employees (teachers, principal, secretaries, and others school staff) in schools that achieved its goals (Annual Performance Award/APA). Thus, it was expected that the bonus pay policy practiced in the municipality would impact the retention of teachers. But how? On the one hand, it could retain teachers in low-performing schools that had a chance of achieving their goal. On the other, it could encourage teachers to seek to migrate to schools with a greater chance of earning bonuses, thus increasing teacher turnover in schools less likely to meet their target. Given this scenario, this study, which is a development arising from this author's thesis (Carrasqueira, 2018)2, seeks to observe whether the school accountability policy in force in Rio de Janeiro city between 2009 and 2016 had an impact on the patterns of teachers’ mobility and attrition. In addition, this article aims to bring evidence from the Brazilian context to dialogue with international literature, contributing to incorporate into the debate studies on accountability policies in developing countries with enormous educational and socioeconomic inequalities. Moreover, it aims to discuss the non-intended impacts of a low-stakes accountability policy, where even though clear standards/goals are set the consequences only involve winning an award and not any sort of punishment. The specific objectives of this article are: a) to verify whether the pattern of teacher mobility is related to the number of times the school has won the Annual Performance Award (APA); and b) to check if there is any relationship between teacher attrition and the number of times the school has won the APA. 2 This article is a deployment of Karina Carrasqueira’s doctoral dissertation, with the supervision by Mariane C. Koslinski. Both authors participated in the elaboration of all parts of the article. Education Policy Analysis Archives Vol. 30 No. 109 4 To achieve these objectives, initially, some bivariate analyses were carried out between teacher mobility/attrition and the number of times the school had won the APA. Then, comparisons were made between the schools’ teachers left and those entered to check for possible mobility patterns; and comparisons between schools that have retained their teachers, and those where teachers have left the system to check for possible attrition patterns. The concept of teacher mobility is the transference of a teacher from one school to another, different from teacher turnover, which refers to the flow of teachers entering and leaving a school. In other words, mobility is related to movement from the teacher's point of view and turnover is how schools view it. Based on the literature (Boyd et al., 2008; West & Chingos, 2009), we classify mobility patterns into four categories: I) Teachers who remain at the same school; II) Teachers who move to another school in the same educational system; III) Teachers who move to a school in another system; and IV) Teachers who abandon the profession. Each of these patterns has implications at both the school and the local educational administration levels. For analytical purposes, in this paper, we will make a distinction between mobility and attrition, considering mobility as moves between schools within Rio de Janeiro municipal system (II), and attrition as the departure of the teacher from Rio de Janeiro municipal educational system, regardless of whether to another school system or to another profession (III and IV). In other words, mobility is when a teacher change schools within Rio de Janeiro municipal system, and attrition is when a teacher leaves Rio de Janeiro municipal system. Literature Review Teacher Mobility and Teacher Attrition Studies on teacher mobility in different educational systems have observed that teachers migrated more when they were in schools that had students with a low socio-economic level, performed poorly in external assessments and were from ethnic minorities (Alves et al., 2013; Hanushek et al., 2005; Torres et al., 2008; West & Chingos, 2009). In Brazil, Cunha (2019) observed that there was a convergence between the characteristics of teachers and schools. Schools with higher socio-economic students had fewer novice teachers and more teachers with higher levels of education. However, research such as Allensworth and colleagues (2009) indicate that the school climate can be an important factor in retaining teachers in schools. In this sense, schools with similar student input may have different levels of turnover due to the difference in their school climate (Carrasqueira & Koslinski, 2019). Another factor that impacts on teacher mobility is educational policies. An example is given by Luschei and colleagues (2013) who compare the systems of South Korea and Mexico. In South Korea, there is a rotation system that requires teachers to change schools every 5 years, preventing the concentration of more qualified and more experienced teachers in schools with a higher socioeconomic level. In Mexico, on the other hand, the autonomy of teachers to decide which school to go to creates greater inequality in terms of the level of turnover and characteristics of teachers between schools. Accountability policies, the central theme of this article, are also studied as factors of impact on mobility and will be discussed in a next section. However, in what concerns leaving a education system altogether, researchers observed that factors related to school characteristics exerted less impact (Allensworth et al., 2009; Sass et al., 2012). What seemed to make a difference were the characteristics of the teachers and the education system, such as job security and salary. Ávalos and Valenzuela (2016), in Chile, applied a survey to 157 teachers who abandoned their careers and the main reasons for the decision to abandon were, in order: discontent with the Relationship among merit pay policies, teacher mobility and attrition 5 possibilities of professional development, low salaries and dissatisfaction with the leadership of management. In the same direction, Gonzalez and colleagues (2008) interviewed eight teachers who left the profession and the main factors reported were related to little administrative support, student indiscipline and low salaries. In Australia, Buchanan and colleagues (2013) used a longitudinal qualitative method in which they interviewed 42 new teachers in three different school years (in the last year, only 14 participants still remained) to understand the reasons that led teachers to stay and leave schools. The main factors of permanence were related to opportunities for professional development and the school climate, characterized by peer collaboration, support and the physical environment of the school. In Brazil, studies point in the same direction, indicating a process of physical and mental exhaustion of teachers throughout their careers (Rebolo 2012; Xavier & Barbosa, 2015). Issues such as violence, devaluation of the profession and little leadership support are the most common reports to justify attrition (Carrasqueira & Koslinski, 2021). School Accountability and Merit Pay Policies in Brazil Accountability policies are born in the midst of the new public management between privatization policies on one hand, and, on the other, the incorporation of the private administration values in the public sector. In education, the first countries that adopted these policies were the USA, Chile, and England, in the 1980s. In the following decades, international organizations such as UNESCO and the World Bank encouraged the adoption of evaluation systems and accountability mechanisms as a way to improve educational indices, especially in developing countries (Carrasqueira, 2018, 2020). Accountability policies are very varied, and may have mechanisms that directly interfere in the autonomy of teachers and principals (high stake policies; e.g. dismissal, relocation, intervention), and others that seek to induce improvements based only on social constraints (report card policies; e.g. ranking of schools, public dissemination of results). In some cases, policies have a combination of punishment and reward mechanisms. Despite the varying designs, policy-makers that put forward school accountability policies intend to offer incentives to school staff for more effective practices and, consequently, promote increase in students learning (Hout & Elliot, 2011; Koslinski et al., 2015; Pontual, 2008). One of the possible incentive mechanisms of school accountability is the merit pay policy, which has become synonymous with the accountability policy in Brazil, since it is the highest impact legally established by the systems that apply this type of policy in the country - although in the set of accountability policies it is considered as low stakes. Although they may have different intensities and designs, merit pay policies are generally based on cash prizes for schools and / or teachers according to the results of their students. When less intense, the responsibility for the outcome is distributed diffusely among all school personnel; when more intense, the responsibility lies with few teachers. There are more simplistic designs in which the prizes are paid to schools that are ranked higher in the evaluations and whose sole criterion is the students' performance, to more complex models that aim at greater justice with the incorporation of socioeconomic variables, value-added methods and the elimination of dispute between schools (Bonamino & Souza, 2012; Brooke, 2006; Ladd, 2001). Accountability policies have proliferated in the local public school systems of Brazilian states and municipalities since 2000, heavily influenced by international organizations and the experience of developed countries, as well as by the prevailing rationality in public administration (Koslinski et al., 2015). Unlike the national assessment system, which is of the report card type and does not Education Policy Analysis Archives Vol. 30 No. 109 6 propose any direct impact on schools, subnational systems have massively adopted performance pay/merit pay policies - for teachers, administrative staff, ancillary employees and, in some cases, students - based on educational goals. In comparison with the high-stake policies of the USA and Chile, where there is intervention in schools and dismissal of teachers, the Brazilian policies are considered low-stake policies, as they have direct bonuses and indirect sanctions. In Brazilian context – as well as in the USA context (Education Endowment Foundation, 2016; Hout & Elliot, 2001) - there was no conclusive and consistent evidence on the impact of school accountability / merit pay policies on school performance (Burgos et al., 2013; Koslinski et al., 2017; Sousa & Koslinski, 2015). In addition, some studies provide evidence regarding unintended effects such as gaming or teaching to the test (Rosistolato et al., 2014; Silva et al., 2013). Attrition, Mobility and School Accountability In Brazil, no research other than this one has ever aimed to analyse the impacts of school accountability policies on attrition or mobility. In the USA, Ladd (2001) reported on a study carried out in North Carolina that used to make payments to teachers presenting high performance. According to the author, there was an unexpected effect: teachers sought to migrate from schools with no chance of earning bonuses to ones that did, which made it difficult for poorly performing schools to retain teachers. The same result was found by Ingersoll, Merrill and May (2016) for a representative sample of schools in the country. Accountability policies had different characteristics and also impacted teacher turnover in varied ways. However, low-performing schools found it more difficult to retain teachers, especially when penalized with sanctions, and high-performing schools that benefited from bonuses, found it easier to maintain their staff. Feng and colleagues (2010) verified the impact of changing the accountability policy in Florida, and observed that the poorly rated schools showed higher turnover compared to those whose performance did not change and those with better ratings. In addition, the authors found that teachers sought to migrate from schools that had performed less than expected to superior schools. The study by Ryan and colleagues (2017) found that accountability policies in three U.S. states directly increased levels of stress, burnout and intention to change schools, and indirectly increased the intention to quit the profession. Although the policies observed by the authors had much higher impacts than Brazilian policies, there is evidence that administrators of poorly performing schools in Rio de Janeiro city were under more pressure, thus increasing the probability of them leaving office (Andrade et al., 2018). Although most studies that investigate the relationship between mobility, attrition and school accountability have been produced in contexts that have adopted high-stake policies with great pressure on school actors, the same feeling that teacher in Ladd’s research (2001) had about the school’s result really reflecting their work can be observed in Brazil (Cerdeira et al., 2014; Cerdeira et al., 2017). Therefore, we suppose that if teachers consider that the goals are not feasible or fair, they will choose to change schools. But, perhaps the mild consequences, with less pressure on teachers, may have less impact on attrition in Brazilian context. Contextualization Characteristics of the Rio de Janeiro Educational System The Rio de Janeiro municipality is a peculiar case in Brazil, due to its context, scope and diversity. In terms of the education system, since it was the country's capital until 1960, public elementary schools were already fully municipalized long before the decentralization process started Relationship among merit pay policies, teacher mobility and attrition 7 in the country in the 1980s. Thus, the municipal education system in Rio de Janeiro is the largest in Brazil, currently attending around 642,000 students, ranging from nursery school to ninth grade3. Also, it has almost 40,000 teachers distributed among 1,542 schools, in addition to more than 13,000 technical and administrative support employees4. In socio-economic terms, according to data from the Human Development Atlas of Brazil (n.d.), in 2010, the Rio de Janeiro municipality had a Municipal Human Development Index (MHDI) of 0.799, slightly above the country's average of 0.727. However, the MHDI of the territorial units within the municipality ranged from 0.604 to 0.959. Thus, in the same municipality, there were regions where almost half of the population was vulnerable to poverty (49%, 14% being extremely poor children), with an average per capita income of around 75% of the minimum salary, the years of study expectation was 7.25 years, and more than half (52%) of the adult population had less than an elementary school education. Moreover, another region where less than 0.5% of the population were vulnerable to poverty, with per capita incomes almost 15 times the minimum, the study expectation was 11.77 years, and more than 2/3 had higher education (69.5%). Rules of Mobility and Attrition in the Rio de Janeiro Public System In the Rio de Janeiro municipal system we can think of mobility as following two paths. The first is via a transfer contest, in which the teacher changes his original school. These migrations are carried out through annual contests, and, in order to register, the teacher must have spent at least five years in the system (that is, in the post from which he/she intends to be transferred), and the change of school occurs after the end of the academic year. The other path is assignment, in which the assigned teacher belongs to one school, but is working in another. In other words, it is as if he/she were on loan. There is no minimum or maximum period for the teacher to stay at the school where he is on loan, nor a limit on the number of times the teacher can be assigned. In addition, the teacher can be assigned to any other school in the system. This is the type of movement mostly observed by this study, since teachers who joined the system in 2009 were only able to participate in a transfer contest in 2015. In this work, attrition refers to teachers who are no longer an employees of the Municipal Education Department. This means that the reference is not to teachers who retired or died, although, with the data used, it is not possible to differentiate these situations. Thus, in this context, attrition is represented by two situations: (i) teachers who tender their resignation; (ii) teachers who abandon the function. Resignation is an official process in which the teacher gives up his position and his departure is promptly notified in the municipality’s Official Gazette. On the other hand, attrition of the function is configured when the teacher has 30 consecutive days of unjustified absence5. In practice, the teacher can resume the position even after a long period of attrition. 3 The Brazilian basic educational system is structured accordingly: child education, which is subdivided in nursery school (0 to 3 years old) and preschool (4 to 5 years old); fundamental education, which is subdivided in primary school (Grades 1-5, 6 to 10 years old) and secondary school (Grades 6-9, 11 to 14 years old); and high school (Grades 1-3, 15 to 17 years old). Municipalities do not offer high school, which is the obligation of the states’ governments, and the numeration of grades is not continuous between fundamental education and high school. Nursery school is the only educational level that is not mandatory. 4 Data from April 2020, obtained from http://www.rio.rj.gov.br/web/sme/educacao-em-numeros 5 Obtained from the orientation document for corrections of the Sector, utilized as a guide for Human Resources Co-ordination procedures. Education Policy Analysis Archives Vol. 30 No. 109 8 The School Accountability Policy of the Rio de Janeiro Municipality Launched in 2009, the school accountability policy of the Rio de Janeiro municipal system was based on a standardized external assessment, an educational index, and a bonus policy. The bonus policy, called Annual Performance Award ([APA]; Prêmio Anual de Desempenho [PAD]), aimed to “reward teaching staff for their efforts to improve the quality of teaching and learning of their students” (Decreto n. 30.860, 2009, Art. 3). It was a monetary prize corresponding to the value of the teacher’s salary. The schools winning awards were those that achieved the growth goals measured by the educational index stipulated by the Municipal Department of Education. All teachers and school staff received the reward when the school reached the growth goal. The calculations for the goals are made by performance index ranges based on the previous year's index. Schools that have lower indexes have goals with higher growth rates. For example, a school with an index of 2 needs to grow by 22%, that is, reach an index of 2.5; whereas a school with an index of 6, would need to grow only 2%, that is, reach 6.1; schools with an index of 7 or higher only need to maintain this value, since it is understood that the closer to the ceiling - which in the case of this index is 10 - the more difficult it is to increase the index. The education performance index takes into account only two dimensions: the students' scores in the external assessment and the school flow - how long, on average, a student takes to complete the educational level being assessed. In this way, the index does not take into account the students' socio-economic background differences – or other schools’ characteristics that principals and teachers have little autonomy to modify – that can impact schools’ performance or ability to achieve their goals. On the other hand, as schools are not competing with each other and each school has got its own target it is possible that a school with a low development index will be awarded a prize. The criteria for awarding the prize have varied over time. First, for the teacher to be fully entitled to the prize, a limit on the number of absences was established. Then it was established that the prize would be divided equally between a fixed and a variable portion. The fixed portion, corresponding to up to 50% of the value of the prize, would be calculated on the teacher’s service time in the school. In other words, the fixed part also varied. The variable part would depend on the number of unjustified absences during the school year. These criteria were introduced to encourage teacher attendance and deter mobility during the academic year. Other minor adjustments related to the targets also occurred over the eight years the program was in effect (Koslinski et al., 2015). Methodological Aspects The data used in this work was drawn from databases provided by the Rio de Janeiro Municipal Department of Education, that is, from the Administrative Management System, the Human Resources Coordination and the Prova Rio Assessment System. In this data there was information about teachers - date of entry into the system, the respective school names and dates of leaving and entering - and about the schools themselves - performance index and if the APA was awarded. The teachers selected had entered the Rio de Janeiro municipal system in 2009, 2010 and 2011. The start of the period chosen marked the beginning of the municipality's accountability policy. In addition, the study wanted to be able to accompany the teachers for at least 5 years until 2016, the last year the policy. It is important to note that, given the limitation of the data, the teachers selected were active in the system in early January 2012. This means that teachers who had abandoned the system before that date could not be observed. Relationship among merit pay policies, teacher mobility and attrition 9 For this study, teachers were excluded if they were in schools ineligible to participate in the APA due to not possessing the educational level or teaching modality assessed6. This represented a loss of about 9% of the teachers in the sample in the attrition analysis and 1.4% in that of mobility. Table 1 summarizes the teacher movement observed in the 8 years covered. Table 1 Description of the Dependent Variables Left Rio de Janeiro system N Percentage Yes 482 14.7% No 2.787 85.3% Total 3.269 100% Changed schools within Rio de Janeiro system N Percentage Yes 2.190 66.4% No 1.111 33.6% Total 3.301 100% To check for possible patterns of attrition in relation to the APA, bivariate analyses and series of comparisons were performed between the last school where teachers were working when they abandoned (leavers), and the last school where teachers were in the end of 2016 for those who remained in the system (stayers). Then, to check the possible patters of mobility in relation to the APA, bivariate analyses and series of comparisons were performed: i) between the first school and the second school where teachers were located; ii) between the first school and the last school where he/she was located. Regarding the methodological aspects, the comparison between schools for analysis of mobility was made in a simple way: First, we created a variable corresponding to how many times the school was awarded the APA until the year in which the teacher left the school of origin (first school); and two other variables corresponding to how many times the school was awarded the APA until the year the teacher entered his/her second and the last school. Second, the value of the variable from the first school (number of times first school received the bonus pay before teacher left it) was subtracted by the value of the variable from the second and the last school (number of times second and last school received the bonus pay at the time teacher entered the school). Finally, to check if the teacher migrated to a school that had won the APA more times, the differences between the characteristics of the home school and the destination school were considered as of a 0.5 or 1 standard deviation (downwards and upwards) in relation to the subtraction results. The differences that were less than the 0.5 or 1 standard deviation, downwards or upwards, were considered equivalent, that is, the teacher migrated to a school with a characteristic equivalent to that of the original school. 6 It is noteworthy that even among regular schools catering for the first segment of fundamental education, not all, at the time of the test, had the school series level assessed. We do not possess the lists of the schools that were assessed each year. Education Policy Analysis Archives Vol. 30 No. 109 10 Besides this, for mobility as much as attrition, mean difference tests were conducted. For mobility, the test of differences in means was of samples in pairs, and for attrition, a test of differences in means was performed for independent samples. This was because, for the mobility analysis, two schools for the same teacher were being compared, while, for the attrition analysis different teachers were being compared. In both cases, this test checked whether there was a difference in means between the groups. Results Tables 2 and 3 show the descriptive statistics of the variables related to the number of times the school has won the APA utilized in the attrition analysis of and in the mobility analysis. Table 2 Descriptive Statistics of Number of Times the School Was Granted the APA for Leavers and Stayers (attrition analysis) N Min. Max. Mean SD Schools teachers left 481 0 6 1.88 1.14 Schools teachers stayed 2779 0 7 2.50 1.24 Table 3 Descriptive Statistics of the Number of Times the School Received the APA until the Year Teacher Left the School of Origin, and until the Year Teacher Entered the Second and the Last School (mobility analysis) N Min. Max. Mean SD First School 2190 0 7 1.33 1.15 Second School 2149 0 6 1.25 1.11 Last School 2087 0 7 1.71 1.25 Figure 1 makes a comparison between those who stayed in and those who left the system according to their schools’ number of APAs. A clear progression was observed indicating that the greater the number of times the school won an APA, the lower the chances of teachers in our cohort leaving the system. A test of differences in means for independent samples was performed comparing the averages of the schools of the teachers who left with the average of the schools of the teachers who remained (table 4), and the result was significant at 1%, corroborating the result of the bivariate analysis presented in Figure 1. Table 4 Mean Difference of the Times School Was Awarded the Bonus: Leavers X Stayers N Mean Dif. Sig. Leavers 481 1.88 -0.62 ** Stayers 2779 2.50 Note. Result of the tests of differences in means in independent samples, between the schools’ teachers who left the first school of the teachers who stayed in the system. **significant at 99% Relationship among merit pay policies, teacher mobility and attrition 11 Figure 1 Bivariate Analysis of Attrition in Relation to the Number of Times the School where the Teacher Was Assigned Won an APA Figure 2 shows the relationship between mobility and the number of times the school won the APA until the year the teacher changed schools (or until 2016 for teachers who did not change schools). Figure 2 indicates that teachers migrated much more from schools that did not receive the award, or conversely, teachers who were in schools that won the award appeared to be more likely to stay longer in these schools. It seems reasonable to assume that schools that are more likely to win an APA - even if the award is only the one corresponding to the 14th salary - are more successful in retaining their teachers, while teachers in schools with little chance of winning, try to migrate to schools with a greater chance of winning. Figure 2 Bivariate Analysis of Mobility in Relation to the Number of Times Teachers’ First School won an APA until the Year They Changed Schools and until 2016 for Those Teachers Who Stayed in the Same School Tables 5 and 6 show the mobility patterns considering only teachers who had changed their assignments. As some teachers have moved more than once during the period considered, the analysis was based on comparisons from the first to the second school, and from the first to the last school of the movers. Table 7 shows the results of the tests of differences in means corresponding to the results of previous tables. 78% 71% 85% 89% 94% 22% 29% 15% 11% 6% 0 1 2 3 4 t o 7 Attrition according to the number of APA the school won Stayers Leavers 12% 39% 61% 78% 88% 61% 39% 22% 0 o r 1 2 3 4 t o 7 Mobility according to the number of APA the school won Stay in same school Move to new school Education Policy Analysis Archives Vol. 30 No. 109 12 Table 5, considering a difference of 0.5 s.d., shows that there is a big difference in the mobility pattern between the first change (first and second school) and the last observed change (first and last school). Since, in the first migration, there is a little difference between the categories, and in the last migration, almost half of the teachers went to schools that won more APAs than the previous school. In table 6, despite the fact most schools were between -1 and 1 standard deviation, the proportion of teacher who moved to schools that won an APA more times than the previous school almost doubled, and the number of teachers that migrate to schools which won less times decreased. This indicates a change in the dynamics of mobility caused by the accountability policy. Table 5 Patterns of Teacher Mobility (0.5 SD) Second school Last school Won more times 32.9% 46.5% Won with same frequency 32.3% 28.8% Won less times 34.8% 24.7% Note. Table shows the teacher mobility patterns in relation to the number of times the school had won an APA. Difference of 0.5 standard deviation. Table 6 Patterns of Teacher Mobility (1.0 SD) Second school Last school Won more times 10.9% 20.9% Won with same frequency 76.0% 68.5% Won less times 13.1% 10.6% Note. Table shows the teacher mobility patterns in relation to the number of times the school had won an APA. Difference of 1.0 standard deviation. Table 7 Difference Between First School and Second and Last Schools N First School Mean New School Mean Dif. Sig. First and second 2149 1.31 1.25 0.06 * First and last 2087 1.32 1.71 -0.39 ** Note. Result of the tests of differences in means in samples in pairs: between the first and second school; and between the first and last school. Relationship among merit pay policies, teacher mobility and attrition 13 Moreover, this may be related to the fact that this award was introduced in the system in 2009, together with the evaluation component (Prova Rio), and the first result and payment of the award only occurred in 2010. Furthermore, as it was a recent event in the system, it may have taken a while to become absorbed into the teachers' decision criteria. That is, it might have taken a while for teachers to learn and internalise the rules of the bonus payment policy and to adapt their strategies accordingly. In addition, as there is no limit on the number of migrations, teachers can progressively move to schools with increasingly attractive characteristics - for example, with a greater chance of winning an APA. Discussion and Study Limitations The bibliographic reviews performed by Hout and Elliot (2011) and Pontual (2008) indicate that in the USA, school accountability programs had no clear effect. But in other contexts, such as in India, there is evidence of a positive effect. One hypothesis proposed by Pontual (2008) is that the positive effect depends on the context, that is, whether the award makes a substantial difference to the teacher's income. Regarding the impact on work conditions, studies have shown that accountability policies increase teachers' stress levels, especially in low-performing schools (Clotfelter et al., 2004; Geiger & Pivovarova, 2018; Ryan et al., 2017). In this study we observed that the number of times a school won an APA was significant to attrition. It may indicate that the financial incentive helped to keep the teacher in the Rio de Janeiro public system. Studies (Àvalos & Valenzuela, 2016; Borman & Dowling, 2008; Gonzalez et al., 2008) indicate that the financial issue has an impact on the teachers' decision to leave the system, although, at no time, was a relationship with awards specified. Only one study investigated the impact of financial incentive programs (Clotfelter et al., 2010), but these programs were of a different type from that practiced by the Rio de Janeiro municipal system7. Another possibility is that winning a prize generates prestige for the school, and being in a school that does not win a prize can generate a moral sanction, that is, a feeling of failure or that the school is bad. Ladd (2001) cites a study of a program in the USA that made payments to teachers presenting high performance. According to the author, it was found that schools attended by low- income students began to face more difficulty in attracting and retaining teachers who sought to migrate to schools where they would have more opportunity to win the award. In Rio de Janeiro, perhaps the prize has had a considerable weight in teachers’ decision to stay or leave a school. Brazilian school teachers generally earn less than other professionals with the same educational level, with Brazil being one of the Latin American countries that has the worst wages for their teachers (Hirata et al., 2019; Liang, 2003). Thus, even though it is a low-stakes policy, without punishment or material consequences, the financial incentive corresponding to one month’s salary is far from negligible. As the autonomy of schools is small in Rio de Janeiro – at least, smaller than the USA context - teachers can have a greater sense of powerlessness to achieve the goals. And considering that mobility rules are "loose", the incentives brought by the policy may be stronger to change schools, rather than changing practices to increase student’s performance. Most of the moves observed in this paper were done through an informal process (the loan) in which the Municipal Education Department has little or no control, and depends only on the willingness of the teacher, the availability of vacancy in the intended school, and the permission of 7 Clotfelter and colleagues (2010) investigated the impact of a financial incentive policy to attract teachers to schools with high proportion of minority students and students from low income families. Education Policy Analysis Archives Vol. 30 No. 109 14 the principals (of the leaving school and the entering school). Thus, the main agents in this process are the teachers and the principals. The move to a more attractive school depends, in particular, on the teacher's good relations and social and informational capital to know which schools will have vacancies. However, it is also possible that principals, using their knowledge of the system, lend to other schools the teachers that they don't want in theirs. Since principals in the Rio de Janeiro public school system do not have the autonomy to dismiss teachers, this would be a way to remove unwanted teachers from the school. The great discretion of the actors involved (Lipsky, 2019) and the patrimonialistic nature of this relationship, in which it is teachers and principals who control the movement of teachers between schools, is an obstacle to the democratization of public services (Weber, 2002) and acts as a factor in perpetuating inequalities. Schools in vulnerable contexts that fail to reach their goals are unprotected against turnover and teacher shortages. Conclusion The school accountability policy of the Rio de Janeiro municipality was in effect between 2009 and 2016. Although it underwent changes over the 8 years, it maintained an annual bonus for school staff who achieved the established goals. Few studies have set out to investigate the impacts of this policy, both in terms of expected results and other unexpected impacts. This research proposed to make an analysis on the possible effects of low stake school accountability policy on the mobility and attrition patterns of teachers in an educational system. The limitations of this study are in the databases used, which do not allow us to distinguish the different reasons for teachers leaving the system (for example, retirement or moving to another school system), nor to identify whether the teachers went on to exercise the teaching profession outside the Rio de Janeiro municipal system or whether they abandoned the profession. Also, it was not possible to observe any abandonment before 2012, nor the mobility of teachers who may have left the system before 2012. Thus, we missed the first three years of new teachers in the system, which is the period that research indicates as the period of greatest abandonment (Ronfeldt et al., 2011). The main contribution of this article is to bring to the international literature a research carried out in the largest local public system in Latin America. Even though the results are similar to those presented by researchers from the USA (Ladd, 2001), it is important to highlight that we are observing an institutional, political, social and economic context that is different from most of the studies that investigate these policies in developed countries and with levels of autonomy of principals and teachers very different from Rio de Janeiro’s system. In a few words, our results indicate that, although it aims to encourage improvement of low- performing schools, the school accountability policy practiced by the Rio de Janeiro municipality may have had an unexpected - or a perverse – effect, an influence on teachers' mobility patterns. A clear tendency was observed for teachers, after becoming used to the policy, to migrate to schools that had won the award more often than their original school. Also of interest was a possible correlation between the number of times a school won an APA and teacher attrition. The literature indicates that the characteristics of teachers and issues related to teaching, such as working conditions, salaries, career, prestige or lack thereof, exert a greater impact on a teachers’ decision to quit the profession. Thus, we can suppose that earning a higher salary can be a retention factor. Another possibility is that being awarded an APA reflects the school's work quality in general and is indicative of a favorable school climate. Relationship among merit pay policies, teacher mobility and attrition 15 References Akiba, M., LeTendre, G. K., & Scribner, J. P. (2007). Teacher quality, opportunity gap, and national achievement in 46 countries. Educational Researcher, 36(7), 369-387. https://doi.org/10.3102/0013189X07308739 Allensworth, E., Ponisciak, S., & Mazzeo, C. (2009). The schools teachers leave: mobility in Chicago public schools. (Research Report) .Consortium on Chicago School Research at the University of Chicago Urban Education Institute. Chicago. Alves, L., Padilha, F., Batista, A. A. G., Érnica, M., & Carvalho-Silva, H. H. (2013, Set./Oct. 29-02). Desigualdades socioespaciais e concorrência entre professores por escolas [Paper presentation]. 36ª Reunião Nacional da ANPEd, Goiânia, Goiás, Brasil. Andrade, F. M., Koslinski, M. C., & Ceneviva, R. (2018). Contextual factors related to school principal’s turnover in the city of Rio de Janeiro. Educação em Revista, 34, 1-26. https://doi.org/10.1590/0102-4698182446 Atteberry, A., Loeb, S., & Wyckoff, J. (2017). Teacher churning: Reassignment rates and implications for student achievement. Educational Evaluation and Policy Analysis, 39(1), 3–30. https://doi.org/10.3102/0162373716659929 Ávalos, B., & Valenzuela, J. P. (2016). Education for all and attrition/retention of new teachers: A trajectory study in Chile. International Journal of Educational Development, 49, 279–290. https://doi.org/10.1016/j.ijedudev.2016.03.012 Bonamino, A. C., & Souza. Z. (2012). Three generations of assessments of basic education in Brazil: Interfaces with the curriculum in/of the school. Educação e Pesquisa, 39(2), 373-388. https://doi.org/10.1590/S1517-97022012005000006. Borman, G. D., & Dowling, N. M. (2008). Teacher attrition and retention: A meta-analytic and narrative review of the research. Review of Educational Research, 78(3), 367-409. DOI: 10.3102/0034654308321455 Boyd, D., Grossman, P., Lankford, H., Loeb, S., & Wyckoff, J. (2008). Who leaves? Teacher attrition and student achievement. (NBER Working Paper 14022). National Bureau of Economic Research. Brooke, N. (2006). O futuro das políticas de responsabilização educacional no Brasil. Cadernos de Pesquisa, 36(128), 377-401. https://doi.org/10.1590/S0100-15742006000200006. Buchanan, J., Prescott, A., Schuck, S., Aubusson, P., Burke, P., & Louviere, J. (2013). Teacher retention and attrition: views of early career teachers. Australian Journal of Teacher Education, 38(3), 112-129. http://dx.doi.org/10.14221/ajte.2013v38n3.9 Burgos, M. B., Santos, M. M., & Ferreira, P. V. G. (2013). Avaliação, alfabetização e responsabilização: Os casos de Minas Gerais e Ceará. Revista Pesquisa e Debate em Educação, 2(2), 24-44. https://periodicos.ufjf.br/index.php/RPDE/article/view/32342/21386 Burkhauser, S. (2017). How much do school principals matter when it comes to teacher working conditions? Educational Evaluation and Policy Analysis, 39(1), 126–145. https://doi.org/10.3102/0162373716668028 Carrasqueira, K. (2018). Associated factors to teachers’ attrition and mobility in Rio de Janeiro municipal system [Doctoral dissertation, Federal University of Rio de Janeiro]. Post-Graduate Program of Education Archive. Carrasqueira, K. (2020, October 19-23). School accoutability: Um estudo sobre a política de responsabilização escolar do município do Rio de Janeiro [Paper presentation]. 12º Encontro da Associação Brasileira de Ciência Política, João Pessoal, Paraíba, Brasil. Education Policy Analysis Archives Vol. 30 No. 109 16 Carrasqueira, K, & Koslinski, M. C. (2019). Associated factors to teacher mobility in the municipality of Rio de Janeiro. Cadernos de Pesquisa, 49(173), 106–130. https://doi.org/10.1590/198053146014 Carrasqueira, K, & Koslinski, M. C. (2021). Teacher attrition in Rio de Janeiro municipal educational system. Educação em Revista, 37(1), 1-20. http://dx.doi.org/10.1590/0102-469820527 Cerdeira, D. G. S., Almeida, A. B., & Costa, M. (2014) Indicadores e avaliação educacional: Percepções e reações a política de responsabilização. Estudos em Avaliação Educacional, São Paulo, 25(57), 198-225. https://doi.org/10.18222/eae255720142845 Cerdeira, D. G. S., Prado, A. P., Rosistolato, R. P. R., Tavares, M. O., & Costa, M. (2017). Conhecimento e uso de indicadores educacionais no município do Rio de Janeiro. Estudos em Avaliação Educacional, 28(69), 926-968. https://doi.org/10.18222/eae.v0ix.4104 Clotfelter, C. T., Ladd, H. F., & Vigdor, J. L. (2010). Teacher mobility, school segregation, and pay-based policies to level the playing field. (Working Paper 44). National Center for Analysis of Longitudinal Data in Education Research. Clotfelter, C. T., Ladd, H. F., Vigdor, J. L., & Diaz, R. A. (2004). Do school accountability systems make it more difficult for low-performing schools to attract and retain high-quality teachers? Journal of Policy Analysis and Management, 23(2), 251–271. https://doi.org/10.1002/pam.20003 Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence. Education Policy Analysis Archives, 8(1), 1-44. https://doi.org/10.14507/epaa.v8n1.2000 Decreto n. 30.860 (2009, July 01). Dispõe sobre os critérios de premiação a ser concedida aos servidores, na forma que menciona. http://leismunicipa.is/bnmuq Education Endowment Foundation (n.d.). Performance pay. Teaching & learning toolkit. https://educationendowmentfoundation.org.uk/evidence-summaries/teaching-learning- toolkit/performance-pay/ Feng, L., Figlio, D. N., & Sass, T. (2010). School accountability and teacher mobility. (NBER Working Paper 16070). National Bureau of Economic Research. Geiger, T., & Pivovarova, M. (2018). The effects of working conditions on teacher retention. Teachers and Teaching: Theory and Practice, 24(6), 604–625. https://doi.org/10.1080/13540602.2018.1457524 Gonzalez, L., Brown, M. S., & Slate, J. R. (2008). Teachers who left the teaching profession: A qualitative understanding. The Qualitative Report, 13(1), 1-11. Hanushek, E. A., Kain, J. F., O’Brien, D. M., & Rivkin, S. G. (2005). The market for teacher quality. (NBER Working Paper 11154). National Bureau of Economic Research. Hirata, G., Oliveira, J. B. A., & Mereb, T. M. (2019) Professores: Quem são, onde trabalham, quanto ganham. Ensaio: Avaliação e Políticas Públicas em Educação, 27(102), 179-203. https://doi.org/10.1590/S0104-40362018002701888 Hout, M., & Eliott, S. (2011). Incentives and test-based accountability in education. National Research Council. Human Development Atlas of Brazil (n.d.). Atlas do desenvolvimento humano do Brasil. http://www.atlasbrasil.org.br/ Ingersoll, R., Merrill, L., & May, H. (2016). Do accountability policies push teachers out? Educational Leadership, 73(8), 44-49. https://repository.upenn.edu/gse_pubs/551/ Koslinski, M. C., Carrasqueira, K., Andrade, F. M., & Cunha, C. P. (2015). Políticas de responsabilização educacional: Modelos possíveis e uma análise preliminar da política do município do Rio de Janeiro. In: A. M. Cavalieri & A. J.Soares, Políticas educacionais no Estado do Rio de Janeiro. Mauad X: FAPERJ. http://leismunicipa.is/bnmuq https://educationendowmentfoundation.org.uk/evidence-summaries/teaching-learning-toolkit/performance-pay/ https://educationendowmentfoundation.org.uk/evidence-summaries/teaching-learning-toolkit/performance-pay/ http://www.atlasbrasil.org.br/ Relationship among merit pay policies, teacher mobility and attrition 17 Koslinski, M. C., Ribeiro E., & Oliveira, L. X. (2017). Indicadores educacionais e responsabilização escolar: Um estudo do -Prêmio Escola Nota Dez-. Estudos em Avaliação Educacional (Online), 28(69), 804-846. https://doi.org/10.18222/eae.v28i69.4087 Ladd, H. F. (2001). School-based educational accountability systems: The promise and the pitfalls. National Tax Journal, 54(2), 385-400. Liang, X. (2003). Remuneração dos professores em 12 países da América Latina: Como se compara a remuneração dos professores com a de outras profissões; o que determina, e quem são os professores? Documentos PREAL, 27, 1-38. Lipsky, M. (2019) Burocracia de nível de rua: Dilemas dos indivíduos nos serviços públicos. ENAP. Luschei, T. F., Chudgar, A., & Rew, W. J. (2013). Exploring differences in the distribution of teacher qualifications across Mexico and South Korea: Evidence from the Teaching and Learning International Survey. Teachers College Record, 115(050304). Muijs, D., Kyriakides, L., van der Werf, G., Creemers, B., Timperley, H., & Earl, L. (2014). State of the art: Teacher effectiveness and professional learning. School Effectiveness and School Improvement, 25(2), 231-256. http://dx.doi.org/10.1080/09243453.2014.885451 Pontual, T. C. (2008) Remuneração por mérito, desafio para a educação. (Estudo comissionado).Fundação Lemann. Rao, R. R., & Jani, R. (2011). Teacher allocation and equity in Malaysian schools. International Journal of Institutions and Economies, 3(1), 130-112. Rebolo, F. (2012). Of the malaise to abandonment of the teaching profession: the story of Estela. Série-Estudos, 33, 143–163. Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005). Teachers, schools, and academic achievement. Econometrica, 73(2), 417–458. https://doi.org/10.1111/j.1468-0262.2005.00584.x Ronfeldt, M., Lankford, H., Loeb, S., & Wyckoff, J. (2011). How teacher turnover harms student achievement. (NBER Working Paper 17176). National Bureau of Economic Research. Rosistolato, R., Prado, A. P., & Fernández, S. J. (2014). Cobranças, estratégias e “jeitinhos”: Avaliações em larga escala no Rio de Janeiro. Estudos em Avaliação Educacional, 25(59), 78-107. https://doi.org/10.18222/eae255920142853 Ryan, S. V., von der Embse, N. P., Pendergast, L. L., Saeki, E., Segool, N., & Schwing, S. (2017). Leaving the teaching profession: The role of teacher stress and educational accountability policies on turnover intent. Teaching and Teacher Education, 66, 1–11. https://doi.org/10.1016/j.tate.2017.03.016 Sass, D. A., Busto Flores, B., Claeys, L., & Pérez, B. (2012). Indentifying personal and contextual factors that contribute to attrition rates for Texas Public School teachers. Education Policy Analysis Archives, 20(15), 1-26. https://doi.org/10.14507/epaa.v20n15.2012 Silva, V. G., Gimenes, N. A. S., Louzano, P., Moriconi, G. M., Principe, L. M., Leme, L. F. (2013). Uso da avaliação externa por equipes gestoras e profissionais docentes: um estudo em quatro redes de ensino público. Coleção Textos FCC (Impresso), 38, 1-109. Sousa, S. Z., & Koslinski, M. C. (2016). Avaliação em larga escala, índices e premiação: Iniciativas de estados brasileiros e seus efeitos. In M. G. J. Setton, Mérito, desigualdades e diferenças: Cenários da (in)justiça escolar Brasil e Portugal. Annablume. Torres, H. G., Carpim, T. P, Gomes, S., & Bichir, R. M. (2008). Educação da perriferia de São Paulo: Ou como pensar as desigualdes educacionais? In L. C. Q. Ribeiro, R. Kaztman (Org.) A cidade contra a escola? Segregação urbana e desigualdades educacionais em grandes cidades da Amárica Latina (pp. 59-90). Letra Capital: FAPERJ; Montevideo: IPPES. Weber, M. (2002). Ensaios de sociologia. LTC. Education Policy Analysis Archives Vol. 30 No. 109 18 West, M. R., & Chingos, M. M. (2009). Teacher effectiveness, mobility and attrition in Florida. In M. G. Springer (Ed.), Performance incentives: Their growing impact on American K-12 education. Brookings Institution Press. Xavier, J. P., & Barbosa, A. F. (2015). Indisciplina, hostilidade no ambiente escolar e o abandono do magistério: As experiências de uma ex-professora da rede pública de Belo Horizonte. Palimpsesto, 14(3), 385–397. About the Authors Karina Carrasqueira Pontifícia Universidade Católica do Rio de Janeiro (PUC-Rio) karina.carrasqueira@gmail.com http://orcid.org/0000-0002-6699-7814 Post-doctoral researcher (PNPD/CAPES) in the area of sociology of education at the Education Department of the Pontifical Catholic University of Rio de Janeiro (PUC -Rio). Her research focuses on the issues of inequalities in educational opportunitie s and public education policies. Mariane C. Koslinski Universidade Federal do Rio de Janeiro (UFRJ) mckoslinski@gmail.com https://orcid.org/0000-0002-9644-5041 Associate professor in the post-graduate program in education at the Federal University of Rio de Janeiro. She works in the area of sociology of education, in the following subjects: educational assessment, analysis of educational policies, social and urban inequalities. education policy analysis archives Volume 30 Number 109 July 26, 2022 ISSN 1068-2341 Readers are free to copy, display, distribute, and adapt this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, the changes are identified, and the same license applies to the derivative work. More details of this Creative Commons license are available at https://creativecommons.org/licenses/by-sa/4.0/. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizon a State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A1 (Brazil), SCImago Journal Rank, SCOPUS, SOCOLAR (China). About the Editorial Team: https://epaa.asu.edu/ojs/index.php/epaa/about/editorialTeam Please send errata notes to Audrey Amrein-Beardsley at audrey.beardsley@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. http://orcid.org/0000-0002-6699-7814 https://orcid.org/0000-0002-9644-5041 https://creativecommons.org/licenses/by-sa/4.0/ http://www.doaj.org/ http://www.doaj.org/ https://epaa.asu.edu/ojs/index.php/epaa/about/editorialTeam mailto:audrey.beardsley@asu.edu https://www.facebook.com/EPAAAAPE