Microsoft Word - v22n3 FINAL.doc Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 12/17/2012 Facebook: /EPAAA Revisions received: 9/30/2013 Twitter: @epaa_aape Accepted: 9/30/2013 education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 22 Number 3 January 20th, 2014 ISSN 1068-2341 Opening the “black box”: Organizational differences between charter schools and traditional public schools Xin Wei Deepa Patel & Viki M. Young SRI International USA Citation: Wei, X., Patel, D., & Young, V. (2014). Opening the “black box”: Organizational differences between charter schools and traditional public schools. Education Policy Analysis Archives, 22(3). http://dx.doi.org/10.14507/epaa.v22n3.2014 Abstract: Using survey data collected from 2,273 teachers in Texas, this study explores differences in school organization that contribute to the experiences (e.g., working conditions, instruction and student engagement in learning, self-efficacy and job satisfaction, and teacher evaluation) of charter school and traditional public school teachers. Researchers used propensity score matching to reduce the impact of selection bias and to produce accurate estimates of the charter-traditional public school differences. Compared with similar teachers in traditional public schools, charter school teachers reported a more supportive teaching environment, higher expectations of students among staff, a greater sense of responsibility for student learning, and higher levels of student engagement in learning. However, they reported, attending fewer professional development trainings focused on instruction and aligned to teaching assignments, fewer opportunities for professional development and collaboration with colleagues, and lower perceived fairness of teacher evaluation. Findings from this study provide valuable insight into the school organization factors that may underlie teacher epaa aape E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 2 turnover and represent unmet needs among charter school teachers, and suggest strategic areas of focus for policymakers, charter management organizations, and charter school leaders in addressing teacher retention and student achievement. Keywords: charter schools; traditional public schools; school working conditions; instruction; teacher evaluation; job satisfaction; teacher efficacy; propensity score matching. Abriendo la “caja negra”: las diferencias de organización entre las escuelas chárter y las escuelas públicas tradicionales Resumen: Usando datos de una encuesta con 2.273 maestros en Texas, este estudio explora las diferencias en la organización escolar que contribuyen a las experiencias (por ejemplo, las condiciones de trabajo , de instrucción y de compromiso con los estudiantes en el aprendizaje, la autoeficacia y satisfacción en el trabajo, y la evaluación de los docentes) de las escuelas chárter y docentes en escuelas públicas tradicionales. Los investigadores utilizaron la correspondencia de puntuación de propensión para reducir el impacto del sesgo de selección y para producir estimaciones precisas de las diferencias entre las escuelas chárter y públicas tradicionales. En comparación con docentes similares en las escuelas públicas tradicionales, docentes de escuelas chárter reportaron un entorno de enseñanza más solidaria, expectativas más altas de los estudiantes entre el personal, un mayor sentido de la responsabilidad por el aprendizaje del estudiante, y niveles más altos de participación de los estudiantes en el aprendizaje. Sin embargo, informaron, que tuvieron un menor número de entrenamientos de desarrollo profesional, enfocados en la instrucción y alineados con labores docentes, menos oportunidades para el desarrollo profesional y de colaboración con colegas, y menor percepción de justicia en la evaluación docente. Los resultados de este estudio proporcionan información valiosa sobre los factores de organización escolar que pueden subyacer en la rotación de maestros y de necesidades no satisfechas entre los docentes de escuelas chárter, y sugieren áreas estratégicas de interés para los responsables políticos, las organizaciones de gestión de las chárter, y líderes de las escuelas autónomas en la retención de maestros y los logros de los estudiantes. Palabras clave: escuelas chárter; escuelas públicas tradicionales; condiciones de trabajo; instrucción; evaluación docente; satisfacción en el trabajo; eficacia docente. Abrindo a “caixa preta”: diferenças organizacionais entre as escolas charter e escolas públicas tradicionais Resumo: Usando dados de uma pesquisa com 2.273 professores em Texas, este estudo explora as diferenças na organização da escola que contribuem para a experiência (por exemplo, condições de trabalho, formação e compromisso com os alunos na aprendizagem, a auto-eficácia e satisfação no trabalho, e avaliação de professores) dos professores nas escolas charter e escolas públicas tradicionais. Os pesquisadores usaram equiparação da propensão dos resultados para reduzir o impacto do viés de seleção e para produzir estimativas precisas das diferenças entre charter e escolas públicas tradicionais. Em comparação com os professores similares em escolas públicas tradicionais, os professores das escolas charter relataram um ambiente de ensino mais favoráveis, altas expectativas dos alunos entre o pessoal, um maior sentido de responsabilidade para a aprendizagem do aluno, e níveis mais elevados de participação dos alunos na aprendizagem. No entanto, eles informaram que tinham menos formação profissional focada em desenvolvimento de instrução e alinhado com missões de ensino, menos oportunidades de desenvolvimento profissional, de colaboração com os colegas, e menor percepção de justiça na avaliação dos professores. Os resultados deste estudo fornecem informações valiosas sobre os fatores de organização escolar, que podem estar subjacentes rotatividade dos professores e as necessidades não satisfeitas, entre O pening the “black box ” 3 professores de escolas charter, e sugerir áreas de interesse estratégico para os formuladores de políticas de gestão de organizações charter, e dirigentes escolares charter na retenção de professores e desempenho dos alunos. Palavras-chave: escolas charter; escolas públicas tradicionais; condições de trabalho; educação, avaliação de professores; satisfação no trabalho; eficácia do professor. Introduction The public has engaged in a spirited debate over whether charter schools are more effective than traditional public schools in raising student achievement. Recent evidence on the effect of charter schools on student achievement is mixed: some studies show large positive effects of charter schools (Abdulkadiroglu, Angrist, Kane, & Pathak, 2011; Angrist, Dynarski, Kane, Pathak, & Walters, 2010; Dobbie & Fryer, 2009; Gleason, Clark, Tuttle, & Dwoyer, 2010); others show modest effects (Hoxby & Murarka, 2009; Hoxby & Rockoff, 2005); and still others indicate no effects or negative effects (Bifulco & Ladd, 2006; Eberts & Hollenbeck, 2002; Furgeson et al., 2012). To date, most studies have focused on the differences in student achievement between charter schools and traditional public schools, but few have sought to open the “black box” of charter schools to explore the factors underlying these differences. As charter management organizations grow and policymakers continue to pass favorable charter laws, a better understanding of organizational differences (e.g., school environment and culture, instruction and learning, teacher job satisfaction and self-efficacy, and teacher evaluation) between charter schools and traditional public schools can help identify the factors that influence student achievement and teacher retention (Betts & Hill, 2006; Gill, Timpane, & Brewer, 2002; Zimmer et al., 2003). Previous studies comparing teachers in charter schools to their colleagues in traditional public schools provide limited understanding of influential organizational factors because they focus solely on differences in teacher characteristics or a limited number of working conditions (Bomotti, Ginsberg, & Cobb, 1999; Cannata, & Peñaloza, 2012; Ni, 2012). Furthermore, case studies or correlational studies predominate (Burian-Fitzgerald, Luekens, & Strizek, 2004; Donaldson & Peske, 2010; Dressler, 2001; Kellor, 2005; Malloy & Wohlstetter, 2003; RPP International, 1999, 2000), with the exception of one study that used a propensity score matching approach to adjust for teacher self-selection into charter schools (Ni, 2012). Therefore, we set out to fill this knowledge gap with a rigorous study of the differences between charter school and traditional public school teachers’ perceptions of working conditions, attitudes towards teaching and reported use of instructional strategies, job satisfaction, and sense of self-efficacy. With information on these differences, policymakers and school leaders can strategically focus efforts on addressing workplace conditions that may underlie teacher retention and student achievement, and represent unmet needs among teachers. Although previous research indicates the importance of the association between school organizational factors and teaching practice and student achievement, the main challenge in evaluating the differences in teachers’ perceptions of organizational factors is selection bias. Teachers who choose to teach in charter schools may differ in ways that are associated with their perceptions. Previous studies suggest that charter school teachers’ background characteristics are different from those of their traditional public school peers. For example, charter school teachers are younger, less experienced, more likely to come from minority backgrounds, and less likely to be certified than their traditional public school colleagues. In addition, they have fewer advanced degrees, graduated from competitive or selective colleges, and have higher turnover rates than traditional public school teachers (Baker & Dickerson, 2006; Bomotti et al., 1999; Burian-Fitzgerald E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 4 et al., 2004; Cannata & Peñaloza, 2012; Center for Applied Research and Educational Improvement, 1997; Colorado Department of Education, 1996; Finn, Manno, Beirlein, & Vanourek, 1997; Hoxby, 2002; National Center for Education Statistics, 2011; RPP International, 1999, 2000). Teachers choose to work in charter schools because they like the ostensible freedom and flexibility in teaching, educational philosophy, charter school mission and community, smaller classes, like- minded coworkers, and accountability for student achievement—all of which they view as features distinguishing charter schools from traditional public schools (Young et al., 2009, 2010a, 2010b; Beirlein, 1996; Bomotti et al., 1999; Cannata & Penaloza, 2012; Malloy & Wohlstetter, 2003; Woodworth, David, Guha, Wang, & Lopez-Torkos, 2008). To reduce the impact of charter school selection bias, we used propensity score matching—a major methodological improvement over previous studies—to select a comparison group of traditional public school teachers who are the “statistical twins” of the charter school teachers in our sample. The logic behind propensity score matching is to select traditional public school teachers who have a similar probability of teaching at a charter school as their matched charter school teachers. We selected matches based on baseline teacher characteristics such as gender, race, education degree, certification, school level, core subject assignment (e.g., math, English, science, social studies), special education assignment, and years of teaching experience. This method creates two similar groups for valid comparison when random assignment is not an option. By creating two comparable groups, we can identify the most significant differences between charter school teachers’ and traditional public school teachers’ perceptions of critical aspects affecting teachers’ work-lives— school working conditions, instructional approaches and student engagement in learning, job satisfaction and efficacy, and teacher evaluation. Literature Review Previous literature indicates that teacher perceptions of school-level factors are predictive of teaching behavior, teacher retention, and student achievement (Borman & Dowling, 2008; Boyd et al., 2011; Darling-Hammond, 2003; Gallagher, Means, & Padilla, 2009; Klassen & Chiu, 2010; Ladd, 2011; Peterson, Anjewierden, & Corser, 2001). Moreover, teachers’ perceptions of working conditions, instruction, job satisfaction and efficacy, and teacher evaluation differ between charter schools and traditional public schools and in some cases affect student achievement (Angrist et al., 2010; Cannata, 2007; Malloy & Wohlstetter, 2003). However, prior studies focus primarily on teacher characteristics or a limited number of school organization factors. This study builds on existing research by comparing charter school teachers’ perceptions to traditional public school teachers’ perceptions of several organizational factors that previous studies have found to influence teacher retention and student outcomes. By examining these differences, policymakers and school leaders in charter and traditional schools can better address the areas for which teachers report less satisfaction; and ultimately, increase teacher retention and improve student achievement. The following section reviews literature examining teacher perceptions of working conditions, instruction, teacher efficacy and job satisfaction, and teacher evaluation. Working Conditions Teachers’ perceptions of working conditions are highly predictive of teaching behavior, teacher retention, and student achievement (Borman & Dowling, 2008; Boyd et al., 2011; Darling- Hammond, 2003; Ladd, 2011; Loeb, Darling-Hammond, & Luczak, 2009), rendering working conditions a crucial, policy-manipulable factor in potentially retaining skilled teachers and fostering positive student outcomes. Working conditions include factors such as school leadership, teacher O pening the “black box ” 5 empowerment, collaboration with colleagues, professional development, mentoring and support, facilities, resources, and student behavior (Borman & Dowling, 2008; Boyd et al., 2011; Ladd, 2011). Effective school leadership, which includes instructional leadership (i.e., expectations and systems facilitating effective instruction) (Boyd et al., 2011; Elmore, 2000), team goal-setting (Hallinger, Bickman, & Davis, 1996), and distributed leadership (i.e., shared decision-making and expertise among staff) (Spillane, Halverson, & Diamonds, 2001), is one of the most important factors influencing teacher behavior and student outcomes (Boyd et al., 2011; Hirsch, Emerick, Church, & Fuller, 2007). Schools with effective leadership attract teachers and create productive and harmonious working relationships among staff members (Borman & Dowling, 2008; Boyd et al., 2011; Johnson & Birkeland, 2003), whereas schools with weak school leadership often have a higher likelihood of teachers leaving the school (Borman & Dowling, 2008; Boyd et al., 2011; Ladd, 2011). The importance of school leadership underscores the need to understand differences between charter and traditional public schools teachers’ perceptions of leadership because insight into differences may help schools better invest resources to achieve improved teacher and student outcomes. Prior studies examining differences in school leadership between charter and traditional public schools are limited. The studies that do explore these differences find mixed results with some studies reporting little difference between charter school leaders and traditional public school leaders and others finding more substantial differences. For example, there was no difference between charter school leaders and traditional public school leaders in Colorado; most charter school leaders in Colorado had prior experiences as school leaders in traditional public schools and many of them received formal leadership training (Dressler, 2001). In addition, Colorado charter school principals reported similar perceptions of job responsibilities and personal-professional challenges as traditional public school principals (Dressler, 2001). However, charter school leaders in other studies reported having more freedom to make decisions regarding personnel, and curriculum and instruction than traditional public school leaders (Burian-Fitzgerald et al., 2004; RPP International, 1999, 2000; Malloy & Wohlstetter, 2003). Using national data from the 2003-2004 Schools and Staffing Survey (SASS), a recent study (Ni, 2012) showed that although charter school and traditional public school teachers’ perceptions of principal leadership were similar, charter school teachers reported higher levels of distributed school leadership than their peers in traditional public schools. The limited evidence on differences in school leadership between charter and traditional public schools suggests the need for future research to further examine this important topic and better understand the impact of school leadership on improving teacher and student outcomes. In addition to school leadership, collaboration among teachers has been found to influence teacher behavior and is associated with higher student performance and lower teacher turnover (Bryk & Schnieder, 2002; Gruenert, 2005; Smith & Ingersoll, 2004). Collaboration and distributed leadership are mutually reinforcing and work together to promote positive school environments (Cannata, 2007). One study found that charter schools were better at promoting a stronger sense of collaborative responsibility among teachers than traditional public schools; however, the difference was very small (Cannata, 2007). Since previous research finds a link between teacher collaboration and teacher retention and student achievement, our study seeks to better identify differences between charter and traditional public school teachers’ perceptions of collaboration to provide valuable insight into the type and frequency of collaboration teachers desire. The number of school resources available to teachers may also influence student achievement. Previous research shows that inadequate facilities and lack of teaching materials are associated with high teacher turnover rates, although the association ranges from small to large E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 6 (Buckley, Schneider, & Shang, 2004; Loeb et al., 2009). A qualitative study found that new teachers in newly built charter schools in Massachusetts reported a lack of sufficient resources (Johnson & Birkeland, 2003). However, in another study conducted almost a decade later using national SASS data (Ni, 2012), both charter school teachers and traditional public school teachers reported receiving adequate resources. In addition, recent evidence on the association between school resources and student achievement is mixed, with some studies indicating a moderate to large positive association (see the review by Greenwald, Jedges, & Laine, 1996), while others showed no association (see the review by Hanushek, 1997). This study incorporates teacher survey measures of resources and support to understand whether and to what extent charter and public school teachers perceive them differently in their respective environments. Adequate support for instruction and classroom management is also a determinant of teacher retention. New teachers are more likely to teach at the same school and remain in the teaching profession compared to those who do not receive sufficient support (Smith & Ingersoll, 2004). Studies regarding whether instructional supports differ for charter school teachers compared to their peers in traditional public schools yielded mixed results. Some research found that charter school teachers reported receiving less support than their peers in traditional public schools (Bomotti et al., 1999; Johnson & Birkeland, 2003), although charter schools provided teachers with professional development tailored to teachers’ needs (Malloy & Wohlstetter, 2003). Other studies showed that charter school teachers compared to their traditional public school peers reported similar levels of professional development and more support in their working environment (Ni, 2012). The importance of adequate support for teacher retention and the mixed findings of previous research suggest the need for future exploration of differences in the types and level of support charter school teachers desire compared to their traditional public school colleagues. Because working conditions influence teacher behavior and retention, and ultimately student achievement, these key dimensions of school organization may help explain any differences in school climate, culture, and effectiveness between charter and traditional public schools. By focusing on these working conditions, we begin to define the elements in the black box that might distinguish charter schools from traditional public schools. Instruction School reform literature suggests an association between student achievement and teachers’ sense of responsibility for student learning, teaching advanced skills, and use of data for instructional purposes (Young et al., 2011; Gallagher et al., 2009; Peterson et al., 2001). Little research, however, documents the differences between charter school teachers and traditional public school teachers on these instruction-related dimensions. Previous studies found that charter school teachers placed a higher priority on academic learning and reported a higher sense of responsibility for student learning (Bomotti et al., 1999). Charter school teachers also reported working longer school days (Malloy & Wohlstetter, 2003; Ni, 2012), implementing intensive instruction in classrooms (Angrist et al., 2010), and advocating for longer grade spans for schools to eliminate the elementary to middle or middle to high school transition (Jacob & Rockoff, 2011; Booker, Sass, Gill, & Zimmer, 2011). Combined with working conditions, teachers’ attitudes towards teaching and their reported instructional approaches capture a comprehensive description of factors shaping teaching and learning that may help to explain differences in effectiveness between charter and traditional public schools. Our study integrates survey factors representing this comprehensive range of school organization and teaching. O pening the “black box ” 7 Teacher Job Satisfaction and Self-Efficacy Not surprisingly, teachers’ sense of self-efficacy and job satisfaction are predictive of teaching, teacher retention, and student achievement (Klassen & Chiu, 2010; Skaalvik & Skaalvik, 2007; Tschannen-Moran & Hoy, 2001). Supportive environments conducive to student learning are likely to increase teachers’ sense of efficacy and job satisfaction, while also promoting effective instruction. In addition to beliefs of self-efficacy, to be effective, teachers must believe that they have the responsibility and ability to improve student learning (Goddard, Hoy, & Hoy, 2004). Although teachers in charter schools were satisfied with their jobs (Bomotti et al., 1999; Bulkley & Fisler, 2003; RPP International, 1999, 2000), charter school teachers reported less job security, a higher likelihood to feel burnout, and a higher likelihood to leave their schools than their peers in traditional public schools (Malloy & Wohlstetter, 2003), These findings suggest other supports may have been lacking among charter schools compared to traditional public schools. Supports matter— first-year charter school teachers’ self-efficacy improved when they received mentoring on lesson planning, textbook use, and classroom management (Leonard, 2002). Understanding differences that exist between charter and traditional public school teachers’ perceptions can point to factors that may be most beneficial in improving their job satisfaction, self- efficacy, and potentially teacher retention. Teacher Evaluation Across the country, teacher evaluation is near the top of the policy agenda, fuelled by federal priorities as expressed in Race to the Top funding requirements and other policy initiatives. At the heart of teacher evaluation initiatives are efforts to use student test scores to evaluate and reward teacher performance, with the assumption that pay for performance will incentivize teachers to change instruction in ways that improve student learning, and ultimately, achievement. A clear gap in the literature exists concerning differences in teacher evaluation policies and practices between charter schools and traditional public schools. We found only two case studies that describe teacher evaluation in charter schools (Donaldson & Peske, 2010; Kellor, 2005), as the vast majority of studies focus on teacher evaluation in traditional public schools. The first study found that some charter schools were able to implement new performance pay systems more quickly than traditional public schools (Kellor, 2005). The second study found that some charter schools consistently tied teacher evaluation to student performance by frequently evaluating their teachers using structured observations, providing feedback, and focusing on growth and improvement (Donaldson & Peske, 2010). However, these findings are based on a small biased sample of charter schools, all of which have earned national reputation of excellent student achievement results. Currently, little research exists on charter school teachers’ understanding of the usefulness, fairness, and transparency of teacher evaluation and on the degree to which pay for performance affects teacher instructional practices in charter and traditional public schools. This study seeks to examine the differences in charter school teachers’ and traditional public school teachers’ perceived fairness of their teacher evaluation system to help explain any variation in teacher instruction; and ultimately, student achievement. Study Purpose Policymakers, private philanthropy, educators, and the general public have a great interest in identifying and replicating successful charter school practices. However, previous research shows great variability in school organization factors and effectiveness of charter schools (Furgeson et al., 2012). In addition, replication can be challenging to do well due to the tension between program E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 8 replication and adaptation to the local context (Racine, 2003). Despite the variability in charter school findings and challenges to replication, charter schools persist as a central strategy to serving underserved populations where the neighborhood school is failing. By examining differences between charter school and traditional public school teachers’ perceptions of school organizational factors, we begin to understand some of the underlying factors that may explain mixed results for charter school effectiveness. Although other factors such as school districts’ and charter management organizations’ oversight and policies can influence teacher experiences, this study focuses solely on school organization factors within the purview of school leadership. The differing teacher perceptions highlighted by this study provide policymakers and school leaders with valuable information on the school organizational factors they can influence to improve teacher retention and student achievement. Method This study uses a rigorous quasi-experimental approach to examine differences between charter school teachers and their matched peers in traditional schools in Texas. Study Setting - Charter Schools in Texas This study is set in rural Texas. The number of Texas charter schools has burgeoned since charter legislation passed in 1995. Approximately 130,000 students attended charter schools in Texas during the 2007-08 school year (Texas Association of School Boards, 2009). The number of students assigned to waiting lists to enroll in Texas charter schools doubled between 2008-09 and 2009-10 (DeGrow, 2009). And Texas charter schools serve on average more economically disadvantaged and minority students than traditional public schools (Texas Association of School Boards, 2009). The Texas charter school movement figures prominently in the debate over charter school effectiveness. No consensus has emerged that Texas charters schools are better than traditional public schools in raising student achievement (Booker, Gilpatric, Gronberg, & Janse, 2008; CREDO, 2009; Hanushek, Kain, & Rivkin, 2002). Moreover, teachers in Texas open-enrollment charter schools have fewer years of teaching experience and have higher rates of turnover than state averages. In addition, administrators and teachers at open-­‐enrollment charter schools earned significantly lower salary than their peers in traditional public schools (Texas Association of School Boards, 2009). Participants The study participants were part of a convenience sample of 2,559 teachers from a charter school district (16 schools and 408 teachers) and an adjacent traditional public school district (44 schools and 2,151 teachers) in a poor, rural area of Texas. Both districts serve student populations that are largely Hispanic (95%), limited-English proficient (40%), and economically disadvantaged (80%). Three hundred and seventy-six charter school teachers and 1,897 traditional public school teachers completed an online survey in spring 2011. The response rate was 92% for charter school teachers and 88% for traditional public school teachers. Table 1 provides descriptive statistics for the participants’ background characteristics. The full sample was overwhelmingly female. In addition, the charter school sample had a higher proportion of White teachers (38.19% vs. 5.84%, �² = 320.96, p < .001, ES = 0.70), a lower proportion of Hispanic teachers (59.07% vs. 91.85%, �² = 279.08, p < .001, ES = -0.70), and a higher proportion of teachers with fewer years of teaching experience (4.74 vs. 10.32, t = -14.75, p < .001, ES = -0.85) than the sample of traditional public school teachers. Hedges’ g effect size is calculated as the difference between the mean of the charter school teachers and the mean of the O pening the “black box ” 9 traditional public school teachers divided by the unadjusted pooled within-group standard deviation (What Works Clearinghouse, 2008). After propensity score matching, the weighted matched sample showed balance between charter school teachers and traditional public school teachers on all background variables (ES ≤ 0.06 on all background variables) as described later in this methods section. Table 1 Descriptive Statistics of Background Characteristics for Teachers in Charter Schools and Traditional Public Schools Teacher background characteristics Full sample Matched sample Charter Traditional public ES Charter Traditional public ES Male 23.24% 27.77% -0.11 26.52% 26.80% 0.006 White 38.08%*** 5.84% 0.70 5.80% 5.80% 0 Hispanic 59.18%*** 91.85% -0.70 92.17% 91.90% 0.01 Bachelor’s degree 84.17% 84.87% -0.02 86.92% 86.46% 0.01 Master’s degree 12.40% 12.55% - 0.004 13.08% 12.43% 0.02 Standard certification 52.77% 51.19% 0.03 54.88% 53.41% 0.03 Elementary teachers 29.19% 31.42% -0.05 36.92% 37.38% -0.01 Core subjects 44.86% 45.95% -0.02 56.17% 53.04% 0.06 Special education 4.32% 4.27% 0.002 3.68% 4.24% -0.03 Years of teaching experiences (mean) 4.74*** 10.32 -0.85 9.97 10.00 -0.003 N 370 1865 182 1086 Note: T-test was performed between charter and traditional public schools on years of teaching experience. Chi-square test was performed on the rest of the variables. ES = effect size. Effect size is calculated by the difference between the mean of the charter school teachers and the mean of the traditional public school teachers divided by the unadjusted pooled within-group standard deviation (What Works Clearinghouse, 2008). ***p < .001 Measures The teacher survey included measures of working conditions (leadership, collaboration with colleagues, professional development, mentoring and support, and resources), instruction and student engagement in learning, self-efficacy and job satisfaction, and teacher evaluation. Table 2 summarizes the definition of each school organization scale derived from the teacher survey. Survey items were replicated from Young et al. (2010a, 2010b), Campbell, Gallagher, Greene, and Yee (2011), SRI International (1999), and Wechsler, Caspary, Humphrey, and Matsko (2010). All of the factors were reported to have a minimum reliability alpha of 0.80 by the previous studies. We conducted factor analysis with our teacher survey sample to confirm the factor patterns, as described later in this methods section. E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 10 Table 2 Definition of School Organization Factors School organization factor Definition Working conditions Overall school leadership (12) The extent to which school leaders provide teachers with opportunities to influence school decisions, communicate goals and expectations, and support teachers with resources, professional development, and opportunities for collaboration. Distributed school leadership (3) The extent to which school leaders provide opportunities for teachers to be involved in important decision making and encourages them to share their opinions. Collaboration with colleagues (8) The extent to which teachers feel supported by their colleagues and included in the school community, and the frequency with which they meet with their colleagues to discuss instruction. Climate of high expectations (3) The extent to which teachers set high standards for teaching and seek to continually improve so all students are learning. Supportive teaching environment (4) The extent to which teachers feel supported by their colleagues and included in the school community. Need for support (10) The extent to which teachers feel they need additional professional support to improve their content knowledge, instructional strategies, and classroom management. Access to PD (3) The extent to which teachers have opportunities to participate in professional development during the school day, collaborate with teachers from other schools, and reflect on their professional development plan with school leadership. Participating in high quality PD (4) The extent to which teachers participate in content-specific professional development that builds on previous knowledge and is ongoing, coherent, and aligns with school priorities. Availability of materials (5) The extent to which teachers have the necessary technology, instructional materials, and classroom supplies. Instruction PD focus on instruction (10) The extent to which teachers feel professional development aligns with their teaching assignment and provides them with necessary content knowledge, instructional strategies, and classroom management strategies. Teacher' responsibility for student learning (3) The extent to which teachers believe they make their expectations clear to students and track student progress towards those expectations. Frequency of teaching advanced skills (13) The degree to which teachers present students with assignments that encourage them to think critically, be creative, and connect content across subject areas. O pening the “black box ” 11 Table 2 (Cont’d.) Definition of School Organization Factors School Organization Factor Definition Responsiveness to student differences (4) The extent to which teachers adjust their instruction to respond to students’ levels of understanding. Student engagement in learning (4) Teachers’ perceptions of the number of students who pay attention in class, actively participate in activities, complete homework, and care about their academic performance. Use of data for instructional purposes (5) The extent to which teachers use data to inform their daily instruction, modify instructional materials, and track student progress. Self efficacy (9) The extent to which teachers believe they are able to provide high quality instruction that meets the needs of all learners and build a classroom environment free of disruption and discipline problems. Job satisfaction (8) The extent to which teachers are satisfied with the teaching profession, school leadership, colleagues, and school environment. Fairness of teacher evaluation (6) The extent to which teacher evaluations are fair and clearly communicated, and provide teachers with formative feedback. Note: PD = professional development. Number of items for each factor is in parentheses. Teacher demographic variables (gender, race, education degree, certification, school type, core subject teacher, special education teacher status, and years of teaching experiences) were obtained from the districts’ human resources departments. The survey data and teacher demographic data were merged together by teacher identification number. Factor Analysis Although scales matching key school organization constructs were replicated from prior studies, we conducted principal axis with varimax rotation factor analysis to explore the internal structure of the survey items. Eigenvalues and scree plot cut-off points were used to guide the dimensionality of the factor space and to let the interpretability of the factors indicate the exact number of factors to retain. Table 3 presents eigenvalues, percent variance explained by each factor, and alpha reliability coefficients. The large, dominant first eigenvalue and explained variance indicate single factors for each set of items. Using the teacher survey data collected for this study, the reliability alpha (a measure of a single, unidimensional latent construct) ranged from 0.81 to 0.96, indicating a high level of inter-correlation among items under each factor. To create these summary measures, we computed factor scores, which are z-scores ranging from approximately -3 to 3, with the mean being zero and the standard deviation being 1. The factor scores indicate the ranking on a latent continuum for that factor. (See Appendix A for the survey items.) E ducation Policy A nalysis A rchives V ol. 22 N o. 3 12 Table 3 Eigenvalues Showing Single Dominant Factors for Each Set of Test Questions Factors Eigenvalues % of Variance explained by alpha 1st 2nd 3rd 1st eigenvalue 2nd eigenvalue 3rd eigenvalue Working conditions Overall school leadership (12) 7.37 0.91 0.69 61.48 7.62 5.78 0.94 Distributed school leadership (3) 2.48 0.31 0.21 82.74 10.28 6.98 0.90 Collaboration with colleagues (8) 4.86 0.87 0.63 60.70 10.85 7.86 0.90 Climate of high expectations (3) 2.28 0.39 0.34 76.85 12.89 11.26 0.84 Supportive teaching environment (4) 3.21 0.38 0.24 80.21 9.35 5.89 0.92 Need for support (10) 7.15 0.58 0.46 71.48 5.76 4.58 0.96 Access to PD (3) 2.00 0.54 0.46 66.65 18.05 15.30 0.75 Participating in high quality PD (4) 3.37 0.27 0.21 84.34 6.75 5.07 0.94 Availability of materials (5) 2.99 0.65 0.56 59.78 13.09 11.11 0.83 Instruction PD focus on instruction (10) 6.86 0.65 0.53 68.64 6.53 5.25 0.95 Teacher' responsibility for student learning (3) 2.23 0.47 0.30 74.23 15.75 10.02 0.82 Frequency of teaching advanced skills (13) 7.19 1.52 0.72 55.39 11.66 5.52 0.93 Responsiveness to student differences (4) 2.57 0.65 0.47 0.64 0.16 0.12 0.81 Student engagement in learning (4) 2.88 0.53 0.32 72.12 13.29 7.91 0.87 Use of data for instructional purposes (5) 3.22 0.60 0.44 64.32 11.94 8.81 0.86 Self efficacy (9) 5.42 0.85 0.61 60.25 9.43 6.77 0.92 Job satisfaction (8) 4.65 0.72 0.63 58.13 8.96 7.90 0.90 Teacher evaluation (6) 4.03 0.79 0.47 67.23 13.09 7.83 0.89 Note. PD = professional development. Number of items for each factor is in parentheses. Principal component factor analyses with varimax rotation were conducted. O pening the “black box” 13 Propensity Score Matching As seen in Table 1 and described above, charter school teachers and their peers in the adjacent traditional public school district are not comparable on pre-existing characteristics, such as race and years of teaching experience. Without matching traditional public school teachers to charter school teachers, any differences we find in the teacher survey results could be due to the differences in race or years of teaching experience between the two groups. To make compelling comparisons between charter school and traditional school teachers on their perceptions, we identified observably similar individuals in these schools using propensity score matching methods. Propensity score techniques have been used in observational studies with cohort or case- control designs (Connors et al., 1996; Gum, Thamilarasan, Watanabe, Blackstone, & Lauer, 2001), social programs (Dehejia & Wahba, 1990), health services programs (Keating, Weeks, Landrum, Borbas, & Guadagnoli, 2001; Mojtabai & Zivin, 2003), and educational programs (Nguyen, Taylor, & Bradley, 2006; Titus, 2007). In this study, the propensity score is the predicted probability of being a charter school teacher based on teacher background characteristics using logistic regression. Increasingly, propensity score methods are used to reduce selection bias in estimating intervention effects when randomized controlled trials are not feasible or ethical (Rosenbaum & Rubin, 1983; 1984). However, recent literature suggests that different propensity score methods can produce inconsistent results (Baser, 2006; Austin, 2008; Lunceford & Davidian, 2004). In the last two decades, very few studies have compared the merits and limitations of various propensity score methods. Thus no consensus has emerged on a preferred approach and there is no guarantee that a propensity score method that works best for one scenario will be the best for another (Austin, 2008). In this study, we applied and compared the relative merits of seven propensity score techniques on the degree to which each technique balances the charter school teacher sample and the traditional public school teacher sample on baseline demographic covariates. We used an R package, MatchIt, to apply the seven propensity score methods to select comparison traditional public school teachers who are the statistical twins of charter school teachers (Ho, Kosuke, Gary, & Stuart, 2007). The first method is called exact match. This method matched all possible traditional public school teachers to each charter school teacher with exactly the same covariate values, resulting in exact balance between the two groups. However, if no traditional public school teacher can be matched to a charter school teacher that charter school teacher is dropped from the final matched sample, resulting in a smaller sample size. The second method is subclassification. This method creates subclasses, where charter and traditional public school teachers in the same subclass are very similar in their covariates. The third method is nearest neighbor method, which finds the closest public school teacher(s) for each charter school teacher using a scalar distance measure based on all the covariates. The fourth method is optimal matching. The optimal method is similar to the nearest neighbor method with the additional benefit of finding control cases with the smallest average absolute distance across all matched pairs. The fifth method is full matching, which is one kind of subclass matching with the additional strength of optimally minimizing a weighted average of the estimated distance measure between each charter and traditional public school teacher within each subclass. The sixth method is genetic matching, which uses an evolutionary search logarithm to identify a set of weights for each covariate to achieve optimal balance between the two groups. Finally, the seventh method is kernel-based matching. This approach “matches” each charter teacher with a weighted average of one or more traditional public E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 14 school teachers where traditional public school teachers who have closer propensity scores to the charter school teacher receive larger weights. Following Austin (2008) and Baser (2006), we applied parametric and nonparametric methods to examine whether balance improved on all baseline covariates after applying each method. Genetic method is preferable because it achieved better baseline equivalence and also retained a relatively large sample size compared to other methods. Descriptive analyses on pre- existing background characteristics for the charter school teachers and their matched comparison teachers from traditional public schools showed that equivalence was greatly improved after applying the genetic matching method (right side of Table 1).1 After matching, the differences on covariates decreased from 0.85 standard deviations to 0.06 standard deviations. Therefore, charter school teachers participating in this study were very similar to their matched peers in traditional public schools on teacher background characteristics. Analysis of Covariance (ANCOVA) The differences between charter school teachers and their traditional public school peers in their perceptions about working condition, instruction and student engagement, job satisfaction and self-efficacy, and teacher evaluation were tested using analysis of covariance (ANCOVA) for the original full sample and the matched sample, separately. No weights were used for ANCOVA for the full sample. For the matched sample, weights generated from the propensity score package to achieve baseline equivalence were used in the ANCOVA models to test the differences between the two groups, following Ho et al. (2007). The differences between charter school and traditional public school teachers are presented in standard deviation units (i.e., effect size). The ANCOVA models statistically controlled for the same teacher background characteristics that we also used for the propensity scoring models (i.e., gender, race, education degree, certification, school type, core subject teacher, special education teacher, and years of teaching experience), providing more accurate estimates of charter and traditional public school teacher differences. Results Table 4 provides the descriptive statistics, as well as the estimated difference between charter school teachers and their peers in traditional public schools on each survey factor using ANCOVA. The left part of Table 3 presents the results for the full (unmatched) sample, while the right part of Table 3 provides the results for the matched sample. Note that the effect sizes presented in Tables 1 and 3 represent the differences of charter school teachers relative to traditional public school teachers in standard deviation units. For example, a positive effect size means charter school teachers’ responses were higher than those of traditional public school teachers on that factor. 1 The full sample includes 370 charter school teachers and 1,865 traditional public school teachers. Out of the full sample, we have complete baseline and survey outcome data on 250 charter school teachers and 1,086 traditional public teachers. Sixty-eight charter school teachers fell outside the area of common support because no good matches to traditional public school teachers could be found due to differences in observable characteristics (i.e., race, ethnicity, and years of teaching experience). Therefore, these 68 charter school teachers were excluded from the analysis so we could not estimate the differences in organizational factors between charter schools and traditional public schools for these 68 participants. The 68 excluded charter school teachers were more likely to be white, less likely to be Hispanic, and more likely to have fewer years of teaching experiences than those who were not excluded. O pening the “black box”: O rganizational differences between charter schools and traditional public schools 15 Table 4 Difference in Teacher Survey Factors Between Charter vs. Traditional Public Schools Factors Full sample Matched sample Charter schools Traditional public schools ES Charter schools Traditional public schools ES N Mean SD N Mean SD N Mean SD N Mean SD Working conditions Overall school leadership 361 -0.24 1.00 1820 0.05 0.99 -0.29** 182 -0.09 1.12 1086 0.07 0.98 -0.14 Distributed school leadership 366 -0.14 1.01 1854 0.03 0.99 -0.17* 182 0.03 1.10 1086 0.04 0.99 -0.01 Collaboration with colleagues 358 -0.20 0.92 1763 0.04 1.02 -0.26** 182 -0.12 0.82 1086 0.06 0.99 -0.21* Climate of high expectations 370 0.24 0.85 1867 -0.05 1.02 0.33*** 182 0.39 0.86 1086 -0.04 1.01 0.49*** Supportive Teaching Environment 361 0.11 0.88 1842 -0.02 1.02 0.15* 182 0.35 0.90 1086 0.00 0.99 0.38*** Need for support 353 0.18 0.91 1784 -0.04 1.01 0.24* 182 -0.19 0.93 1086 -0.06 1.01 -0.14 Access to PD 360 -0.08 0.83 1845 0.02 1.03 -0.12 182 -0.14 0.79 1086 0.00 1.01 -0.17 Participating in high quality PD 359 -0.24 0.91 1827 0.05 1.01 -0.32*** 182 -0.37 0.81 1086 0.02 0.98 -0.47*** Availability of materials 370 -0.20 0.97 1854 0.04 1.00 -0.25 182 0.08 1.01 1086 0.06 1.00 0.02 Instruction PD Focus on Instruction 358 -0.32 0.99 1781 0.07 0.99 -0.39*** 182 -0.20 1.12 1086 0.09 0.99 -0.27*** Teacher' responsibility for student learning 370 0.13 0.89 1870 -0.02 1.02 0.17*** 182 0.36 0.94 1086 -0.01 1.01 0.39*** Frequency of teaching advanced skills 342 -0.08 0.97 1734 0.03 1.00 -0.11 182 0.14 0.85 1086 0.08 0.97 0.07 Student engagement in learning 369 0.34 0.85 1828 -0.08 1.02 0.48*** 182 0.47 0.89 1086 -0.08 1.00 0.60*** E ducation Policy A nalysis A rchives V ol. 22 N o. 3 16 Table 4 (Cont’d.) Difference in Teacher Survey Factors Between Charter vs. Traditional Public Schools Full Sample Matched Sample Charter schools Traditional public schools ES Charter schools Traditional public schools ES Factors N Mean SD N Mean SD N Mean SD N Mean SD Responsiveness to student differences 362 -0.19 0.98 1826 0.06 0.98 -0.26*** 182 -0.05 0.88 1086 0.08 0.96 -0.15 Use of data for instructional purposes 362 -0.05 1.03 1817 0.02 0.98 -0.07 182 0.08 0.90 1086 0.05 0.95 0.03 Self efficacy 362 -0.45 0.99 1847 0.09 0.98 -0.55*** 182 0.03 0.91 1086 0.10 0.97 -0.07 Job satisfaction 354 -0.16 0.94 1813 0.03 1.01 -0.21 182 0.06 1.03 1086 0.03 1.00 0.02 Teacher evaluation 357 -0.26 1.07 1798 0.05 0.97 -0.29*** 182 -0.20 1.27 1086 0.10 0.95 -0.25*** Notes. PD = professional development. ES= effect size. All models controlled for gender, race, education, certification, school type, subject, special education teacher, and years of teaching experience. Hedges’s g effect size is calculated by the difference between mean of the charter school teachers and the mean of the traditional public school teachers divided by the unadjusted pooled within-group standard deviation. *p < .05. **p < .01. ***p < .001 O pening the “black box”: O rganizational differences between charter schools and traditional public schools 17 For the matched sample, we found that charter school teachers perceived their working conditions differently from traditional public school teachers. Charter school teachers reported higher expectations among teachers for student performance (ES = 0.49), a more supportive teaching environment (ES = 0.38), but less frequent collaboration with colleagues (ES = -0.21), and fewer chances to participate in high-quality professional development (ES = -0.47). In terms of instruction, charter school teachers reported less instructional support (ES = -0.27), a greater sense of responsibility for student learning (ES = 0.39), and higher levels of student engagement in learning (ES = 0.60). In addition, charter school teachers reported less perceived fairness in teacher evaluation (ES = -0.21) than traditional public school teachers did. Charter school teachers’ perceptions were not different from those of traditional public school teachers on overall school leadership, distributed school leadership, need for support, access to professional development, availability of materials, frequency of teaching advanced skills, responsive to student differences, use of data for instructional purpose, self-efficacy, and job satisfaction. We found statistically significant differences between charter school teachers and traditional public school teachers in the full sample but not in the matched sample on the following school organization factors: overall and distributed school leadership, need for support, responsiveness to student differences, and self-efficacy. The differences in the unmatched sample might be due to the pre-existing demographic and teaching experience differences rather than to true organizational differences between charter and traditional public schools. For example, in the full sample, charter school teachers reported significantly lower levels of overall and distributed school leadership; however, the differences were not significant for the matched sample. The differences in the full sample might not reflect any true differences in leadership environment between the two types of schools. Rather, they might be due to the fact that charter schools teachers were much younger (5.6 years younger), and less experienced. Participating in distributed leadership roles for less experienced charter school teachers may be different from the more experienced teachers in traditional public school. Similarly, charter school teachers’ reporting higher needs for support compared to traditional public schools teachers in the full sample but not in the matched sample also likely reflects charter school teachers’ lower levels of experience and acute mentoring and development needs at the start of their teaching careers. In other cases, even though results under both the full and matched samples were both significant and in the same direction, the effect sizes changed substantially due to the reasons we discussed above. These results for the full and matched comparisons demonstrate the necessity of accounting for and eliminating the pre-existing differences between charter school teachers and traditional public school teachers when studying organizational differences across school types. Discussion This study compared reports of working conditions, instruction and student engagement in learning, self-efficacy and job satisfaction, and teacher evaluation between charter and traditional public schools teachers. Our results are consistent with previous studies that suggest charter schools environments dedicated to serving underrepresented student populations are academically oriented, exhibit high expectations for student performance, are instructionally supportive, and attract mission-driven teachers (Young et al., 2009, 2010a, 2010b; Bomotti et al., 1999; Cannata & Penaloza, E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 18 2012; Woodworth et al., 2008). Our study contributes further data indicating that charter schools exhibit practices and features that might, according to literature, promote student achievement. The findings from this study also highlight charter school teachers’ potentially unmet needs. Teachers and administrators in Texas open-enrollment charter schools earned significantly lower salaries and had higher turnover rates than teachers statewide (Texas Association of School Boards, 2009). In addition to lower pay, charter school teachers may face additional challenges in their working conditions and evaluation processes that may underlie that turnover and represent a potential opportunity for policymakers and school leaders to address factors that could improve teacher retention. Echoing findings reported by Bomotti et al. (1999) and Johnson and Birkeland (2003), we found that charter school teachers compared to their traditional school peers reported receiving fewer opportunities for high-quality professional development, and particularly professional development focused on instruction. Contradictory to a previous study that found charter schools were better than traditional public schools at promoting teacher collaboration (Cannata, 2007), charter school teachers in this study reported lower levels of collaboration with colleagues than those in traditional public schools. Our findings suggest that substantial administrative support and collaboration time with colleagues need to be provided to improve working conditions in charter schools. For example, lack of high-quality professional development can be detrimental to teacher retention, particularly for new teachers (Smith & Ingersoll, 2004). Charter school leaders need to provide teachers with high- quality professional development, which may mean that they need to garner additional resources to release teachers from the classroom, extend the work day, or provide tuition reimbursement. In addition, charter school leaders could foster teacher collaboration to help retain effective and skilled teachers since previous studies have found strong associations between teacher collaboration and low turnover and high student achievement (Bryk & Schnieder, 2002; Gruenert, 2005; Smith & Ingersoll, 2004). This study is also one of the first to explore charter school teachers’ perception of their evaluation systems. Charter school teachers in our study reported less usefulness, fairness, and transparency in their teacher evaluation system compared to their peers in traditional public schools. As suggested by previous literature, teachers’ perceptions that the evaluation system is unfair impacts teaching negatively (Milanowski & Heneman, 2001), can undermine teacher morale, discourage potential teachers from entering the profession, and lead to high teacher turnover (Baker et al., 2010). This findings suggest that charter school leaders and charter management organizations might benefit from assessing the level of teacher understanding and buy-in of their current evaluation system or any potentially new evaluation system. District systems across the country are adopting new educator evaluation systems and early implementation studies are emerging to inform how teacher buy-in, transparency, type of feedback on instruction, perceived fairness, and consequences of the evaluation systems affects teachers and teaching (Donaldson & Papay, 2012; Humphrey et al., 2012; White, Cowhy, Stevens, & Sporte, 2012). The role of students in contributing to teacher perceptions of their environment remains an area for further investigation. The charter schools and adjacent traditional public schools included in this study both serve overwhelmingly poor and minority students, many of who are English language learners. Their average achievement on state standardized tests differs, however, with higher percentages of the charter school students meeting or exceeding state standards than those at the traditional public school. Although charter school teachers reported higher levels of student engagement, we lack information on student recruitment and selection and student achievement in O pening the “black box” 19 the current analysis. Therefore, we cannot determine whether the charter schools are enrolling high- ability and highly motivated students at the expense of leaving low-achieving and disengaged students in traditional public schools. Alternatively, as students are teachers’ most proximate context (Little & McLaughlin, 1993), teachers’ willingness to expend extra effort to support their students may be motivated by their perceptions of students’ commitment to their own learning. Indeed, teachers’ sense of responsibility for student learning and reported use of teaching advanced skills such as solving problems with multidisciplinary approaches has been related to students’ positive attitudes towards academics and effort-based learning (Young et al., 2010a). To the extent that students who actively choose to attend charter schools differ in motivation from those attending their neighborhood schools by default, charter school students may be part of a reinforcing circle. A high expectations culture is manifested in teacher attitudes and supports for students, who then internalize these expectations for academic seriousness and exhibit behavior that reflects the high expectations culture. Several limitations are necessary to consider when interpreting these findings. First, the teachers in this study were recruited from rural Texas, which limit the generalizability of the findings to all charter school teachers in the United States. Second, although propensity score matching balanced the observed teacher characteristics between charter school teachers and their peers in adjacent traditional public, it cannot balance unmeasurable characteristics such as teacher commitment, motivation to teach, and instructional skills. Third, due to the fact that students and their families elect to enroll in charter schools, students in charter schools may be different from their peers in the adjacent traditional public schools on unmeasurable characteristics such as motivation and parental support. These student differences may also contribute to the differences in charter school and traditional public school teachers’ responses. Despite these limitations, this study represents one of the first to test a variety of organizational factor differences between charter schools and traditional public schools using a rigorous quasi-experimental design. Our findings have implications for policymakers and educators, especially in light of how and why charter schools are different from traditional public schools along dimensions previously found to relate to student achievement. Conclusions Charter and traditional public schools in our study compete in a local labor market for effective teachers. Findings from this study provide valuable insights into the key school organizational factors that charter management organizations and traditional public schools need to strengthen to achieve sustainable growth. Teachers at charter schools and traditional public schools differed in the characteristics they reported as strong working conditions. Compared with similar teachers in traditional public schools, charter school teachers reported a more supportive teaching environment, higher expectations of students among staff, a greater sense of responsibility for student learning, and higher levels of student engagement in learning. These findings suggest that, to compete with charter management organizations to attract and retain high quality teachers, traditional public school districts may need to focus efforts on fostering school environments that embody these characteristics. Moreover, to the extent that individual teachers hone their skills in engaging students in content and evolve their expectations for E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 20 student learning, districts may want to examine how effectively they recruit for and develop such attributes among their staff. Charter school teachers, however, also reported less access to high quality professional development, less collaboration with colleagues, and perceived the teacher evaluation system as less fair in comparison to similar teachers in traditional public schools. The environment from charter school teachers’ perspectives then is one of focus on student learning but potentially in isolation and with inadequate supports. Such a picture may not be surprising. When charter management organizations grow quickly, central capacity and infrastructure to support schools can be strained, systems such as teacher professional development become more critical when hiring at scale, and the addition of new grades and new staff each year as the schools grow to full enrollment can destabilize teachers’ sense of collegiality. Thus, to attract and retain teachers, charter schools may need to provide their staff with more instructional support and professional development, schedule more opportunities to collaborate with their colleagues, and build a more transparent, useful, and fair teacher evaluation system. With knowledge on how charter school and traditional public school teachers’ perceptions of school organizational factors differ, policymakers and school leaders can strategically focus their efforts on addressing workplace conditions that may underlie teacher retention and student achievement, and represent unmet needs among teachers. References Abdulkadiroglu A., Angrist, J., Kane, T., & Pathak, P. (2011). Accountability and flexibility in public schools: Evidence from Boston's charters and pilots. Quarterly Journal of Economics, 126(2), 699-748. http://dx.doi.org/10.1093/qje/qjr017 Angrist, J. D., Dynarski, S. M., Kane, T. J., Pathak, P. A. & Walters, C. (2010). Who benefits from KIPP? Journal of Policy Analysis and Management, 31(4), 837-860. http://dx.doi.org/10.1002/pam.21647 Austin, P. C. (2008). A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003. Statistics in Medicine, 27, 2037-2049. http://dx.doi.org/10.1002/sim.3150 Baker, E. L., Barton, P. E., Darling-Hammond, L., Haertel, E., Ladd, H. F., Linn, R. L., … Shepard, L. A. (2010). Problems with the use of the student test scores to evaluate teachers (Research Report No. 278). Washington, DC: Economic Policy Institute. Retrieved on March 18, 2012 from http://www.epi.org/page/-/pdf/bp278.pdf Baker, B. D. & Dickerson, J. L. (2006). Charter schools, teacher labor market deregulation, and teacher quality: Evidence from the Schools and Staffing Survey. Education Policy, 20(5), 752-778. http://dx.doi.org/10.1177/0895904805284118 Baser, O. (2006). Too much ado about propensity score models? Comparing methods of propensity score matching. Value in Health, 9(6), 377-385. http://dx.doi.org/10.1111/j.1524-4733.2006.00130.x Bifulco, R. & Ladd, H. F. (2006). The impact of charter schools on student achievement: Evidence from North Carolina. Education Finance and Policy, 1(1), 50-90. http://dx.doi.org/10.1162/edfp.2006.1.1.50 Betts, J. & Hill, T. (2006). Key issues in studying charter schools and achievement: A review and suggestions for national guidelines. Seattle: Center on Reinventing Public Education. O pening the “black box” 21 Bomotti, S., Ginsberg, R., & Cobb, B. (1999). Teachers in charter schools and traditional schools: A comparative study. Education Policy Analysis Archives, 7(22), 1-22. Booker, K., Gilpatric, S., Gronberg, T., & Jansen, D. (2008). The effect of charter schools on traditional public school students in Texas: Are children who stay behinid left behind? Journal of Urban Economics, 64(1), 123–145. http://dx.doi.org/10.1016/j.jue.2007.10.003 Booker, K., Sass, T. R., Gill, B., and Zimmer, R. (2011). The effects of charter high schools on educational attainment. Journal of Labor Economics, 29(2), 377-415. http://dx.doi.org/10.1086/658089 Borman, G. D., & Dowling, N. M. (2008). Teacher attrition and retention: A meta-analytic and narrative review of the research. Review of Educational Research, 78(3), 376-409. http://dx.doi.org/10.3102/0034654308321455 Burian-Fitzgerald, M., Luekens, M. T., & Strizek, G. A. (2004). Less red tape or more green teachers: Charter school autonomy and teacher qualifications. In K. Bulkley & P. Wohlstetter (Eds.), Taking Account of Charter Schools: What's Happened and What's Next (pp. 11-31). New York: Teachers College Press. Boyd, D., Grossman, P., Ing, M., Lankford, H., Loeb, S., & Wyckoff, J. (2011). The influence of school administrators on teacher retention decisions. American Educational Research Journal, 48(2), 303-333. http://dx.doi.org/10.3102/0002831210380788 Bryk, A. S. & Schneider, B. (2002). Trust in schools: A score resource for improvement. New York: Russell Sage Foundation. Bulkley, K., Fisler, J. (2003). A Decade of Charter Schools: from Theory to Practice. Educational Policy, 17(3), 317-342. http://dx.doi.org/10.1177/0895904803017003002 Buckley, J., Schneider, M. & Shang, Y. (2004). The effects of school facility quality on teacher retention in urban school districts. Washington, DC: National Clearinghouse for Educational Facilities. Retrieved on March 18, 2012 from http://www.edfacilities.org/pubs/teacherretention.html Campbell, A., Gallagher, A.H., Greene, L., & Yee, K. (2011). National study of the teacher incentive fund [Survey]. Prepared for the U.S. Department of Education. Menlo Park, CA: SRI International. Cannata, M. (2007). Teacher community in elementary charter schools. Education Policy Analysis Archives, 15(11). Retrieved on November 18, 2012, from http://epaa.asu.edu/epaa/v15n11/. Cannata, M. & Peñaloza, R. (2012). Who are charter school teachers? Comparing teacher characteristics, job choices, and job preferences. Education Policy Analysis Archives, 20(29). Retrieved on November 18, 2012, from http://epaa.asu.edu/ojs/article/view/1021 Center for Applied Research and Educational Improvement (1997). Minnesota charter schools evaluation: Interim report. Minneapolis: The University of Minnesota. Colorado Department of Education (1996). Colorado charter school information packet and handbook. Denver: Author. Connors, A. F. J., Speroff, T., Dawson, N. V., Thomas, C., Harrell, F. E. J., Wagner, D.,…Knaus, W.A. (1996). The effectiveness of right heart catheterization in the initial care of critically ill patients. Support investigators. Journal of the American Medical Association, 276(11), 889-897. http://dx.doi.org/10.1001/jama.1996.03540110043030 E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 22 Center for Research on Education Outcomes (CREDO) (2009). Multiple choice: Charter School Performance in 16 States. Retrieved on March 7, 2012, from http://credo.stanford.edu/reports/MULTIPLE_CHOICE_CREDO.pdf Darling-Hammond, L. (2003). Keeping good teachers: Why it matters, what leaders can do. Educational Leadership, 60(8), 6–13. DeGrow, B. (2009). Texas Charter School Wait List Doubles in One Year. Retrieved on June 18, 2013, from http://news.heartland.org/newspaper-article/2009/12/24/texas-charter- school-wait-list-doubles-one-year Dehejia, R. H., & Wahba, S. (1990). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association, 94(448), 1053-1062. http://dx.doi.org/10.1080/01621459.1999.10473858 Dobbie, W., & Fryer, R. G. (2009). Are high quality schools enough to close the achievement gap? Evidence from a social experiment in Harlem (Working Paper No. 15473). Cambridge, MA: The National Bureau Economic Research. Retrieved on March 7, 2012, from http://www.nber.org/papers/w15473.pdf?new_window=1 Donaldson, M. L., & Papay, J. P. (2012). Reforming Teacher Evaluation: One District's Story. Washington, DC: Center for American Progress. Retrieved on September 19, 2013, from http://eric.ed.gov/?id=ED539748. Donaldson, M. L., & Peske, H. G. (2010). Supporting effective teaching through teacher evaluation: A study of teacher evaluation in five charter schools. Washington, DC: Center for American Progress. Retrieved on September 19, 2013, from http://www.americanprogress.org/wp- content/uploads/issues/2010/03/pdf/teacher_evaluation.pdf Dressler, B. (2001). Charter school leadership. Education and Urban Society, 33(2), 170-185. http://dx.doi.org/10.1177/0013124501332006 Eberts, R. W. & Hollenbeck, K. M. (2002). Impact of charter school attendance on student achievement in Michigan (Working Paper No. 02-080). Kalamazoo, MI: W. E. Upjohn Institute for Employment Research. Retrieved on September 19, 2013, from http://research.upjohn.org/cgi/viewcontent.cgi?article=1097&context=up_workingpap ers Elmore, R. F. (2000). Building a new structure for school leadership. Washington, DC: Albert Shanker Institute. Finn, C., Manno, B., and Bierlein, L., and Vanourek, G. (1997). Charter schools in action: Final report. Washington, DC: Hudson Institute. Furgeson, J., Gill, B., Haimson, J., Killewald, A., McCullough, M., Nichols-Barrer, I.,…Lake, R. (2012). Charter-school management organizations: Diverse strategies and diverse student impacts. Princeton, NJ and Seattle, WA: Mathematica Policy Research and the University of Washington’s Center on Reinventing Public Education. Gallagher, L. P., Means, B., & Padilla, C. (2008). Teachers’ Use of Student Data Systems to Improve Instruction: 2005 to 2007. Washington, D.C.: Office of Planning, Evaluation and Policy Development, U.S. Department of Education. Gill, B., Timpane, M., Ross, K., & Brewer, D. (2002). What we know and what we need to know about vouchers and charter schools. Santa Monica: RAND. O pening the “black box” 23 Gleason, P., Clark, M. Tuttle, C. C., & Dwoyer, E. (2010). The evaluation of charter school impacts: Final report. Washington, DC: National Center for Education Evaluation and Regional Assistance. Goddard, R. D., Hoy, W. K., & Hoy, A. W. (2004). Collective efficacy: Theoretical developments, empirical evidence, and future directions. Educational Researcher, 33(3), 3- 13. http://dx.doi.org/10.3102/0013189X033003003 Greenwald, R. Hedges, L. V., & Laine, R. D. (1996). The effect of school resources on student achievement. Review of Educational Research, 66(3), 361-396. http://dx.doi.org/10.3102/00346543066003361 Gruenert, S. (2005). Correlations of collaborative school cultures with student achievement. NASSP Buletin, 89(645), 43-55. Gruenert, S. (2005). Correlations of collaborative school cultures with student achievement. NASSP Buletin, 89(645), 43-55. http://dx.doi.org/10.1177/019263650508964504 Gum, P. A., Thamilarasan, M., Watanabe, J., Blackstone, E. H., & Lauer, M. S. (2001). Aspirin use and all-cause mortality among patients being evaluated for known or suspected coronary artery disease: A propensity analysis. Journal of the American Medical Association, 286(10), 1187-1194. http://dx.doi.org/10.1001/jama.286.10.1187 Hallinger, P., Bickman, L. & Davis, K. (1996). School context, principal leadership and student reading achievement. The Elementary School Journal, 96(5), 527-549. http://dx.doi.org/10.1086/461843 Hanushek, E. A. (1997). Assessing the effects of school resources on student performance: An update. Educational Evaluation and Policy Analysis, 19(2), 141-164. Hanushek, E. A., Kain, J. F. & Rivkin, S. G. (2002). The Impact of Charter Schools on Academic Achievement. Stanford, CA: Stanford University, Hoover Institute. Hirsch, E., Emerick, S., Church, K., & Fuller, E. (2007). Teacher working conditions are student learning conditions: A report on the 2006 North Carolina teacher working conditions survey. Hillsborough, NC: Center for Teaching Quality. Retrieved June 10, 2013, from http://ncteachingconditions.org/sites/default/files/attachments/twcnc2006.pdf Ho, D., Kosuke, I. Gary, K., & Stuart, E (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15, 199-236. http://dx.doi.org/10.1093/pan/mpl013 Hoxby, C. M., (2002). Would school choice change the teach profession? Journal of Human Resources, 37(4), 846-891. http://dx.doi.org/10.2307/3069619 Hoxby, C. M., & Murarka, S. (2009). Charter schools in New York City: Who enrolls and how they affect student achievement (Working Paper No. 14852). Cambridge, MA: The National Bureau Economic Research. Retrieved June 10, 2013, from http://www.nber.org/papers/w14852.pdf?new_window=1 Hoxby, C. M., & Rockoff, J.E. (2005). The impact of charter schools on student achievement. Harvard Institute of Economic Research Working Paper Series Paper. Humphrey, D. C., Gallagher, H. A., Yee, K. M., Goss, G. K., Campbell, A. Z., Cassidy, L. J., & Mitchell, N. M. (2012). Teacher Incentive Fund: First Implementation Report, 2006 and 2007 Grantees. Washington, D.C.: U.S. Department of Education. Retrieved on September 19, 2013 from http://www.sri.com/sites/default/files/publications/tif_1st- implementationreport2012.pdf E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 24 Jacob, B. A., & Rockoff, J. E. (2011). Organizing schools to improve student achievement: Start times, grade configurations, and teacher assignments. Washington DC: The Hamilton Project at the Brookings Institution. Johnson, S. M. & Birkeland, S. E. (2003). Pursuing a “sense of success”: New teachers explain their career decisions. American Educational Research Journal, 40(3), 581-617. http://dx.doi.org/10.3102/00028312040003581 Keating, N. L., Weeks, J. C., Landrum, M. B., Borbas, C., & Guadagnoli, E. (2001). Discussion of treatment options for early-stage breast cancer: Effect of provider specialty on type of surgery and satisfaction. Medical Care, 39(7), 681-691. http://dx.doi.org/10.1097/00005650-200107000-00005 Kellor, E. M. (2005). Catching up with the Vaughn express: Six years of standards-based teacher evaluation and performance pay. Education Policy Analysis Archives, 13(7). Retrieved on June 10, 2013, from http://epaa.asu.edu/epaa/v13n7./ Klassen, R. M., & Chiu, M. M. (2010). Effects on teachers’ self-efficacy and job satisfaction: Teacher gender, years of experience, and job stress. Journal of Educational Psychology, 102(3), 741-756. http://dx.doi.org/10.1037/a0019237 Ladd, H. F. (2011). Teachers’ perceptions of their working conditions: How predictive of planned and actual teacher movement? Educational Evaluation and Policy Analysis, 33(2), 235-261. http://dx.doi.org/10.3102/0162373711398128 Leonard, J. (2002). The case of the first-year chater school. Urban Education, 37(2), 219-240. http://dx.doi.org/10.1177/0042085902372004 Little, J. W., and McLaughlin, M.W., eds. (1993). Teachers' work: Individuals, colleagues, and contexts. New York: Teachers College Press. Loeb, S., Darling-Hammond, L., & Luczak, J. (2009). How teaching conditions predict teacher turnover in California schools. Peabody Journal of Education, 80(3), 44-70. Loeb, S., Darling-Hammond, L., & Luczak, J. (2009). How teaching conditions predict teacher turnover in California schools. Peabody Journal of Education, 80(3), 44-70. http://dx.doi.org/10.1207/s15327930pje8003_4 Lunceford, J. K. & Davidian, M. (2004). Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Statistics in Medicine, 23(19), 2937-2960. http://dx.doi.org/10.1002/sim.1903 Malloy, C. L. & Wohlstetter, P. (2003). Working conditions in charter schools: What is the appeal for teachers? Education and Urban Society, 35(2), 219-241. http://dx.doi.org/10.1177/0013124502239393 Milanowski, A. T. & Heneman, H. G. (2001). Assessment of teacher reaction to a standard- based teacher evaluation. Journal of Personnel Evaluation in Education. 15(3), 193-212. http://dx.doi.org/10.1023/A:1012752725765 Mojtabai, R., & Zivin, J. G. (2003). Effectiveness and cost-effectiveness of four treatment modalities for substance disorders: A propensity score analysis. Health Services Research, 38(1, Part 1), 233-259. http://dx.doi.org/10.1111/1475-6773.00114 National Center for Education Statistics (2011). 2011 Digest of Education Statistics, Table 106. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. O pening the “black box” 25 Nguyen, A. N., Taylor, J., & Bradley, S. (2006). The estimated effect of Catholic schooling on educational outcomes using propensity score matching. Bulletin of Economic Research, 58(4), 285-307. Ni, Y. (2012). Teacher working conditions in charter schools and traditional public schools: A comparative study. Teacher College Record, 114(03), 1-21. Peterson, M., Anjewierden, J., & Corser, C. (2001). Designing an effective concurrent enrollment program: A focus on quality of instruction and student outcomes. New Directions for Community Colleges, 113, 23-32. http://dx.doi.org/10.1002/cc.5 Racine, D. (2003). Dissolving Dualities: The Case for Commonsense Replication. Nonprofit and Voluntary Sector Quarterly, 32(2), 307-314. http://dx.doi.org/10.1177/0899764003032002009 Rosenbaum, P., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55. http://dx.doi.org/10.1093/biomet/70.1.41 Rosenbaum, P., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79(387), 516-524. http://dx.doi.org/10.1080/01621459.1984.10478078 RPP International. (1999). The state of charter schools, third year report: National Study of Charter Schools. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement. RPP International. (2000). The state of charter schools: 2000. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement. Skaalvik, E. M., & Skaalvik, S. (2007). Dimensions of teacher self-efficacy and relations with strain factors, perceived collective teacher efficacy, and teacher burnout. Journal of Educational Psychology, 99(3), 611–625. http://dx.doi.org/10.1037/0022-0663.99.3.611 Smith, T. M. & Ingersoll, R. M. (2004). What are the effects of induction and mentoring on beginning teacher turnover? American Educational Research Journal, 41(3), 681-714. http://dx.doi.org/10.3102/00028312041003681 Spillane, J. P., Halverson, R. & Diamonds, J. B. (2001). Investigating school leadership practice: A distributed perspective. Educational researcher, 30(3): 23-28. http://dx.doi.org/10.3102/0013189X030003023 SRI International (1999). SRI statewide teacher survey for teaching and California’s future: The status for the teaching profession. Santa Cruz: The Center for the Future of Teaching and Learning. Texas Association of School Boards (2009). Charter Schools in Texas: Facts and Figures. Retrieved on June 10, 2013, from http://www.tasb.org/legislative/legislative/update/documents/charters.pdf Titus, M. A. (2007). Detecting selection bias, using propensity score matching, and estimating treatment effects: An application to the private returns to a master's degree. Research in Higher Education, 48(4), 487-521. http://dx.doi.org/10.1007/s11162-006-9034-3 Tschannen-Moran, M., & Hoy, A. W. (2001). Teacher efficacy: Capturing an elusive construct. Teaching and Teacher Education, 17, 783–805. http://dx.doi.org/10.1016/S0742- 051X(01)00036-1 Wechsler, M. E., Caspary, K., Humphrey, D.C., & Matsko, K.K. (2010). Teacher induction in the Midwest. Menlo Park, CA: SRI International. E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 26 What Works Clearinghouse. (2008). Procedures and standards handbook (Version 2.0). Washington, DC: US Department of Education, Institute for Education Sciences. White, B. R., Cowhy, J., Stevens, W. D., & Sporte, S. E. (2012). Designing and Implementing the Next Generation of Teacher Evaluation Systems Lessons Learned from Case Studies in Five Illinois Districts. Chicago, IL: Consortium on Chicago School Research. Retrieved on September 19, 2013, from http://eric.ed.gov/?id=ED542563 Woodworth, K. R., David, J. L., Guha, R., Wang, H., & Lopez-Torkos, A. (2008). San Francisco Bay Area KIPP schools: A study of early implementation and achievement. Final report. Menlo Park, CA: SRI International. Young, V., Adelman, N., Bier, N., Cassidy, L., House, A., Keating, K., et al. (2010a). Evaluation of the Texas High School Project. First comprehensive annual report. Austin, TX: Texas Education Agency. Young, V., Adelman, N., Bier, N., Cassidy, L., Keating, K., Padilla, C., et al., (2010b). Evaluation of the Texas High School Project. Second comprehensive annual report. Austin, TX: Texas Education Agency. Young, V., Adelman, N., Cassidy, L., Goss, K., House, A., Keating, K., et al., (2011). Evaluation of the Texas High School Project. Third comprehensive annual report. Austin, TX: Texas Education Agency. Young, V., Humphrey, D., Wang, H., Bosetti, K., Cassidy, L., Wechsler, M., et al. (2009). Renaissance Schools Fund-supported schools: Early outcomes, challenges, and opportunities. Menlo Park, CA and Chicago, IL: SRI International and Consortium on Chicago Schools Research. Zimmer, R., Buddin, R., Chau, D., Daley, G., Gill, B., Guarino, C., Hamilton, L., Krop, C., McCaffrey, D., Sandler, M., & Brewer, D. (2003). Charter school operations and performance: Evidence from California. Santa Monica, CA: RAND Corporation O pening the “black box” 27 Appendix A Survey Items Working conditions Teacher reported overall school leadership (Young et al., 2010a, 2010b) 1. Setting high standards for teaching. 2. Setting high standards for student learning. 3. Making expectations for meeting instructional goals clear to the staff. 4. Knowing what's going on in my classroom. 5. Ensuring that the school runs smoothly. 6. Inspiring the very best in the job performance of all teachers. 7. Supporting regular use of student assessment data. 8. Promoting teachers' ongoing professional development (including the development of teacher professional learning communities.) 9. identifying and implementing supports for improved student learning 10. Providing time and resources for teachers to collaborate and plan together. 11. Enforcing school rules for student conduct and backing teachers up when they need it 12. Teachers in my school trust the school administration Teacher reported distributed school leadership (Young et al., 2010a, 2010b) 1. Teachers are involved in making the important decisions 2. Teachers have a lot of informal opportunities to influence what happens (e.g. offering suggestions or bringing up new ideas to the principal/school leader). 3. Teachers are encouraged to express their opinions without fear of criticism or retaliation. Teacher reported frequency of collaboration with colleagues (Young et al., 2010a, 2010b) 1. Observe each other's classroom instruction 2. Share ideas on teaching. 3. Discuss what was learned at a workshop or conference. 4. Share and discuss student work. 5. Discuss strategies for teaching and learning. 6. Share and discuss research on effective teaching methods. 7. Plan lessons and units together in a formal meeting structure. 8. Discuss student assessment data with other teachers to make instructional decisions. Teacher reported climate of high expectations (Young et al., 2010a, 2010b) 1. Teachers set high standards for teaching. 2. Teachers are continually seeking new ideas about teaching and learning in the classroom. 3. Most teachers work very hard to make sure that all students are learning. Supportive teaching environment (Wechsler et al., 2010) 1. Teachers in this school trust each other. 2. Teachers feel supported by colleagues to try new ideas. 3. Teachers in this school feel responsible to help each other do their best. 4. The school makes a conscious effort to make new teachers feel welcome. E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 28 Need for support (Wechsler et al., 2010) 1. The subject matter/content I teach. 2. Instructional techniques appropriate for the grade level/subject matter I teach. 3. Classroom management techniques appropriate for the students I currently teach. 4. The use of textbooks or other curricular materials for my current position 5. The use of data (e.g. analyzing student work or student test scores) to plan instruction 6. Adapting instruction to meet the needs of students at varying academic levels. 7. Planning lessons and designing instruction. 8. Creating a positive learning environment. 9. The use of informal and formal assessment strategies. 10. Evaluating and reflecting upon my own teaching practices. Teacher reported access to PD (Young et al., 2010a, 2010b) 1. Created or reflected on individual professional development plans with the assistance of the school leadership (e.g., principal, lead teachers). 2. Participated in professional development during regularly scheduled time during the school day. 3. Had opportunities to work productively with teachers from other schools. Teacher reported frequency of participating in high quality PD (Young et al., 2010a, 2010b) 1. Attended professional development that has been sustained and coherent, rather than disconnected. 2. Attended professional development that was closely connected to my school’s improvement plan. 3. Attended professional development that built on my previous knowledge. 4. Attended subject-matter-specific professional development. Availability of materials (Wechsler et al., 2010) 1. I have the necessary textbooks and print resources to teach. 2. I can get instructional materials (e.g. lab supplies, math manipulatives, classroom library books) without buying them myself. 3. I can get the classroom supplies (e.g. paper, pencils, staples, tape) I need without buying them myself. 4. I have curriculum with a clear scope and sequence that aligns with state and district content standards. 5. I have sufficient technology (e.g. graphic calculators, computers, science equipment, AV equipment) for all students to access and the equipment is reliable for instructional purposes) Instruction PD focus on instruction (Wechsler et al., 2010) 1. The subject matter/content I teach. 2. Instructional techniques appropriate for the grade level/subject matter I teach. 3. Classroom management techniques appropriate for the students I currently teach. 4. The use of textbooks or other curricular materials for my current position 5. The use of data (e.g. analyzing student work or student test scores) to plan instruction 6. Adapting instruction to meet the needs of students at varying academic levels. O pening the “black box” 29 7. Planning lessons and designing instruction. 8. Creating a positive learning environment. 9. The use of informal and formal assessment strategies. 10. Evaluating and reflecting upon my own teaching practices. Teacher reported sense of responsibility for student learning (Young et al., 2010a, 2010b) 1. Teachers make their expectations for meeting instructional goals clear to students 2. Teachers use of data and/or student work to carefully track students' academic progress 3. Most teachers believe that all students in this school can do well academically. Teacher reported frequency of reaching advanced skills (Young et al., 2010a, 2010b) 1. Evaluate and defend their ideas or views. 2. Orally present their work to peers, staff, parents, or others. 3. Work on multidisciplinary projects. 4. Tackle a problem with no obvious solutions or with multiple approaches 5. Invent or design a product or process that applies key concepts of the class 6. Use evidence to support their ideas. 7. Report on or paraphrase a single text. 8. Clearly state a main idea, thesis, or argument. 9. Demonstrate original thought, ideas, or analysis. 10. Consider multiple solutions or perspectives. 11. Synthesize information from multiple sources. 12. Complete a sequence of logical steps necessary to reach a conclusion 13. Present their own examples. Teacher reported responsiveness to student differences (Young et al., 2010a, 2010b) 1. Encouraged high-achieving students to do additional advanced work. 2. Attempted to assess students’ problem-solving processes, not just answers. 3. Adjusted instructional strategies to respond to students’ levels of understanding. 4. Modified my lesson significantly to meet students’ needs. Teacher reported student engagement in learning (Young et al., 2010a, 2010b) 1. Regularly pay attention in class. 2. Actively participate in class activities. 3. Always turn in their homework. 4. Care about how well they do in this class. Teacher reported use of data for instructional purposes (Young et al., 2010a, 2010b) 1. Set goals for individual student achievement. 2. Modify instructional strategies. 3. Select instructional materials. 4. Track students’ academic progress. 5. Arrange for remediation, tutoring, or special instruction for students. Teacher job satisfaction (Wechsler et al., 2010) 1. The freedom I have to teach. 2. School policies that support my work. 3. My profession (e.g. thinking ahead 3 years, I am sure I will still be in the education field). E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 30 4. My professional development opportunities. 5. The caliber of my colleagues. 6. The leadership opportunities at my school. 7. The support I receive from parents/guardians in teaching their children. 8. The support I receive from my principal/school leader to be a successful teacher. Self-efficacy (Wechsler et al., 2010) 1. I am confident in my ability to teach. (reworded) 2. I can handle challenging classroom management and discipline situations 3. If a student in my class becomes disruptive and noisy, I know techniques to redirect him/her quickly. 4. If a student does not remember information I gave in a previous lesson, I know how to increase his/her retention in the next lesson (reworded) 5. If one of my students couldn’t do a class assignment, I can accurately assess whether the assignment was at the correct level of difficulty. 6. I am able to adapt my instruction so I meet the needs of students at varying academic levels equally well. 7. With additional effort, I can get through to even the most difficult or unmotivated students. (reworded) 8. I can provide an alternative explanation when students are confused 9. I can implement a variety of assessment strategies. Teacher evaluation (Campbell et al., 2011) 1. The formal evaluation I received was fair. 2. I have altered my instructional practice as a result of my formal evaluation. 3. I received useful feedback from my formal evaluation. 4. The teacher evaluation process is clearly communicated to teachers. 5. The formal evaluation process has helped establish common goals for student learning and teacher instruction at my school. 6. I received targeted professional development based on feedback from my evaluation O pening the “black box” 31 About the Authors Xin Wei SRI International xin.wei@sri.com Xin Wei, PhD, is a senior research analyst at SRI International, Menlo Park, California. Her research interests include education evaluation, statistical modeling, experimental designs, and psychometric theory, with a focus on disadvantaged populations. Wei has a doctorate in educational psychology, measurement and statistics, from Stanford University. Deepa Patel SRI International deepa.patel@sri.com Deepa Patel is a research analyst working with SRI International, Menlo Park, California. She is involved in research projects on K–12 education reform and teacher development. Patel holds a master’s degree in public policy from the University of California, Berkeley Goldman School of Public Policy, and a master’s degree in teaching from Fordham University. Viki Young SRI International viki.young@sri.com Viki M. Young is a senior researcher in the Education Division at SRI International. Her research spans high school and district reform, charter school scale-up, human capital development, and data use for instructional purposes. She holds a Ph.D. in education policy analysis from Stanford University. Acknowledgements The authors wish to thank Frances Miller and Paul Hu for their contribution to data collection and statistical analysis. The authors also wish to thank the anonymous reviewers and the editor for providing especially detailed and helpful comments on various drafts. Lastly, the authors thank the U.S. Department of Education for supporting this research. However, any opinions expressed are those of the authors and do not represent the positions or polices of the funding agency. E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 32 education policy analysis archives Volume 22 Number 3 January 20th, 2014 ISSN 1068-2341 Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non- commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or EPAA. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizona State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A2 (Brazil), SCImago Journal Rank; SCOPUS, SOCOLAR (China). Please contribute commentaries at http://epaa.info/wordpress/ and send errata notes to Gustavo E. Fischman fischman@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. O pening the “black box” 33 education policy analysis archives editorial board Editor Gustavo E. Fischman (Arizona State University) Associate Editors: Audrey Amrein-Beardsley (Arizona State University) Rick Mintrop, (University of California, Berkeley) Jeanne M. Powers (Arizona State University) Jessica Allen University of Colorado, Boulder Christopher Lubienski University of Illinois, Urbana- Champaign Gary Anderson New York University Sarah Lubienski University of Illinois, Urbana- Champaign Michael W. Apple University of Wisconsin, Madison Samuel R. Lucas University of California, Berkeley Angela Arzubiaga Arizona State University Maria Martinez-Coslo University of Texas, Arlington David C. Berliner Arizona State University William Mathis University of Colorado, Boulder Robert Bickel Marshall University Tristan McCowan Institute of Education, London Henry Braun Boston College Heinrich Mintrop University of California, Berkeley Eric Camburn University of Wisconsin, Madison Michele S. Moses University of Colorado, Boulder Wendy C. Chi* University of Colorado, Boulder Julianne Moss University of Melbourne Casey Cobb University of Connecticut Sharon Nichols University of Texas, San Antonio Arnold Danzig Arizona State University Noga O'Connor University of Iowa Antonia Darder University of Illinois, Urbana- Champaign João Paraskveva University of Massachusetts, Dartmouth Linda Darling-Hammond Stanford University Laurence Parker University of Illinois, Urbana- Champaign Chad d'Entremont Strategies for Children Susan L. Robertson Bristol University John Diamond Harvard University John Rogers University of California, Los Angeles Tara Donahue Learning Point Associates A. G. Rud Purdue University Sherman Dorn University of South Florida Felicia C. Sanders The Pennsylvania State University Christopher Joseph Frey Bowling Green State University Janelle Scott University of California, Berkeley Melissa Lynn Freeman* Adams State College Kimberly Scott Arizona State University Amy Garrett Dikkers University of Minnesota Dorothy Shipps Baruch College/CUNY Gene V Glass Arizona State University Maria Teresa Tatto Michigan State University Ronald Glass University of California, Santa Cruz Larisa Warhol University of Connecticut Harvey Goldstein Bristol University Cally Waite Social Science Research Council Jacob P. K. Gross Indiana University John Weathers University of Colorado, Colorado Springs Eric M. Haas WestEd Kevin Welner University of Colorado, Boulder Kimberly Joy Howard* University of Southern California Ed Wiley University of Colorado, Boulder Aimee Howley Ohio University Terrence G. Wiley Arizona State University Craig Howley Ohio University John Willinsky Stanford University Steve Klees University of Maryland Kyo Yamashiro University of California, Los Angeles Jaekyung Lee SUNY Buffalo * Members of the New Scholars Board E ducation P olicy A nalysis A rchives V ol. 22 N o. 3 34 archivos analíticos de políticas educativas consejo editorial Editor: Gustavo E. Fischman (Arizona State University) Editores. Asociados Alejandro Canales (UNAM) y Jesús Romero Morante (Universidad de Cantabria) Armando Alcántara Santuario Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Fanni Muñoz Pontificia Universidad Católica de Perú Claudio Almonacid Universidad Metropolitana de Ciencias de la Educación, Chile Imanol Ordorika Instituto de Investigaciones Economicas – UNAM, México Pilar Arnaiz Sánchez Universidad de Murcia, España Maria Cristina Parra Sandoval Universidad de Zulia, Venezuela Xavier Besalú Costa Universitat de Girona, España Miguel A. Pereyra Universidad de Granada, España Jose Joaquin Brunner Universidad Diego Portales, Chile Monica Pini Universidad Nacional de San Martín, Argentina Damián Canales Sánchez Instituto Nacional para la Evaluación de la Educación, México Paula Razquin UNESCO, Francia María Caridad García Universidad Católica del Norte, Chile Ignacio Rivas Flores Universidad de Málaga, España Raimundo Cuesta Fernández IES Fray Luis de León, España Daniel Schugurensky Arizona State University Marco Antonio Delgado Fuentes Universidad Iberoamericana, México Orlando Pulido Chaves Universidad Pedagógica Nacional, Colombia Inés Dussel FLACSO, Argentina José Gregorio Rodríguez Universidad Nacional de Colombia Rafael Feito Alonso Universidad Complutense de Madrid, España Miriam Rodríguez Vargas Universidad Autónoma de Tamaulipas, México Pedro Flores Crespo Universidad Iberoamericana, México Mario Rueda Beltrán Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Verónica García Martínez Universidad Juárez Autónoma de Tabasco, México José Luis San Fabián Maroto Universidad de Oviedo, España Francisco F. García Pérez Universidad de Sevilla, España Yengny Marisol Silva Laya Universidad Iberoamericana, México Edna Luna Serrano Universidad Autónoma de Baja California, México Aida Terrón Bañuelos Universidad de Oviedo, España Alma Maldonado Departamento de Investigaciones Educativas, Centro de Investigación y de Estudios Avanzados, México Jurjo Torres Santomé Universidad de la Coruña, España Alejandro Márquez Jiménez Instituto de Investigaciones sobre la Universidad y la Educación, UNAM México Antoni Verger Planells University of Amsterdam, Holanda José Felipe Martínez Fernández University of California Los Angeles, USA Mario Yapu Universidad Para la Investigación Estratégica, Bolivia O pening the “black box” 35 arquivos analíticos de políticas educativas conselho editorial Editor: Gustavo E. Fischman (Arizona State University) Editores Associados: Rosa Maria Bueno Fisher e Luis A. Gandin (Universidade Federal do Rio Grande do Sul) Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Brasil Jefferson Mainardes Universidade Estadual de Ponta Grossa, Brasil Paulo Carrano Universidade Federal Fluminense, Brasil Luciano Mendes de Faria Filho Universidade Federal de Minas Gerais, Brasil Alicia Maria Catalano de Bonamino Pontificia Universidade Católica-Rio, Brasil Lia Raquel Moreira Oliveira Universidade do Minho, Portugal Fabiana de Amorim Marcello Universidade Luterana do Brasil, Canoas, Brasil Belmira Oliveira Bueno Universidade de São Paulo, Brasil Alexandre Fernandez Vaz Universidade Federal de Santa Catarina, Brasil António Teodoro Universidade Lusófona, Portugal Gaudêncio Frigotto Universidade do Estado do Rio de Janeiro, Brasil Pia L. Wong California State University Sacramento, U.S.A Alfredo M Gomes Universidade Federal de Pernambuco, Brasil Sandra Regina Sales Universidade Federal Rural do Rio de Janeiro, Brasil Petronilha Beatriz Gonçalves e Silva Universidade Federal de São Carlos, Brasil Elba Siqueira Sá Barreto Fundação Carlos Chagas, Brasil Nadja Herman Pontificia Universidade Católica –Rio Grande do Sul, Brasil Manuela Terrasêca Universidade do Porto, Portugal José Machado Pais Instituto de Ciências Sociais da Universidade de Lisboa, Portugal Robert Verhine Universidade Federal da Bahia, Brasil Wenceslao Machado de Oliveira Jr. Universidade Estadual de Campinas, Brasil Antônio A. S. Zuin Universidade Federal de São Carlos, Brasil