Dialog, 18(1), 92-96 Copyright © 2015, ISSN: 1930-9325 Authors Note: The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A060021 to the University of Virginia – funding the National Center for Research on Early Childhood Education (NCRECE). The work of the first author was partially supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305B090002 to the University of Virginia. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. Correspondence concerning this manuscript should be addressed to Manuela Jiménez, T. Denny Sanford School of Social and Family Dynamics, Po Box 873701 Tempe, AZ 85287-3701. Email: Manuela.Jimenez@asu.edu RESEARCH-TO-PRACTICE SUMMARY Coach Reports of Teachers’ Practice: An Alternative for Assessing Teacher-Child Interactions in Early Childhood Classrooms Manuela Jimenez Arizona State University Jennifer LoCasale-Crouch University of Virginia There is a need for valid, cost-effective assessments of effective teacher practice that can be used as an alternative to observational assessments. This paper examines one alternative, coaches’ reports, by comparing ratings of teachers’ interactions with children using two measures: the Teacher Knowledge and Skills Scale (TKSS), a coach-rating measure, and the Classroom Assessment Scoring System (CLASS), an established observational measure. The study also examines the role that the coach-teacher relationship plays in coaches’ ratings. A sample of 162 early childhood teachers and 12 coaches implementing a coaching intervention participated in this study. Results show that coaches provided ratings similar to those of trained observers, but they also tended to rate higher the interactions of teachers with whom they have higher-quality relationships. The paper discusses the implications of these findings for the assessment process in early childhood education. Keywords: teacher assessment, teacher-child interactions, coaching, ratings, professional development. Assessment of teachers’ practice represents a key piece of the process to ensure the quality of early childhood education programs (Zaslow, Tout & Martinez-Beck, 2010). Observational instruments are often recommended for this assessment, in part due to their focus on specific teaching behaviors associated with children’s improved outcomes, since these behaviors can then be linked to policies and professional development meant to improve teaching practice (Kane, COACH REPORTS 93 Taylor, Tyler & Wooten, 2010; Pianta, 2006). However, considerable investments in aspects such as observer training, travel time and continuous reliability checks are needed to ensure the reliable use of these instruments, which can turn them into a costly, labor-intensive task. These barriers point to a need for valid, cost-effective assessments of effective teacher practice. The recent popularity of coaching as a support for early childhood teachers (Gupta & Daniels, 2012; Isner et al., 2011) presents coaches’ reports of teachers’ practice as an alternative assessment. Although there are different coaching models (Isner et al., 2011), most of them entail a cyclical process that involves planning, observations of teachers’ practice in the classroom, and reflection and feedback about the practice (National Center on Quality Teaching and Learning (NCQTL), 2012). As part of this process, coaches repeatedly observe teachers’ practice during extended periods of time, which means that coaches are already collecting information about teachers’ practice. Their regular observations can help them provide more reliable reports (Bill and Melinda Gates Foundation, 2012), thus serving as a potential cost-effective alternative to observational assessments. Coaches, however, can be susceptible to bias in their ratings due to outside influences, such as the quality of the coach-teacher relationship (Jacob & Lefgren, 2008; Ostrander, 1996). The constant interaction needed to ensure the quality of the coaching process can lead to close, high-quality relationships between coaches and teachers. Based on studies of supervisors’ ratings of subordinates (Lefkowitz, 2000), critics worry that coaches will give higher ratings to the teachers with whom they have higher-quality relationships. Research has shown, however, that other observers of teachers’ practice, such as principals and students, are able to provide valid assessments of teachers’ behavior when they use reliable instruments (Bill and Melinda Gates Foundation, 2012; Li, Hughes, Kwok, & Hsu, 2012). Still, a better understanding of how the coach-teacher relationship relates to coaches’ assessment of teachers’ practice is warranted to ensure the validity of their reports. Coaches’ reports of teachers’ practice can be a potentially valid alternative to costly observational assessments. However, before adopting these reports as part of teachers’ assessments there is a need to establish their validity and to assess the possible bias that the coach-teacher relationship can introduce to these ratings. Thus in order to examine the potential of coaches as reporters of teachers’ practice, this study first looked at the correspondence between coaches’ ratings of teachers’ practice and the ratings made by external observers using a standardized observational assessment. Next, this study examined the association between coaches’ ratings and the quality of their relationship with the teachers they rated. Lastly, and with the goal of further understanding the coach-teacher relationship, the study examined whether coaches’ accuracy (i.e. the degree of correspondence between coaches’ and observers’ ratings) varied depending on the quality of the coach-teacher relationship. THIS STUDY Study participants included 12 coaches and 152 teachers who were taking part in a larger study evaluating MyTeachingPartner (MTP; Pianta, Mashburn, et al., 2008), a web-mediated coaching model. MTP was developed using the Teaching Through Interactions framework (TTIF; Hamre et al., 2013), which served as the guiding framework for the coaching process. This study focused on ratings of a specific aspect of teachers’ practice, their interactions with children. The study compared two instruments developed using the TTI framework: the Teacher Knowledge 94 JIMENEZ & LOCASALE-CROUCH and Skills Scale (TKSS, LoCasale-Crouch & Hamre, 2008a), a coach-ratings measure; and the Classroom Assessment Scoring System (CLASS, Pianta, La Paro & Hamre, 2008), an observational assessment. These specific measures were chosen to increase the likelihood that any difference on the ratings given to teachers on these assessments was due to the fact that they were rated by different reporters. To examine the role that the coach-teacher relationship plays in coaches’ ratings, coaches were asked to rate the quality of their relationship with each of the teachers with whom they worked, using the Teacher-Coach Relationship Scale (LoCasale- Crouch & Hamre, 2008b). As part of the coaching process, teachers regularly filmed 30-minute long segments of their teaching practice. These videos were then sent to the study team. Coaches observed each video, provided feedback, and supported teachers’ analysis of their interactions with children through written prompts and a series of coach-teacher conferences. At the end of the coaching process, coaches rated teachers’ interactions with children based on their recollection of the videos using the TKSS. The videos were also independently reviewed by external observers trained to rate teachers’ interactions with children using the CLASS. Observers made their ratings immediately after viewing the teaching segment. KEY FINDINGS AND IMPLICATIONS Analyses of coaches’ and trained observers’ assessments of teachers’ practice show that both groups provided similar ratings of teacher-student interactions. Additionally, coaches tended to rate a teacher’s interactions with children higher when they had a higher-quality relationship with the teacher. Despite this, the accuracy of coaches’ ratings did not vary depending on their relationship with the teacher. In summary, coaches’ ratings corresponded well with observers’ ratings and this correspondence was not influenced by the quality of the coach-teacher relationship. This finding shows that coaches’ reports can be used as an additional assessment of early childhood teachers’ interactions with children by providing evidence that coaches using a standardized instrument, such as the TKSS, can provide relatively accurate ratings of teacher-child interactions. The study, however, also shows an association between coaches’ ratings and the coach-teacher relationship, in which coaches tend to score higher ratings for teacher-student interactions for those teachers with whom they have a higher quality relationship. Although the accuracy of coaches’ ratings was not found to be associated with the coach-teacher relationship, the previous findings highlight the need to consider the coach-teacher relationship when deciding to use coaches’ ratings of teacher-child interactions. One possible explanation for coaches’ tendency to provide higher ratings for the teachers with whom they have a higher-quality relationship may lie in the role that coaches’ affect towards teachers plays during their observations. Coaches’ affect can influence the type of information that coaches attend to during the observation, with coaches unintentionally paying more attention to information that matches their perception of the teacher (i.e. coaches pay more attention to effective teacher-child interactions when observing a teacher with whom they have a higher-quality relationship). In this case, coaches would not be using all the information available to rate teachers’ interactions with children. To minimize this issue, coaches could be required to follow a protocol for their observations meant to help them use all the information available to make their ratings. This protocol could involve asking coaches to take detailed notes COACH REPORTS 95 of their observations of the teaching behaviors assessed in the measure, which would allow coaches to base their ratings in specific interactions seen in the observation instead of their general impression after the observation. Another way to increase the precision of coaches’ ratings is to use them in combination with reports from other observers. Multisource assessments are more reliable than single reporter assessments (Bill and Melinda Gates Foundation, 2012; Li et al., 2012), and some centers already collect other ratings of teacher-child interactions such as teachers’ self-ratings (Howes et al., 2008). These reports, along with ratings from teacher aides or from other teachers, could be used to complement the coaches’ ratings. Since these ratings are already being collected they would not be an extra burden on the teachers, maintaining the cost-effectiveness of the measure. It is important, however, to keep in mind that a high-quality coach-teacher relationship is key for coaching’s effectiveness. Thus, the effect that asking coaches to rate their teachers’ interactions with children may have for the coaching process should be considered before deciding to use coaches’ ratings. To ensure that the use of coaches’ ratings does not affect the coaching process their goal should be to help teachers improve their practice in the classroom and to identify program- or center-wide issues that should be targeted in future professional development efforts. These goals should also be clearly and explicitly communicated to teachers. It should also be noted , however, that there are certain aspects of this study that can limit the generalizability of its findings. This study examined the ratings provided by coaches participating in MTP. This is only one of several different coaching approaches being currently implemented in early childhood education programs, and consideration should be given to any differences between MTP and the specific coaching model used in order to assess how to best employ these findings to inform decisions for each particular situation. The main difference that may limit the generalizability of this study’s findings is the fact that MTP is a web-mediated coaching model. These coaching models have been gaining popularity in the last few years but are still less common than in-person coaching models (Isner et al., 2011). Since most web-mediated coaching models use videos instead of live observations, coaches in the former models may find the rating process easier than coaches that do in-person observations. With videos, coaches can watch an interaction as often as they feel necessary, while coaches doing in-person observations have to be able to focus on specific interactions in the middle of an environment full of distractions that can burden their attention and cloud their ratings. Although findings from this study show that coaches and observers can provide similar ratings of teacher-children interactions when using video observations, there is a need for more research to examine whether this correspondence translates to ratings based on in-person observations. CONCLUSION Despite the aforementioned limitations, this study suggests that coaches can provide relatively accurate ratings of teachers’ practice. Coaches’ reports can be a useful addition to the toolbox of assessments of quality in early childhood education, and could be a reliable alternative to the more costly observational assessments. However, certain protocols should be adopted to ensure that these ratings provide useful, reliable information, and do not affect the coaching process. These protocols could strengthen the contribution that the use of coaches’ ratings can make to the assessment process in early childhood education. 96 JIMENEZ & LOCASALE-CROUCH REFERENCES Bill and Melinda Gates Foundation (2012). Gathering Feedback for Teachers. Combining High-Quality Observations with Student Surveys and Achievement Gains. Gupta, S. S., & Daniels, J. (2012). Coaching and professional development in early childhood classrooms: Current practices and recommendations for the future. NHSA Dialog, 15, 206-220. Hamre, B. K., Pianta, R.C., Downer, J. T., DeCoster, J., Mashburn, A.J., Jones, S., … Hamagami, A. (2013). Teaching through interactions: testing a developmental framework of effective teaching in over 4,000 classrooms. Elementary School Journal, 13, 461-487. doi: 0013-5984/2013/11304-000 Howes, C., Pianta, R., Bryant, D., Hamre, B., Downer, J., & Soliday-Hong, S. (2008). Ensuring effective teaching in early childhood education through linked professional development systems, quality rating systems and state competencies: The role of research in an evidence-driven system (pp. 1-16). Arlington, VA: National Center for Research on Early Childhood Education Jacob, B. A. & Lefgren, L. (2008). Can principals identify effective teachers? Evidence on subjective performance evaluation in education. Journal of Labor Economics, 26, 101-136. doi: 10.1086/522974 Isner, T., Tout, K., Zaslow, M., Soli, M., Quinn, K., Rothenberg, L., & Burkhauser, M. (2011). Coaching in early care and education programs and Quality Rating and Improvement Systems (QRIS): Identifying promising features. Report produced for the Children’s Services Council of Palm Beach County. Washington, DC: Child Trends. Kane, T. J., Taylor, E.S., Tyler, J. H., & Wooten, A. L. (2010). Identifying effective classroom practices using student achievement data. NBER working paper 15803. Cambridge, MA: National Bureau of Economic Research Lefkowitz, J. (2000). The role of interpersonal affective regard in supervisory performance ratings: A literature review and proposed causal model. Journal of Occupational and Organizational Psychology, 73, 67-85. doi: 10.1348/096317900166886 Li, Y., Hughes, J. N., Kwok, O. & Hsu, H. (2012). Evidence of convergent and discriminant validity of child, teacher, and peer reports of teacher-student support. Psychological Assessments, 24, 54-65. doi: 10.1037/a0024481 LoCasale-Crouch, J., & Hamre, B. K. (2008a). Teacher Knowledge and Skills Survey. Unpublished manuscript, Center for the Advanced Study of Teaching and Learning, University of Virginia. LoCasale-Crouch, J., & Hamre, B. K. (2008b). Teacher-Coach Relationship Scale. Unpublished manuscript, Center for the Advanced Study of Teaching and Learning, University of Virginia. National Center on Quality Teaching and Learning (2012). What do we know about coaching? Retrieved from http://eclkc.ohs.acf.hhs.gov/hslc/tta-system/teaching/docs/What-Do-We-Know-About-Coaching.pdf Ostrander, L. R. (1996). Multiple Judges of Teacher Effectiveness: Comparing Teacher Self-Assessments with the Perception of Principals, Students and Parents. Paper presented at the Annual Meeting of the American Educational Research Association (New York, NY, April 8-12). Retrieved from http://eric.ed.gov/?id=ED399267 Pianta, R.C. (2006). Standardized classroom observations from pre-k to 3 rd grade: A mechanism for improving access to consistently high quality classroom experiences and practices during the p-3 years. Discussion paper 103, Early Childhood Research Collaborative. Retrieved from http://humancapitalrc.org/dp_catalog/DP103.pdf Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). Classroom Assessment Scoring System (CLASS), Manual, Pre-K. Baltimore: Paul H. Brookes. Pianta, R. C., Mashburn, A. J., Downer, J. T., Hamre, B. K. & Justice, L. (2008). Effects of web-mediated professional development resources on teacher-child interactions in pre-kindergarten classrooms. Early Childhood Research Quarterly, 23, 431-451. doi:10.1016/j.ecresq.2008.02.001 Zaslow, M., Tout, K., & Martinez-Beck, I. (2010). Measuring the quality of early care and education programs at the intersection of research, policy, and practice. OPRE Research-to-Policy, Research-to-Practice Brief. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. http://eclkc.ohs.acf.hhs.gov/hslc/tta-system/teaching/docs/What-Do-We-Know-About-Coaching.pdf