Microsoft Word - Godfrey-Hurrell Final.doc Dialog, 21(1), Copyright © 2018, ISSN: 1930-9325 RESEARCH ARTICLE Effects of Coaching and Performance Feedback on Preschool Teachers’ Implementation of the Teaching Pyramid Model Kristi Godfrey-Hurrell Central Piedmont Community College Vivian Correa University of North Carolina at Charlotte Laura Truesdell Scuola Creativa Previous research has indicated that a strong connection exists between children’s social and emotional development and their academic performance in school. Yet, the literature has also shown that teachers may be inadequately prepared to help children in obtaining prosocial skills. The purpose of the current study is to describe a coaching model that was used with three child care center teachers as they implemented strategies connected with three targeted items from the Teaching Pyramid Model, a comprehensive framework for supporting children’s social and emotional development. Data were collected across four phases: pre- baseline, baseline, intervention, and maintenance. Results showed that using coaching plus performance feedback with teachers leads to increased levels of implementation of strategies associated with the Teaching Pyramid Model. Teachers also indicated positive attitudes toward the use of the coaching model. Additionally, child social and emotional outcome data showed improvement for four of six children included. Limitations of student and implications for future research are provided. Keywords: behavior, coaching, professional development, social and emotional skills, preschool Young children with social, emotional, and/or behavioral problems are at an increased risk for poor school and mental health outcomes. In fact, the prevalence rate of young children exhibiting challenging behaviors has been documented between 10-30% while those who experience mild MULTICOMPONENT PROFESSIONAL DEVELOPMENT 51 to moderate levels of behavioral issues ranges from 10-15%, both of which have shown to negatively affect cognitive functioning, overall development, and school readiness (Cooper, Masi, & Vick, 2009; Fox & Smith, 2007; Perry, Holland, Darling-Kuria, & Nadiv, 2011). Researchers have suggested that these problems persist across time, and children with early adaptation difficulties are at higher risk for continued struggles (American Academy of Pediatrics [AAP] Council on Early Childhood, AAP Committee on Psychosocial Aspects of Child and Family Health, AAP Section on Developmental and Behavioral Pediatrics, 2016; Fantuzzo, Bulotsky-Shearer, McDermott, McWayne, Frye, & Pearlman, 2007). Further, children who have social-emotional competency problems often display negative behaviors with peers and adults, find themselves socially isolated, and experience negative interactions before entering primary grade educational settings; of particular note, these behaviors have been found to first manifest within child care settings (Brown & Conroy, 2011; Perry et al., 2011). This often leads to an increased amount of child care teachers who struggle with identifying and meeting the social-emotional and behavioral needs of young children. Additionally, children with behavioral problems are at an increased risk of peer rejection and preschool expulsion. According to U.S. Department of Education, recent data revealed preschool expulsions and suspensions as happening regularly with more than 8,000 pubic preschoolers being suspended at least once in their young educational careers (Child Care Aware ® of America, 2015). Therefore, identification and intervention for children who may be at risk for social and emotional delays and/or behavioral problems could be key to positive child outcomes and increased school readiness. McCabe and Altamura (2011) reviewed assessments and intervention programs that have demonstrated empirical efficacy on the remediation of social, emotional, and behavioral problems in young children with a variety of disabilities and referral concerns. They found that interventions that targeted behavior problems, social skill delays, and emotion regulation deficits were the most effective. Ten programs were reviewed and among them was the Teaching Pyramid Model. McCabe and Altamura (2011) described the model as, “one designed to promote social-emotional development and ameliorate behavioral concerns in young children that includes components that work at all ecological levels of the young child” (p. 553). The model is similar to other multi-tiered approaches such as Positive Behavior Supports (PBS) and Multi- Tiered System of Support (MTSS). The pyramid framework includes four levels of practice: (a) positive relationships with children, families, and colleagues; (b) classroom preventive practices; (c) social and emotional teaching strategies; and (d) intensive individualized interventions (Fox, Dunlap, Hemmeter, Joseph, & Strain, 2003). The Teaching Pyramid Model assists in solving social-emotional and behavioral issues through the examination of adult behaviors, classroom practices, and adult-child relationships. According to Fox et al. (2003), this multi-tiered intervention model which examines teacher-child relationships, the classroom environment, social and emotional teaching strategies, and individualized interventions could have the greatest potential to help all children develop prosocial skills, while effectively addressing the most challenging child behaviors with which teachers struggle. Therefore, there is a need to investigate how to best support teachers in implementing such intervention models. Joyce and Showers (2003) provided support for the use of coaching when training teachers to use new strategies and interventions. Researchers found that teachers who are coached are more likely to practice new strategies more often and with greater skill, adapt strategies more appropriately to their own goals and environment, and retain and increase skill over time. Additionally, when compared to uncoached teachers, teachers who were coached were 52 GODFREY-HURRELL, CORREA, & TRUESDELL more likely to assist children in understanding the new strategies and were able to demonstrate a clearer understanding of why and how learned strategies or methods are used (Joyce & Shower, 2003). More recently, Schachter (2015) examined empirical research on the design, delivery, and measurement of the effects of professional development for early childhood teachers to provide understanding of what the field has done and give direction for future professional development. Coaching was found to be a popular method for delivering professional development and provided participants with a more intimate relationship that focused on individualized learning and education. One such examination from McCollum, Hemmeter, and Hsieh (2011) used a coaching and performance feedback approach to professional development with an experimental group of seven teachers. In that study, researchers looked at the effects of coaching on emergent literacy instruction. Results indicated significant positive effects as a result of the performance feedback offered to the teachers in the experimental group. Additionally, Fox, Hemmeter, Synder, Perez Binder, and Clarke (2011) explored the effects of coaching teachers on the implementation of strategies within the Teaching Pyramid Model. The authors found that the use of coaching and performance feedback had positive effects on the participants’ abilities to implement the teaching strategies associated with the Teaching Pyramid Model. Fox et al. (2011) stipulated that, not only was a framework such as the Teaching Pyramid Model effective in supporting all children’s social-emotional development and providing extra support for those children who exhibited challenging behaviors, it was critical in offering teachers the right kind of help in implementing the model. They further reported that the individualized coaching provided to the teachers in the study was an essential component in improving teacher performance, especially given the multifaceted nature of the model. Despite the seemingly positive outcomes of the Fox et al. (2011) study, it had limitations. It was one of the only studies of its kind on the Teaching Pyramid Model and did not address specific child outcomes related to social-emotional skills. It is evident that more research is needed to confirm the benefit of the framework for both teachers in the classroom and children receiving the supports needed to enhance their social and emotional competence and decrease behavioral concerns. The purpose of the current study was to extend research efforts showing the benefits of The Teaching Pyramid Model framework with teachers and children in the early childhood environment. This study also adds to the research on the need for continued professional development of teachers to ensure high quality intervention implementation by identifying how coaching can be used to support teachers. Therefore, the current study applied the Teaching Pyramid Model and coaching strategies to three teachers who worked in a child care program with little to no special education training. Specifically, researchers sought to answer the following: (1) To what extent will a multicomponent professional development model that includes coaching and performance feedback impact teachers’ implementation of three targeted skills associated with the Teaching Pyramid Model Observation Tool (TPOT)? (2) To what extent will teachers’ implementation of three targeted skills associated with the TPOT impact targeted children’s scores on the Ages and Stages Questionnaire- Social/Emotional (ASQ-SE)? (3) What will be the teachers’ perceptions of the multicomponent professional development model? MULTICOMPONENT PROFESSIONAL DEVELOPMENT 53 METHODS Participants and Setting This study took place in a private faith-based community child care program located in an urban setting that served children 6 weeks to 6 years of age. The program enrolled 100 children in a total of seven classrooms. Of the total enrolled, 5-10%, or 10 of the 100 children enrolled, of the program’s population had tuition subsidized by the state and were represented ethnically by 12 different countries. Each classroom had at least two teachers; teachers were 20% Caucasian and 80% African-American. This child care program was selected to participate because of the director’s interest in the study’s topic. Coaching sessions occurred within each teacher’s classroom and data were gathered within the same environment. Performance feedback and coaching took place in a separate room within the child care program and through email. Participants in this study included three preschool teachers working in the same faith- based community child care program and six children who were enrolled in the teachers’ classrooms. The participants met eligibility requirements. For example, all teachers completed two 2-hr trainings supported by the local Child Care Resource and Referral Agency in the first tier of the Teaching Pyramid Model, which covered universal practices, within a year prior to the start of the study. Additional criteria for involvement in the study included being open to performance feedback and individualized coaching, serving as a lead teacher in a classroom with children of at least 2 years of age, and working with a child with challenging behaviors in his or her classroom. Upon conclusion of the study, teachers were provided a $75 gift card as an incentive. Pseudonyms were assigned to each of the teachers and letters were assigned to each child. Participant 1. Mary was a 53-year-old African-American woman with an Associate’s degree in Early Childhood Education. At the time of the study, she had 30 years of teaching experience, all of which had occurred at the child care program that served as the study’s setting. She had 20 children enrolled in her classroom with one teaching assistant; all children were 3- year-olds. Participant 2. Sharon was a 29-year-old African-American woman with the state’s Early Childhood Credentials 1 and 2, and she was working toward an Associate’s degree in Early Childhood Education at a local community college. She had 13 years of teaching experience at the time of the study and was currently teaching 2-year-olds and had 18 children enrolled in her classroom with two teaching assistants. Participant 3. John was a 29-year-old African-American male with 14 years of teaching experience. He studied for two years at a four-year university, majoring in community recreation, and was working toward an Associate’s degree in Early Childhood Education at a local community college. His classroom consisted of 21 children who ranged in age from 4-6 years with one co-teacher and one assistant teacher. Child Participants. There were six children included in the study. Teachers were asked to choose two children from their classrooms who they believed had “challenging behaviors”. For the purpose of this study, “challenging behaviors” was defined as, “any repeated 54 GODFREY-HURRELL, CORREA, & TRUESDELL pattern of behavior that interferes with or is at risk of interfering with optimal learning or engagement in prosocial interactions with peers and adults” (Smith & Fox, 2003, p.7). Once identified, parent or guardian permission was secured, and teachers were asked to fill out the Ages & Stages Questionnaire-Social and Emotional (ASQ-SE; Squires, Bricker, & Twombly, 2002). All six children were between the ages of 32-53 months at the time of the study, four children were Caucasian and two were African-American. To protect child identity, children were labeled Child A through Child F; Child A and B were enrolled in Sharon’s classroom, Child C and D were enrolled in Mary’s classroom, and Child E and F were enrolled in John’s classroom. Pre ASQ-SE and cutoff scores were: Child B (40 months) 100 with a cutoff of 59, Child C (42 months) 65 with a cutoff of 59, Child D (52 months) 30 with a cutoff of 70, and Child F (53 months) 30 and a cutoff of 70 (see Table 1). The cutoff scores were used to establish need for intervention with regards to behavior. For example, if child scored above the cutoff, the child needs further evaluation and intervention, while a child who scored below the cutoff should be monitored and provided support (Squires et al., 2002). TABLE 1 Pre-test and Post-test Data on ASQ-SE Scores Participant Age in Months ASQ-SE Cutoff Baseline Post Intervention Child A 32 57 15 45 Child B 40 59 100 65 Child C 42 59 65 45 Child D 52 70 30 15 Child E 53 70 40 60 Child F 53 70 30 15 Design A single case, multiple probe across three teachers design (Hitchcock et al., 2014) was used to examine the effects of a multicomponent professional development intervention on the three teachers’ implementation of three targeted skills connected with the Teaching Pyramid Model. A single case, multiple probe design was used to document an observable change across participants; essentially to see if the implementation of the intervention was responsible for that change. Additionally, researchers used the single case design because of the small sample size and literature does support the use of this design with small sample sizes (Hitchcock et al., 2014). Further, this design allows researchers to observe the functional relation of the intervention through replication which allows for stronger statements to be made as to the reliability of the results. Therefore, the researchers implemented four phases: pre-baseline, baseline, intervention, and maintenance. Data were graphed, and decisions regarding when teachers moved through the phases were based on a visual analysis of that data. In essence, the researcher examined the data for the teacher variables according to the Kratochwill et al. (2010) features. Six features were examined within and between condition data patterns: (a) level, (b) trend, (c) variability, (d) immediacy of effect, (e) overlap, and (f) consistency of data patterns across similar conditions. The researcher was able to analyze data using such features as level, trend, and consistency of data patterns. These features were then used to analyze data of the research to determine if there was a functional relation. MULTICOMPONENT PROFESSIONAL DEVELOPMENT 55 During the baseline phase, the teacher demonstrating the most stable individualized score began intervention first while the other teachers remained in baseline. When the first teacher reached stability in intervention, the second lowest scoring teacher entered intervention. The same process continued until the final teacher entered intervention. Teachers were expected to reach an 80% criterion level before moving to maintenance phase. Dependent Variables Teacher-focused dependent variable. The primary dependent variable in this study involved measuring each teachers’ ability to implement three targeted skills connected with the Teaching Pyramid Model. The three targeted skills were chosen based on their overall scores on the Teaching Pyramid Observation Tool (TPOT; Hemmeter, Fox, & Snyder, 2014). The TPOT was used in the study to measure each teacher’s ability to foster children’s social and emotional development within the framework of the Teaching Pyramid Model. The TPOT is a standardized rating scale that involves a 2-hr observation (e.g., assessment observations) of a teacher in an early childhood classroom, and an interview with that teacher. The TPOT incorporates five types of items: observation only (items 1-9), observation and interview (items 10-12), interview only (items 13-15), red flags (items 16-31), and a challenging behavior item (this item is not assigned a number). The full TPOT, from which all 31 items are recorded, is scored after both the observation and interview have been completed. In the current study, the full TPOT was scored for each teacher after the first assessment observation. During the subsequent assessment observations, an individualized TPOT was used for each teacher, in which the three lowest-scoring items and their subsequent indicators were scored. Child-focused dependent variable. A second dependent variable in this study was the social and emotional outcomes for children who displayed challenging behaviors. Six children across all three classrooms were identified in order to determine if each child’s social and emotional development was positively affected by the implementation of teaching practices connected with the Teaching Pyramid Model. Children were chosen based on both teacher recommendation and parent permission. The teachers administered the ASQ-SE on each of their targeted children before intervention and upon completion of the maintenance phase; the pre- and post-ASQ-SE were scored by the primary researcher. The ASQ-SE was chosen because of its alignment with the framework of the Teaching Pyramid Model. It also has good reliability and validity results, is inexpensive, and is easy to use (Henderson & Strain, 2009). The ASQ-SE is a 10- to 15-minute assessment that is given in a questionnaire format. Data Collection and Analysis Data were collected using two versions of the TPOT to measure each teacher’s ability to implement three targeted skills connected with the Teaching Pyramid Model. The full TPOT was used to: (a) establish pre-baseline scores, (b) determine three low scoring items, and (c) to measure follow-up scores during a posttest phase (e.g., maintenance). A modified TPOT was then developed for each teacher based on the results from the full TPOT pretest during the pre- 56 GODFREY-HURRELL, CORREA, & TRUESDELL baseline phase. To do this the three lowest scoring items from the full TPOT pretest were identified for each teacher and these items and data were then presented to the teachers during their first individual coaching sessions. The research design utilized visual analysis of the data; reviewing any change in: (a) level, (b) trend, (c) variability, (d) immediacy of effect, (e) overlap, and (f) consistency of data patterns across conditions. As data were collected on research question one, they were entered into a graph where researchers were then able to visually examine these changes and determine similarity across participants under the same condition. Data for research question two were entered into a chart and compared from pre to post study to see if ASQ-SE scores changed. Information concerning teachers’ perceptions of the coaching model were also compared from pre to post study. Procedure Following recruitment and selection, researchers made an initial visit to the child care program to obtain written consent. After completing the consent forms, teachers were asked to select two children in their classrooms who exhibited challenging behaviors per the definition used in this study. Parents of the children identified signed consent forms for participation in the study. After receiving parental consent, researchers used the TPOT to conduct initial evaluations of teachers. Following pre-baseline and baseline, coaching sessions were conducted and videotaped within the program in a separate room for 6-13 sessions; each participant varied in amount of coaching session needed. Performance feedback was provided during the coaching sessions and in weekly emails sessions and were sent to each teacher between 6-13 sessions. Data were collected through assessment observations, which were implemented within each of the three teachers’ classrooms by the second researcher. Pre-baseline. During the first week, both researchers conducted a 2-hour full TPOT pretest on each teacher, which involved using and scoring the entire instrument. Observations took place during each teacher’s scheduled daily activities. For the full TPOT pretest, each teacher was observed for approximately 2 hours during a range of classroom routines, including large and small group activities, center activities, and transitions between activities. Baseline. Three TPOT items of concern were identified based on low scores; the second researcher began conducting assessment observations in the classroom two days per week focusing on these low scoring items for 3-9 sessions. Observations were scheduled for each teacher based on individual class schedules and took place in each teacher’s classroom for up to 2 hours per observation. During baseline, teachers did not receive any coaching or performance feedback. The second researcher collected baseline data on the indicators connected with the three low scoring items or items of concern. Baseline data were collected for each teacher by using the individualized TPOT to observe the three targeted items, and assessment observations lasted up to 2 hours and involved a variety of classroom activities (e.g., circle time, free play time) and occurred two days per week. The individualized TPOT meant that only the three items of concern and the associated indicators for that item were scored. Researchers had initially planned to provide coaching on all three items of concern, but time constraints and teachers’ schedules limited the implementation of the intervention to one of the three items. Thus, at the beginning of MULTICOMPONENT PROFESSIONAL DEVELOPMENT 57 the intervention phase, each teacher was asked to choose one item of focus on which he or she would like to receive coaching and performance feedback. The chosen item then became the focus of intervention. For example, Mary had low scores for Item 5, Teaching Children Behavior Expectations, Item 7, Teaching Social and Emotional Competencies, and Item 12, Supporting Friendship Skills. Mary chose to focus on Item 7, Teaching Social and Emotional Competencies, as her target area of focus. This choice was individual for each teacher but was based on what the teacher felt their children needed and what they felt was most beneficial for their individual classrooms and their own professional development needs. For the items chosen, there were indicators that were to be addressed to score the item appropriately. Item 5, Teaching Children Behavior Expectations (TBE), had seven indicators that were to be scored yes or no during the observation. For example, the first indicator, TBE1: Teacher has posted behavior expectations or rules that are positively stated, include a visual, and are limited in number (less than five per major activity), had to be visually seen in the teacher’s room during the observation in order for the item to be scored a yes. Each teacher needed to have at least three stable baseline data points on the targeted item, based on the TPOT scores, before moving to the intervention phase. Intervention. The intervention in this study involved coaching and performance feedback. Each teacher entered the intervention phase at a staggered rate with a total of 6-13 sessions for all three teachers (e.g., John received 6 sessions, Sharon received 11 sessions, Mary received 13 sessions). The primary researcher was responsible for all intervention sessions and served as the coach; these sessions occurred twice per week for each teacher. Over the course of the week, each teacher engaged in an initial planning session (20-30 minutes), observations (60- 90 minutes), coaching (occurred during observations), and performance feedback (30 minutes). The coaching model used involved action planning, feedback, and building new skills (Hemmeter et al., 2014; Lentini & Fox, 2008). For example, the primary researcher (e.g., coach) opened the initial planning meeting by checking in with the teacher about progress during the previous week and reviewed the teacher’s professional development plan which included what was going well in the classroom, what was most challenging, and what was to be worked on for the week with regards to the targeted item on the TPOT. The coach then provided supportive feedback by presenting teacher data based on classroom observation and highlighting areas of strength and progression of goals, this was then followed by suggestive feedback where strategies were presented for increasing TPOT scores and additional goals were discussed. To assist the teacher in reflecting on practices, active listening and open-ended questioning were used to discuss what happened during the observations (e.g., When you provided specific feedback to the child, what occurred afterwards?). Both the coach and teacher discussed what action was to be taken, what materials/supports/resources would be needed, and the planned action steps that would lead to successful implementation of indicators associated with the targeted item. The initial planning meeting then ended with a recap of what had occurred and what is to take place for the week. Focused observation would occur once per week and this involved the coach acting as observer and coach using the teachers’ individualized plan from the initial planning meetings. The coach provided feedback and engaged in modeling during this time as needed. At the conclusion of each week, the coach provided each teacher with additional performance feedback via email that followed similar steps as the face-to-face meeting. For instance, the coach began the email with opening comments that included general, positive statements of what occurred that week and followed those comments with supportive feedback of implemented practices that included teacher data and specific examples observed (i.e., You did 58 GODFREY-HURRELL, CORREA, & TRUESDELL a great job of naming the child’s feelings and talking about ways to express those feelings. For example, you said…). In addition, the email included suggestive comments of ideas for improving implementation of practices and a review of planned actions followed by a closing statement of encouragement with plans to meet the following week where this process would repeat until the teacher reached criterion. Data on the primary dependent variable were collected on the target area of focus during baseline and intervention phases, however researchers continued to collect data on all three low scoring items for each teacher to determine if a carry-over effect (e.g., generalization) would occur for uncoached items. The coach supported and provided instruction for each teacher by identifying goals and desired outcomes and by reviewing activities that would help the teacher reach those goals, as discussed above; these goals were recorded on a coaching log that documented the coaching process. The second researcher provided the primary researcher with scores on each of the three targeted items for each teacher. A 1-hr assessment observation and was conducted once a week. The data collection forms used included the three targeted items and researchers indicated whether the indicators within the identified items were observed (yes) or not observed (no) during each observation. A percentage of indicators scored yes were recorded and graphed for visual analysis. A target criterion level was set at 80% for each teacher for his/her target item of focus. Each teacher had to meet the 80% criterion at least three consecutive observations during intervention to conclude that phase and move on to the maintenance phase. For example, Sharon and John targeted Item 5, Teaching Children Behavior Expectations, which had seven indicators that had to be observed and scored a yes 80% of the time for three consecutive observations. Maintenance. The maintenance phase of this study occurred five weeks after the conclusion of the intervention phase for each teacher and varied dependent upon when the teacher finished the intervention phase. The primary researcher conducted a full TPOT assessment lasting 2 hours with each teacher during each teacher’s scheduled daily activities. Data were collected on all items. Interobserver Agreement Interobserver agreement (IOA) was assessed on 30% of the data across the first three phases of the study (pre-baseline, baseline, and intervention). To prepare for observer reliability, both researchers attended a training workshop on the TPOT, during which they watched videos and practiced scoring the TPOT. Both researchers successfully completed the workshop by reaching an 80% agreement level as determined by the workshop’s developers on the full instrument. During the pre-baseline and baseline phases of the study, the primary researcher rated each teacher’s skill level related to the three targeted items chosen from the TPOT. IOA was determined by comparing the primary researcher’s overall score to the second researcher’s score. The formula used for this calculation was the number of agreements divided by the number of agreements plus disagreements times 100, which resulted in the percentage of agreement (Kratochwill et al., 2010). The data indicated a mean of 85% and a range of 77% to 100% during the pre-baseline phase, a mean of 97% during baseline, and a mean of 92% during intervention. MULTICOMPONENT PROFESSIONAL DEVELOPMENT 59 Treatment Fidelity Fidelity was collected for 30% of the coaching and performance feedback sessions that took place during the intervention phase. The second researcher used fidelity checklists and conducted observations of the first researcher and reviewed emails with performance feedback for 30% of the sessions. Data showed a mean of 71% with a range of 36% - 88% for the observations and a mean of 81% with a range of 75% to 87% for the email with performance feedback. Researchers did meet to discuss the variability in the data and tried to come to a consensus on the coaching checklist steps and requirements, however the fidelity means were low and will be discussed within the limitations. Social Validity Social validity was assessed to determine the extent to which the coaching and performance feedback was socially acceptable to the teachers included in the study. Data collection took place at the conclusion of the study and involved the three teachers completing a four-item consumer satisfaction questionnaire on a 5-point Likert Scale ranging from strongly disagree (1) to strongly agree (5). Teachers were asked to rate perceptions on the social importance of the Teaching Pyramid Model, social acceptance of the coaching and performance feedback, and social significance of their individual skill change. RESULTS Research Question 1 The visual results of the percentage of targeted TPOT items correctly implemented and the other two identified items of concern for each teacher are shown in Figure 1, while results on the mean and range for each targeted item are presented in Table 2. These data indicated that the coaching intervention resulted in higher implementation of strategies connected with the one targeted item selected by each teacher. Some change was also noted for each teacher in the other two items of concern that were not specifically targeted by the coaching intervention, indicating that the intervention generalized to the implementation of other classroom strategies. All teachers reached and surpassed the 80% criterion level and demonstrated improvement in their implementation of the strategies related to the targeted items. In addition to examining the immediacy of effect from baseline to intervention for teachers’ use of strategies, percent of non- overlapping data (PNDs; Scruggs & Mastropieri, 1998) were calculated. It is suggested (Scruggs & Mastropieri) that 90% of data points above baseline should be considered very effective, 70% to 90% above baseline to be effective, 50% to 70% above baseline to be questionable, and below 50% to be ineffective. PNDs were calculated separately for each teacher between baseline and intervention by identifying the highest performance data point during baseline and counting all intervention performance data points that were higher than the highest baseline point and dividing that number by the total number of data points in intervention and multiplying by 100. 60 GODFREY-HURRELL, CORREA, & TRUESDELL Figure 1. Percentage Correct on Each Teacher’s Implementation of TPOT Items. SESSIONS Mary Sharon John Intervention Baseline Maintenance PE R C EN TA G E C O R R EC T O N T PO T IT EM S MULTICOMPONENT PROFESSIONAL DEVELOPMENT 61 TABLE 2 Percentage of TPOT Practices Implemented For Targeted Items Targeted Item Participant Baseline M (range) Intervention M (range) Teaching behavior expectations Mary 0 0 Sharon 0 52 (0-86) John 0 49 (0-86) Teaching social and emotional competencies Mary 0 51 (13-88) Sharon 11 (0-13) 33 (13-38) John 12 (0-13) 35 (13-63) Teaching friendship skills Mary 0 24 (11-56) Teaching problem solving Sharon 11 16 (11-33) John 9 (0-11) 23 (11-44) Mary. The baseline data for Mary indicated that she was not implementing any of the strategies associated with the indicators for any of the three targeted items. All of her scores remained at 0% during the baseline phase; therefore, she was the first teacher to enter intervention. Out of the three targeted items for Mary (e.g., Item 5: Teaching Children Behavior Expectations, Item 7: Teaching Social Skills and Emotional Competencies, and Item 12: Teaching Friendship Skills), Mary chose to focus on Item 7: Teaching Social Skills and Emotional Competencies. She began receiving coaching and performance feedback, and her implementation of the skills associated with the targeted item moved steadily up to a mean of 51% with a range of 13% - 88% and PND of 100% since all data points in intervention were above those in baseline thus indicating a strong intervention effect. The mean was calculated by adding together the overall scores of all observations during each phase and then dividing that number by the total number. Although Mary was not specifically coached on how to teach friendship skills, her implementation of that item, Item 12: Teaching Friendship Skills, showed slight improvement, reaching a mean of 20% with a range of 0% to 56% (see Table 2) and a PND of 100%. The third item, Item 5: Teaching Children Behavior Expectations, remained the same at 0% and a PND of 0%. Mary’s maintenance data showed 0% for Item 5: Teaching Children Behavior Expectations, 38% for Item 7: Teaching Social Skills and Emotional Competencies, and 90% for Item 12: Teaching Friendship Skills. These data showed a decrease in the target item but an increase in one of the uncoached items, Item 12: Teaching Friendship Skills. Sharon. During baseline, the data indicated that Sharon was not fully implementing any strategies related to her identified three items, Item 5: Teaching Children Behavior Expectations (0%), Item 7: Teaching Social Skills and Emotional Competencies (13%), and Item 11: Teaching Friendship Skills (11%). She had stable baseline scores and was the second teacher to enter intervention. Before entering intervention, she was presented with the three items of 62 GODFREY-HURRELL, CORREA, & TRUESDELL concern and she chose to focus on Item 5: Teaching Children Behavior Expectations. The coaching and performance feedback she received focused initially on encouraging her to choose five or fewer rules and then post a visual representation of those rules in the classroom, which is required as a prerequisite for any other indicators under that particular item to be scored “yes.” When she completed and posted the visual display of data, her scores improved significantly, moving from 0% to 57% from one assessment observation to the next, giving her a mean score of 48% with a range of 0% to 86% (see Table 2); however her PND was 63% reflecting a questionable or small effect. She also demonstrated improvement in the other two areas of concern that were not addressed in intervention, reaching a mean of 33% with a range of 13% — 38% for Item 7: Teaching Social Skills and Emotional Competencies and a PND of 72% (effective), and a mean of 21% with a range of 11% —33% for Item 11: Friendship Skills and a PND of 27% thus indicating a small effect. Maintenance data for Sharon showed 86% for Item 5, 50% for Item 7, and 67% for Item 11. John. John’s three targeted items of concern consisted of, Item 5: Teaching Children Behavior Expectations, Item 7: Teaching Social Skills and Emotional Competencies, and Item 11: Teaching Problem Solving. During the baseline phase, John’s scores on the individualized TPOT showed the most variance. Although his scores for Item 5 remained steady at 0%, a range of baseline scores were noted for Items 7 and 11. Baseline scores for John indicated a mean of 13% with a range of 0% to 25% for Item 7: Teaching Social Skills and Emotional Competencies and a mean of 9% with a range of 0% to 11% for Item 11: Teaching Problem Solving (see Table 2). Because of the unstable scores, John was the last teacher to enter the intervention phase. When he was presented with the three items of focus, he chose to work on Item 5: Teaching Behavior Expectations. As explained above, a visual representation of the rules was needed before any of the indicators on the TPOT could be marked as “yes”. John’s scores moved from 0% to 71% within one observation after posting the rules; total intervention data showed a mean of 49% with a range of 0% to 86% for his target item and a PND of 66%. He immediately was able to identify the rules he wanted to use because he was already talking about them with the children. The coaching and performance feedback John received also positively affected his use of non-targeted strategies associated with Item 7: Teaching Social Skills and Emotional Competencies, resulting in a mean score of 35% with a range of 13% to 63% and a PND of 66%. The strategies connected with Item 11: Teaching Problem Solving, resulted in a mean of 23% and a range of 11% to 44% and a PND of 33%. Maintenance data for John showed 100% for both Items 5 and 7 and 89% for Item 11. These data demonstrated an increase in skill for the targeted item and both uncoached items. In addition to targeted TPOT items, data was also taken to identify any carryover effect on other TPOT items not targeted. All teachers increased in their full TPOT scores with 13 or less coaching sessions from pre-baseline to maintenance. Mary’s data for the pre- overall TPOT score (see Table 3) during pre-baseline was 40% and the post overall TPOT score was 65%; which showed a 15% increase over the course of the study. Sharon demonstrated that she was able to maintain skills learned and showed increased carry over with the two uncoached target items. Additionally, Sharon’s pre- and post- overall TPOT score showed a 21% increase from the pre-baseline score of 58% to the post-TPOT score of 79% (see Table 3). Further, John showed a 17% increase from his pre-baseline overall TPOT score of 60% to 87% at post overall TPOT score (see Table 3). MULTICOMPONENT PROFESSIONAL DEVELOPMENT 63 TABLE 3 Pretest Posttest Scores on Full TPOT Participant Pretest Score Posttest Score 1 40% 65% 2 58% 79% 3 60% 87% Research Question 2 Data for children were measured through a pre- and post-test of the ASQ-SE. Each teacher was given the corresponding questionnaire to fill out on the targeted children at the beginning of the study and again after the study had completed. Data showed that four of the six children displayed less challenging behaviors post intervention while two children showed an increase in behavioral issues (see Table 1). Three different ASQ-SE forms were used to correspond with the age of each target child. Most showed a decrease in challenging behavior (see Table 1) with some variability with Child A and Child E. Child A (32 months) scored a 15 at baseline and a 45 at post intervention, a 30% increase in behaviors while Child B’s post intervention score showed 35% less challenging behaviors. Child C showed 20% less, Child D was scored as having 15% less challenging behaviors, however Child E (53 months) scored a 40 at baseline and a 60 at post intervention, a 20% increase. Child F score resulted in 15% less challenging behaviors. Research Question 3 Results from the social validity questionnaire showed that all (100%) teachers found the Teaching Pyramid Model to be helpful with 66% reporting that they strongly agreed that coaching and performance feedback was beneficial to implementation of strategies in the classroom. Further, 100% of teachers marked that they disagreed or strongly disagreed that they would have been able to implement the Teaching Pyramid Model easily without coaching and performance feedback support. One teacher wrote, “Working with the coaching project has helped me look at what children need individually on an emotional and social level so they can communicate independently with their peers around them”. DISCUSSION The results of this study demonstrate that the coaching plus performance feedback used with teachers led to higher levels of implementation of strategies associated with the targeted TPOT item. It should be noted that the universal practices outlined in the Teaching Pyramid Model— those related to building positive relationships and structuring supportive environments—were not identified as areas of concern. Instead, the targeted items fell under the third tier, which covers the explicit teaching of social and emotional skills (Hemmeter et al., 2014). The teachers thus demonstrated a fundamental understanding of the strategies associated with the model at the 64 GODFREY-HURRELL, CORREA, & TRUESDELL beginning of the study. Perhaps this was due to the previous training all had received with the Teaching Pyramid Model. Once the item of concern was identified and the teachers entered the baseline phase of the study, they demonstrated low scores related to the targeted items; many of the scores were recorded as 0%. At the end of the intervention phase, all teachers reached a score of at least 80% for the targeted items that they chose and in some cases the intervention seemed to influence the generalization of uncoached items as well. It would seem that the coaching intervention was the stimulus for the changes. However, when examining the effect size (e.g., PNDs) of the current study results are varied and should be interpreted with caution. To increase effect size, it may be beneficial to extend the amount of time in intervention to further examine the impact of the intervention on teacher behavior. The current study attempted to replicate and extend the findings of a previous study in which three early childhood special education teachers received coaching in strategies related to all tiers of the Teaching Pyramid Model (Fox et al., 2011). Although the current study only addressed a small portion of the strategies connected with the model, it adds to the literature by focusing on teachers within a child care program versus an early childhood special education public school setting and using effect size to demonstrate intervention strength. Researchers applied the principles of both the Teaching Pyramid Model and the particular coaching model used in the previous study to a different population of teachers, including three African- Americans with Associate degree level training in early childhood education. Previous studies have targeted teachers with Bachelor’s and/or Master’s level degrees with special training in special education. In addition, the Teaching Pyramid Model strategies were applied to a group of 2-year-olds in one of the teacher’s classrooms (e.g., Sharon), which was an age group not addressed by Fox et al. (2011). Based on the teacher’s performance, it would seem that the strategies can be applied to teachers serving children as young as 2 years of age. Even though the child outcome data were variable, improvements were seen for four of the six children. This finding supports the fact that providing coaching and performance feedback to teachers is associated with increased teacher skill and knowledge of practices in the classroom, thus affecting child outcomes. Limitations The study was affected by varied limitations. First, for much of the study, outside of the pre- baseline phase, the TPOT was altered to focus on only three particular items. The full TPOT covers a total of 31 items; the researchers were trained to implement and score the instrument as a comprehensive instrument and changing its application may have affected its internal validity. Both researchers were also closely involved in the study as coach and observer, and both researchers were invested in its results. However, the IOA data indicated good reliability therefore providing some assurance of internal validity. Second, the ASQ-SE is a questionnaire that is typically filled out by the child’s teacher and parent, however in this study only the teacher filled out the ASQ-SE so scores may not have been a full representation of the child’s social and emotional development and behavior and since teachers did provide the scoring, it may have impacted the reliability. It is suggested that additional measures and observation be used to determine each child’s trajectory and that information should be gathered from multiple sources. MULTICOMPONENT PROFESSIONAL DEVELOPMENT 65 Finally, the treatment fidelity means for the coaching and performance feedback were low. The checklist for coaching and performance feedback provided steps that the primary researcher who served as the coach was to follow, however there were several occasions where the coach took the lead of the teacher and found it difficult to keep to the exact steps. For example, Sharon had some difficulty making decisions and the coach had to continually prompt her and provide leading questions to create goals, thus diverting from the coaching steps. Providing coaching can be a dynamic process and in this case called for changes in the procedures to assure the teacher was receiving needed supports specific to his/her needs. Implications for Future Research The findings of this study can be extended in multiple ways. First, future research should consider expanding the coaching intervention over a longer period of time which would allow for more targeted items on the TPOT and could possibly increase effect size and reliability of the intervention. If resources allow, the majority of the school year should be devoted to multiple targeted TPOT items for teachers across phases outlined in the multi probe design. It is suggested that more time be given to appropriately train teachers on all tiers of the Teaching Pyramid Model (e.g., Universal Promotion, Secondary Prevention, and Tertiary Intervention), provide coaching supports, and extend the maintenance phase to include multiple observations to assure long term implementation, or sustainability, of strategies specific to children and teachers’ needs. It is also suggested that the Teaching Pyramid Model be implemented in accordance to what is developmentally appropriate for the group of children being served; some methods may not be suitable for children under 2 years of age. Secondly, children’s gains on multiple pretest and posttest outcomes should also be included. Child assessment data should be collected through multiple sources (e.g., repeated observations of child’s behavior in the classroom environment) to verify the impact the teacher intervention has on the presence of challenging behaviors and/or social and emotional issues in children. It may even be beneficial to include parent report on children’s behavior as part of the data collection and social validity. Having a comprehensive view of a child’s behavior may provide the coach and the teacher with ample sources of information to thoroughly understand the dynamic needs of the children being served. Finally, the current study’s main strength was its focus on a diverse early childhood teacher population: African-American teachers within a private faith-based community-based child care program that served primarily white, middle-class toddlers and preschoolers with varying levels of education and training in early childhood. Future research could investigate the impact teachers’ level of education and work settings have on meeting the needs of young children at high risk for having challenging behaviors and children who may be identified as having disabilities. The findings of this study demonstrated that teachers benefit from the individualized coaching model outlined above. The data also indicated that the coaching model could be applied to a different population of teachers and varied ages of children, with positive results. The study utilized a coaching and performance feedback intervention on one of the items of the TPOT and participants showed improvements in their implementation of the strategies associated with that segment of the Teaching Pyramid Model. Further research can build on these outcomes by expanding the length of the study and by keeping the focus on teachers within child care settings. 66 GODFREY-HURRELL, CORREA, & TRUESDELL REFERENCES American Academy of Pediatrics [AAP] Council on Early Childhood, AAP Committee on Psychosocial Aspects of Child and Family Health, AAP Section on Developmental and Behavioral Pediatrics. (2016). Addressing early childhood emotional and behavioral problems. Pediatrics, 138(6), 1-7. Retrieved from http://pediatrics.aappublications.org/content/early/2016/11/17/peds.2016-3025 Brown, W. H., & Conroy, M. A. (2011). Social-emotional competence in young children with developmental delays: Our reflection and vision for the future. Journal of Early Intervention, 33, 310-320. doi:10.1177/1053815111429969 Child Care Aware ® of America. (2015). 114th & 115th congress, 2016-2017 public policy agenda. Retrieved from http://usa.childcareaware.org/wp-content/uploads/2015/11/FINALCCAoA-2016-2017-Policy-Agenda.pdf Cooper, J. L., Masi, R., & Vick, J. (2009). Social-emotional development in early childhood: What every policymaker should know. National Center for Children in Poverty. Retrieved from www.nccp.org/publications/pdf/text_882.pdf Fantuzzo, J., Bultosky-Shearer, R., McDermott, P. A., McWayne, C., Frye, D., & Perlman, S. (2007). Investigations of dimensions of social-emotional classroom behavior and school readiness for low-income urban preschool children. School Psychology Review, 36, 44-62. doi:10.1037/0012-1649.44.1.139 Fox, L., Dunlap, G., Hemmeter, M. L., Joseph, G., & Strain, P. S. (2003). The teaching pyramid: A model for supporting social competence and preventing challenging behavior in young children. Young Children, 58, 48-52. Fox, L., Hemmeter, M., Snyder, P., Binder, D. P., & Clarke, S. (2011). Coaching early childhood special educators to implement a comprehensive model for promoting young children’s social competence. Topics in Early Childhood Special Education, 31, 178-192. doi:10.1177/0271121411404440 Fox, L., & Smith, B. (2007). Promoting social, emotional and behavioral outcomes of young children served under IDEA. Technical Assistance Center on Social Emotional Intervention for Young Children and Center on the Social Emotional Foundations for Early Learning (Issue Brief). Retrieved from http://challengingbehavior.fmhi.usf.edu/do/resources/documents/brief_promoting.pdf Hemmeter, M. L., Fox, L., & Snyder, P. (2014). Teaching pyramid observation tool for preschool classrooms (TPOT) manual, research edition [Measurement tool]. Baltimore, MD: Brookes. Henderson, J., & Strain, P. (2009) Screening for social emotional concerns: Considerations in the selection of instruments. Tampa, Florida: University of South Florida, Technical Assistance Center on Social Emotional Intervention for Young Children. Retrieved from http://challengingbehavior.fmhi.usf.edu/do/resources/documents/roadmap_1.pdf Hitchcock, J. H., Horner, R. H., Kratochwill, T. R., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2014). The What Works Clearinghouse single-case design pilot standards: Who will guard the guards? Remedial and Special Education Advance online publication, 35(3), 145-152. doi:10.1177/0741932513518979 Joyce, B., & Showers, B. (2003). Student achievement through staff development. Retrieved from http://literacy.kent.edu/coaching/information/Research/randd-engaged-joyce.pdf Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2010). Single-case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf. Lentini, R., & Fox, L. (2008). Coaching teachers in the Teaching Pyramid (pdf). Retrieved from http://www.challengingbehavior.org/explore/presentation_docs/11.08_coaching_teacher.pdf. McCabe, P. C., & Altamura, M. (2011). Empirically valid strategies to improve social and emotional competence of preschool children. Psychology in the Schools, 48, 513-540. doi:10.1002/pits.20570 McCollum, J. A., Hemmeter, M. L., & Hsieh, W. (2011). Coaching teachers in emergent literacy instruction using performance-based feedback. Topics in Early Childhood Special Education, 33, 28-37. doi:10.1177/0271121411431003. Perry, D., Holland, C., Darling-Kuria, N., & Nadiv, S. (2011). Challenging behavior and expulsion from child care. Zero to Three, 4-11. Retrieved from http://main.zerotothree.org/site/DocServer/32-2_Perry.pdf Schachter, R. (2015). An analytic study of the professional development research in early childhood education. Early Education and Development, 26, 1057-1085. Retrieved from http://dx.doi.org/10.1080/10409289.2015.1009335 MULTICOMPONENT PROFESSIONAL DEVELOPMENT 67 Scruggs, T. E., & Mastropieri, M. A. (1998). Summarizing single subject research: Issues and applications. Behavior Modification, 22, 221-242. doi:10.1177/01454455980223001 Smith, B., & Fox, L. (2003). Systems of service delivery: A synthesis of evidence relevant to young children at risk of or who have challenging behavior. Tampa, FL: Center for Evidence-Based Practice: Young Children with Challenging Behavior, University of South Florida. Squires, J., Bricker, D., & Twombly, E. (2002). Ages & Stages Questionnaires: Social-motional, A parent- completed, child-monitoring system for social-emotional behaviors. Baltimore, MD: Paul H. Brookes.