Dialog, 23(1), Copyright © 2020, ISSN: 1930-9325 Current Early Educator Knowledge, Practice, and Needs Regarding Informal Assessment Audra I. Classen The University of Southern Mississippi Jean Kang The University of North Carolina at Greensboro Gregory A. Cheatham The University of Kansas Early educators are urged to use authentic assessments which assess young children's development using systematic observation of real-life experiences and activities (Susman-Stillman, Bailey, & Webb, 2014). However, only a limited number of studies are available regarding early educators' current practices and needs associated with systematic informal assessment (Early et al., 2007; Madaus, Rinaldi, Bigaj, & Chafouleas, 2009; Pretti-Frontczak, Kowalski, & Brown, 2002). Researchers provide a framework for this study by dividing the DEC recommended assessment practices (2014), and NAEYC assessment indicators of effectiveness (2003) into four themes: choosing assessment methods, collecting data, collaborating with families, and analyzing data. The framework was then used to design the survey instrument for the purpose of determining early educators' current: (a) use of informal assessment methods, (b) knowledge and beliefs about the effectiveness of informal assessment methods, and (c) needs for training related to using informal assessment methods. Results indicate that early educators working in settings other than Head Start revert to using two informal assessment methods (i.e., anecdotal notes, event/frequency) which match most teachers' self- rating of their knowledge regarding anecdotal notes. Finally, early educators in the current study reported needing additional training related to all informal assessment methods except for anecdotal notes. Keywords: Informal Systematic Assessment, Head Start, ECSE Preschool RESEARCH ARTICLE 2 CLASSEN, KANG & CHEATHAM INTRODUCTION The fields of early childhood (EC) and early childhood special education (ECSE) have advocated for high-quality preschools using quality curricula (Catalino & Meyer, 2015; Copple & Bredekamp, 2009; NAEYC & NAECS/SDE, 2003) that include informal and formal assessment opportunities. According to the Division for Early Childhood (DEC), “assessment informs intervention” and is vital to providing quality instruction and services for young children who have developmental delays/disabilities (DEC, 2014, p. 8). Intentional, systematic, and effective intervention requires the early educator (EE) to use multiple data collection methods (DEC, 2014; NAEYC & NAECS/SDE, 2003) to regularly monitor children's progress, interpret data, and make data-based decisions (Codding, Skowron, & Pace, 2005). Various assessments allow professionals to learn what skills children have mastered and what skills require educators’ support to develop. In fact, Head Start and many ECSE programs urge educators to use authentic assessments, which assess young children’s development using systematic observation of real life experiences and activities (Susman-Stillman, Bailey, & Webb, 2014). However, consistent implementation of ongoing, authentic assessment in EC education settings and everyday practices remains a concern due to the push for more standardized testing in EC settings (Schultz, Kagan, & Shore, 2009). Furthermore, only a limited number of studies are available regarding ECs’ current practices and needs associated with assessment (Early et al., 2007; Madaus, Rinaldi, Bigaj, & Chafouleas, 2009; Pretti-Frontczak, Kowalski, & Brown, 2002). Systematic Informal Assessments (SIA) For the purpose of this study, we defined systematic, informal assessments (SIA) as precise observation approaches with structured procedures for regularly obtaining individual child-level data in natural routines and context that are teacher-developed or adapted (Classen & Cheatham, 2015; Wolery & Ledford, 2014; Wortham & Hardin, 2016). Specifically, this study was interested in exploring the following SIA; (a) anecdotal notes, (b) event/frequency sampling, (c) partial- interval sampling, (d) whole interval sampling, (e) momentary time sampling, (f) duration sampling, and (g) latency sampling. The authors’ chose the aforementioned SIAs because they can provide a holistic view of all children’s skills, with and without disabilities; and connect outcomes to interventions and programmatic changes (Bagnato, 2005; Macy, Bricker, & Squires, 2005). There are reliable and validated assessments that are criterion referenced which collect data through systematic observation (e.g., Assessment, Evaluation, and Programming System (AEPS) and Teaching Strategies Gold). Criterion-referenced assessments provide information on individual children’s strengths and needs in specific areas measured. However, those assessments often include limited number of items to be observed that are not child specific or EEs often need more information in order to provide individualized supports (Wortham & Hardin, 2016). Thus, criterion referenced assessments are beyond the scope of this article (for more information see Lambert, 2017; Wortham & Hardin, 2016). An advantage of SIAs is that these methods are flexible enough to collect individual child specific information such as function of behavior, interest, preferences, skill development and knowledge acquired (Bagnato, 2005; Wortham & Hardin, 2016). Therefore, when correctly implemented, SIAs can supplement diagnostic evaluations, formative and summative evaluations and EEs can have deeper understanding of each child as a whole (Wortham & Hardin, 2016). EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 3 Professional Association Guidance for Informal Assessment Practices The Division for Early Childhood (DEC) of Council for Exceptional Children (CEC), and the National Association for Education of Young Children (NAEYC) each provide guidance for the use of informal assessment procedures (see Table 1). These organizations recommend that early intervention assessment should (a) monitor ongoing child progress; (b) occur in typical environments, activities, and routines; and (c) identify functional strengths and abilities (Bagnato McLean, Macy, & Neisworth, 2011). Finally, the four themes in Table 1 were used to develop the survey questions. Table 1. Alignment of Assessment Practices (DEC) and Indicators (NAEYC) with Assessment Themes Practices and Indicators Four Themes DEC NAEYC Choosing Assessments Collecting Data Analyzing Data Collaborating w/ Families 1 11 X X 2 11 X X 3 3 X X 4 3 X X 5 3 X X 6 8 X X X 7 7 X X 8 9 X 9 5 X X 10 6 X X 11 11 X X 1 X X X X 2 X X 4 X X X 10 X X Note. X=DEC assessment practices and NAEYC indicators present in each theme. Furthermore, the DEC assessment practices six through nine support the use of SIA. These recommended practices urge ECs to: 4 CLASSEN, KANG & CHEATHAM (a) use various methods, including observation and interviews, to gather assessment information from multiple sources, including the child’s family and other significant individuals in the child’s life; (b) obtain information about the child’s skills in daily activities, routines, and environments such as home, center, and community; (d) implement systematic ongoing assessment to identify learning targets, plan activities, and monitor the child’s progress to revise instruction as needed (DEC, 2014, p.8). Implications for Informal Assessment Professional Development (PD) Even though the literature provides evidence that SIA provide trustworthy child progress and program impact evidence, SIAs are not widely used (Neisworth & Bagnato, 2004). In recent research, EEs report a lack of confidence in selecting informal assessments, as we define SIA and other criterion referenced informal assessments, during specific classroom routines (Krasch & Carter, 2009). In addition, EEs may not consider partnering with families in choosing and using SIA or other informal assessments advantageous (Banerjee & Luckner, 2013). Similarly, researchers revealed inconsistent informal assessment use in a recent survey study conducted in Minnesota (Susman-Stillman et al., 2014). These researchers hypothesized that administrators not requiring informal assessment data or EEs not understanding the purposes of informal assessment limited EEs’ informal assessment use. Further challenges have been described in the research as: (a) limited time to implement, document, and research evidence-based assessment tools (Banerjee & Luckner, 2013; Susman-Stillman et al., 2014); (b) an inability to keep young children engaged during assessments (Banerjee & Luckner, 2013); (c) competing priorities; and (d) difficulties integrating assessment into regular routines (Bagnato, Neisworth, & Pretti-Frontczak, 2010; Susman-Stillman et al., 2014). Other research reported that EEs have associated PD on informal assessment with their improved understanding of the links between quality education and positive outcomes for all young children (Early et al., 2007; Waitoller & Artiles, 2013). Thus, the research literature suggests PD on informal assessment needs to (a) develop EEs’ understanding that informal assessment is important to their daily practice, (b) improve EEs’ capacity to perform various informal assessments, and (c) enhance EEs’ self-efficacy and “buy in” related to informal assessments (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). However, our ability to support and meet EEs’ PD needs related to SIA, is diminished by our limited understanding of their informal assessment perspectives (Wilkins, 2008). Therefore, the purpose of this study was to answer the following research questions: (a) How and for what purpose do EEs (i.e., EC - Head Start and child care setting or ECSE - public preschool settings) currently use informal assessment methods? (b) What are EEs’ perspectives about their knowledge of informal assessment methods? (c) What are EEs’ beliefs about the effectiveness of informal assessment methods? (d) What are EEs’ needs for training related to using informal assessment methods? and (e) How do EE participants’ demographic characteristics impact their responses regarding informal assessment? EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 5 METHOD An online survey was developed and distributed to EEs in two states. A descriptive, multivariate analysis was conducted. The following sections describe in detail participants, data collection, and data analyses. Recruitment EEs from across two states were invited to participate in the study through multiple recruitment strategies. Researchers distributed an email (more detail is provided below) containing an explanation of the study's purpose, the participation criteria, and consent for participating statement. If a professional agreed to take part in the study, s/he completed the survey through the link provided within the email. Researchers included participants if they: (a) were currently teaching in an EC setting (e.g., ECSE district classroom, Head Start classroom, child-care center, or kindergarten classroom), (b) were serving children between three to five years old, and (c) had at least one child with a disability in the classroom. EC policy differs across states. Therefore, authors purposefully chose two states at different places in the EC workforce improvement process to provide a representation similar to the national context (for individual state ratings see, Whitebook, McLean, Austin, & Edwards, 2018). State One. Demographic data for this southeastern state is: 59% Caucasians, 37.5% African Americans, 1.2% two or more races, 1% Asians, .9% other races, and .4% Native North Americans. Survey emails were sent directly to 20 EEs in one county school district using publicly available email addresses. These educators were then asked to forward the survey link to colleagues fitting the inclusion criteria using the snowball method. Additionally, the state Head Start director agreed to share the survey’s purpose in a state-wide PD and email the survey link to three Head Start Centers representing rural and urban areas of the state. Approximately 73 Head Start educators agreed to participate through this distribution. Finally, one EC special education (ECSE) coordinator in a local urban school district provided time for approximately 15 ECSE teachers to complete the survey. Thus, approximately 150 EEs across state one received the survey and 88 EEs completed the survey. State Two. Demographic data for this southeastern state is: 71.2% Caucasians, 22.1% African American, 1.6% American Indian, 2.8% Asian, and 9.1% Hispanic or Latino. The director of a state EC professional association division agreed to distribute the survey link via email to approximately 3,000 members (e.g., EEs, administrators, professionals in related areas, higher education faculty, and families within this organization’s listserv). A reminder email asking email recipients to complete the survey was sent to non-completers two weeks after the initial email. Although no exact numbers on how many members are currently working in preschool settings are available, the organization’s annual trend data shows about one-third of members work in preschool settings. As a result, 63 EEs completed the survey. 6 CLASSEN, KANG & CHEATHAM Participants and Settings The total number of 151 EEs responded to the survey; 88 from state one and 63 from state two. However, thirty-seven total participant surveys were excluded, because the participants failed to answer a significant portion of the informal assessment section (i.e., they only answered two questions in this section). The total number of participants included in final data analyses process was 114 (see Table 2). Participants’ mean age was 42.9 years and 99% (n=113) were female. The largest racial group represented was African American (n=67; 58%) followed by Caucasian (n=45; 39%). Additionally, two other participants identified themselves as other race (n=1; 99%), and more than one race (n=1; 99%). The EC workforce nationally is much more diverse than K-12 educators (US Department of Education, 2016). Our EE demographics are comparable to the recent EC Workforce Index reporting national EE demographics to be 60% white, 58% Hispanic, and 48% African American (Whitebook, McLean, Austin, & Edwards, 2018). Forty-four participants (38.6%) reported holding multiple licensures and 38 (33.3%) reported holding Birth-kindergarten state licensure. Due to differences in licensure categories, researchers combined any professional licensure or certificate associated with teaching children between birth and kindergarten (e.g., ECSE classroom, Head Start classroom, child-care center, kindergarten classroom) into the Birth-K category. Educators reported working in the EC field for an average of 5.48 years and in their current setting for an average of 3.5 years. EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 7 Table 2. Participant Demographic Information (N=114) Gender Female Male n=113 (99.1%) n=1 (.9%) Age Mean = 42.9 Range = 21-80 Race African American or Black Caucasian or White Other Multiple n=67 (58.8%) n=45 (39.5%) n=1 (0.9%) n=1 (0.9%) Geographic Location of the Classroom Rural Suburban Urban Other n=60 (52.6%) n=19 (16.7%) n=28 (24.6%) n=6 (5.3%) Licensure Birth-Kindergarten Other Missing n=57 (50.0%) n=56 (49.1%) n=1 (.9%) Years in Current Classroom Mean = 3.5 Range = 1-24+ EC Classroom Setting Head Start Other Setting Missing n=66 (57.9%) n=46 (40.3%) n=2 (1.7%) Disability Type in Current Classroom * Multiple responses allowed VI HI Autism DD LD EBD ID OHI n=7 n=7 n=25 n=61 n=8 n=27 n=5 n=25 Years Working in EC Mean = 5.48 Range = 1-24+ Students with IFSP/IEP Mean = 2.65 Range = 0-20 Total Students in Classroom Mean = 16.47 Range = 3-24 Note. VI=Visually impaired; HI=Hearing impaired; DD=Developmentally delayed; LD=Learning disability; EBD=Emotional behavioral disorder; ID=Intellectual disability; and OHI=Other health impaired; EC=Early childhood 8 CLASSEN, KANG & CHEATHAM Just as the EEs varied in their experience and licensure, so did their classroom characteristics. Sixty participants (52.6%) were currently teaching in classrooms located in rural areas. Participants reported their current work was at a Head Start program (n=66, 57.9%), preschool special education only classroom (n=18, 15.8%), infant and toddler services (n=5; 4.4%), ECSE inclusive setting (n=5; 4.4%), and licensed child care setting (n=6, 5.3%). Researchers recoded the participant work setting data into Head Start (n=66, 57.9%), other setting (n=46, 40.3%), and missing (n=2, 1.7%). Finally, participants reported various child disabilities within their classrooms across both states; Developmental Delay (DD) was the most common (n=61, 53.5%), followed by Emotional Behavior Disorder (EBD) (n=27, 23.7%), Other Health Impairment (OHI), and Autism (n=25, 21.9%). Participants reported a mean of 2.65 students in each classroom had an IFSP/IEP. Data Collection Procedures Researchers designed an online survey to explore EEs’ knowledge about and use of informal assessment methods to measure social-emotional and academic skills. The principal investigators of this study taught assessment courses in EC teacher preparation programs and developed the survey based on extensive review of research literature. After the researchers developed the initial survey, six EC experts provided feedback regarding the survey’s length, clarity of questions, need for additional questions, and provided general suggestions. Based on the reviewers’ feedback, researchers revised the survey by adding, combining, removing, and reordering questions. After the primary investigators obtained institutional review board approval from each state, the survey was separately distributed in each state using Qualtrics. Survey Instrument The final survey consisted of two parts with thirty questions. The first part of the survey aimed to gather demographic information while the second part focused on participants’ informal assessment practices, and training needs. Demographic Information. Researchers divided the demographic portion of the survey into two subsections with twelve questions. The first subsection, “Information about You” asked five questions about the participants’ age, gender, years of experience, licensure or certification, and race/ethnicity. The second subsection, contained seven questions focused on participants’ current work setting. The questions included information on geographic location (i.e., urban, suburban, rural), type of current classroom, the total number of children in the classroom, and types of disabilities represented in the classroom. Current Informal Assessment Practice. The second part of the survey included a comprehensive definition for each informal assessment method to ensure participants had a shared understanding of each assessment method. Participants were first asked about their use of various informal assessment methods, such as what method they were currently using, how often they used each method, about whom they collected data, and how they used the data. Next, participants answered questions about how they select informal assessment methods, and what, if any, informal EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 9 assessment methods are required by their program administrators. Finally, participants shared information about what prevents them from using various informal data collection methods. Perception of Assessment Knowledge and Effectiveness. Additionally, the participants were asked to rate their informal assessment knowledge and their beliefs about the effectiveness of each informal assessment method. Participants rated their informal assessment knowledge of each method using a 5-point Likert scale (1=no knowledge, 2=very little knowledge, 3=some knowledge, 4=practicing knowledge, 5=expert knowledge). Participants rated each method’s effectiveness using a 6-point rating scale with one being “very ineffective” to six being “very effective.” Training Received and Needs. Participants also answered questions about their training experiences, and they described training they desired for each assessment method. Participants selected all options that applied to their experiences given the following choices: a) self-taught, b) university/college course work, c) mentoring on the job, d) in-service training, and e) no training. Next, the participants indicated the training level they would require to effectively implement each informal assessment method using a 4-point rating scale question (1=None, 2=Some, 3=Quite a bit, 4=An extreme amount). Data Analyses Procedures After closing each survey, researchers downloaded the Qualtrics data from each state to Statistical Package for Social Sciences (SPSS Inc.) version 24 for analyses. After researchers from each state reviewed, cleaned, and assigned a code for each participant to protect confidentiality, researchers merged the two data sets. Analyses by respondents and variables did not show any significant trends in missing data. As mentioned previously, 114 participant surveys were considered complete and included in the final analysis. Responses from these 114 participants were analyzed in two ways: First, descriptive analyses were conducted including frequencies, percentages, means, and standard deviations. Second, independent samples t-tests were conducted to explore the relationships between participants’ demographic characteristics (e.g., the education setting, years of experience) on the informal assessment methods used, the average rating of the participants’ informal assessment knowledge and effectiveness, and level of training need. RESULTS Results are organized into four sections that correspond to the research questions posed in this study. The four sections include: (a) current use of informal assessment methods, (b) educators’ perceptions about their informal assessment knowledge, (c) educators’ perceptions about each informal assessment method’s effectiveness, and (d) training needs. Within each of these sections, researchers provide additional results reporting the relationship between educators’ demographic characteristics (i.e., EC education setting, and years of teaching experience) and their informal assessment perspectives. 10 CLASSEN, KANG & CHEATHAM Current Use of Informal Assessment Methods The survey asked participants several questions about their current informal assessment methods. Following sections will discuss the results in more detail. Numbers and Types of Methods Used. The majority of participants (n=95, 83.3%) reported currently using 1-2 informal assessment methods. EEs reported anecdotal notes as the method most widely used (n=100, 87.7%), followed by event/frequency (n=35, 30.7%), other (n=20, 17.5%), and partial-interval sampling (n=13, 11.4%). Eight participants (7.0%) reported not using any informal assessment methods. In addition, duration (n=8, 7.0%), latency (n=8, 7.0%), momentary time (n =7, 6.1%), and whole interval (n=5, 4.3) sampling assessment procedures were reported to be used the least. Method Selection and Frequency Use. Participants reported that many program administrators (n=85, 74.5%) required EEs to use anecdotal notes and some required event/frequency sampling (n=20, 17.5%). Very few EEs indicated administrators required momentary time (n=6, 5.2%), latency (n=4, 3.5%), whole interval (n=3, 2.6%), duration (n=3, 2.6%), and partial-interval (n=2, 1.7%) sampling. When specific data collection methods were not required, EEs selected informal assessment methods using various criteria (see Figure 1). Fifty- one educators (44.7%) used anecdotal notes daily or several times daily while 20 educators (17.5%) used event/frequency daily or several times daily (see Table 3). Figure 1. Early educators’ selection criteria for each assessment method (N=114). EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 11 Table 3. Note. Only 103 participants answered this question. Wk. = Week; Mo. = Month. How Data is Collected and Used. Participants reported that they used informal assessment methods to obtain a developmental snapshot of all young children with and without disabilities. For example, participants reported collecting informal data about children with Individualized Education Program (IEP) (n=82, 71.9%); an Individualized Family Service Plan (IFSP) (n=15, 13.1%); and children without disabilities (n=57, 50.0%). When asked how they used the collected data, 74 participants (64.9%) used the data to make instructional decisions for both children with and without disabilities. Yet it is more concerning that 56 participants (49.1%) did not use SIA to make instructional decisions. Similarly, 73 educators (64.0%) used data to prepare for parent-teacher conferences, 56 participants (49.1%) used data to inform interventions, and 55 participants (48.2%) used data for writing, monitoring, and revising IFSP/IEP goals. Relationship Between Demographic Characteristics and Educator Perspectives. There was a statistically significant (i.e., p<.01, or p<.05) difference in the mean number of informal assessment methods used by Head Start teachers (M=1.55, SD=1.01) and teachers working in settings other than Head Start (M=2.17, SD=1.53); t (110) = -2.60, p=.012). Thus, Head Start teachers used significantly fewer informal assessment methods compared to those in settings other than Head Start. In addition, there was significant difference in how often educators used anecdotal notes, event/frequency sampling, partial interval sampling, and duration sampling between the groups (Table 4): Head Start teachers used anecdotal notes significantly less often Early Educators’ Self-Rating of Frequency Use Per Informal Assessment (N=114) Daily Several Times Daily 1x Wk. 2x Wk. 3x Wk. 2x Mo. 3x Mo. Never Missing Anecdotal n=30 26% n=21 18% n=10 9% n=7 6% n=5 4% n=28 25% n=2 2% n=0 0% n=11 10% Frequency n=14 12% n=6 5% n=6 5% n=2 2% n=5 4% n=13 11% n=1 1% n=56 49% n=11 10% Partial n=2 2% n=3 3% n=7 6% n=3 3% n=5 4% n=8 7% n=0 0% n=75 66% n=11 10% Whole n=3 3% n=3 3% n=4 4% n=4 4% n=2 2% n=7 6% n=0 0% n=80 70% n=11 10% Momentary n=5 4% n=2 2% n=4 4% n=1 1% n=2 2% n=8 7% n=0 0% n=81 71% n=11 10% Duration n=4 4% n=2 2% n=2 2% n=2 2% n=4 4% n=8 7% n=2 2% n=79 69% n=11 10% Latency n=2 2% n=2 2% n=3 3% n=2 2% n=2 2% n=7 6% n=1 1% n=84 74% n=11 10% 12 CLASSEN, KANG & CHEATHAM than those in other settings. However, Head Start teachers used event/frequency, partial interval sampling, and duration sampling significantly more often than teachers working in settings other than Head Start. When comparing teachers with more than ten years to those with less than ten years of experience there was no significant difference in the mean number of informal assessment methods used or frequency of use. Table 4. T-Test Results Head Start vs. Other Education Settings – Frequency of Use (N=114) HEAD START OTHER MISSING n M SD n M SD t df Anecdotal Notes 63 3.68 2.40 39 5.23 2.01 -3.35** 100 12 Event/Frequency 63 7.17 1.59 39 4.62 2.88 5.78** 100 12 Partial Interval 63 7.24 1.71 39 6.03 2.81 2.70** 100 12 Whole Interval 63 7.33 1.76 39 6.51 2.48 1.95* 100 12 Momentary Time 63 7.24 1.93 39 6.72 2.38 1.20 100 12 Duration 63 7.29 1.78 39 6.38 2.66 2.04* 100 12 Latency 63 7.29 1.83 39 6.85 2.41 1.04 100 12 Note. Only 102 participants answered this question; **p<.01; *p<.05 Educators’ Self-Assessment of their Informal Assessment Knowledge Participants rated their knowledge for each informal assessment method using a 5-point Likert scale (i.e., 1=no knowledge, 2=very little knowledge, 3=some knowledge, 4=practicing knowledge, 5=expert knowledge). The mean rating score for anecdotal notes was higher (4.39, SD=0.546) than other informal assessment methods, which ranged between 2.02 (i.e., latency) and 2.69 (i.e., event/frequency). These data can be viewed in Table 5. EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 13 Table 5. Systematic Informal Assessment Knowledge Self Rating (N=114) N M SD Missing Combined Practicing and Expert Knowledge N (%) Anecdotal Notes 105 4.39 .54 9 102 (97.1%) Event/Frequency 104 2.69 1.29 10 34 (32.4%) Partial Interval 105 2.30 1.11 9 17 (16.2%) Whole Interval 105 2.26 1.09 9 15 (14.3%) Momentary Time 105 2.20 1.06 9 12 (11.4%) Duration 105 2.18 1.09 9 14 (13.3%) Latency 105 2.02 1.01 9 9 (8.6%) Note. Only 104-105 participants answered this question; Influences of Educator’s Demographic Characteristics. A statistically significant (i.e., p<.01 or p<.05) difference was found in the way educators from Head Start versus those in settings other than Head Start rated their knowledge of anecdotal notes and event/frequency (Table 6). EEs working in settings other than Head Start had mean scores ranging between 2.58 (i.e., latency) to 2.95 (i.e., partial interval) compared to the Head Start mean scores ranging from 1.67 (i.e., latency) to 1.89 (i.e., partial interval). Although educators working in the other than Head Start settings had higher mean scores for their informal assessment method knowledge, a significant difference was not found. Additionally, when comparing teachers with more than ten years of experience to those with less than ten years, there was no significant difference in how educators rated their knowledge of informal assessment methods. 14 CLASSEN, KANG & CHEATHAM Table 6. T-Test Results for Systematic Informal Assessment Knowledge Rating (N=114) HEAD START OTHER MISSING n M SD n M SD t df Anecdotal Notes 63 4.46 .59 40 4.30 .46 1.45** 101 11 Event/Frequency 63 2.17 1.22 39 3.49 .97 -5.67* 100 12 Partial Interval 63 1.89 .96 40 2.95 1.03 -5.27 101 11 Whole Interval 63 1.84 .91 40 2.90 1.05 -5.37 101 11 Momentary Time 63 1.81 .91 40 2.78 1.00 -5.04 101 11 Duration 63 1.76 .89 40 2.78 1.07 -5.18 101 11 Latency 63 1.67 .84 40 2.58 1.03 -4.87 101 11 Note. Only 102-103 participants answered this question; **p<.01, *p<.05 Educators’ Beliefs about Informal Assessment Effectiveness In terms of effectiveness, participants used a 6-point Likert scale to rate their beliefs about each informal assessment methods’ effectiveness (i.e., 1=very ineffective, 2=ineffective, 3=somewhat ineffective, 4=somewhat effective, 5=effective, and 6=very effective). A summary of these findings can be found in Table 7. Participants rated anecdotal notes as effective (mean = 5.08, SD=1.09) while other methods were rated between somewhat ineffective (i.e., latency sampling=3.27, whole interval=3.28, partial and momentary sampling=3.29, and duration=3.34) to somewhat effective (i.e., event/frequency sampling=3.69). EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 15 Table 7. Systematic Informal Assessment Effectiveness Ratings (N=114) n M SD Missing Combined Effective and Very Effective N (%) Anecdotal Notes 99 5.08 1.09 15 80 (70.1%) Event/Frequency 93 3.69 1.58 21 42 (36.8%) Partial Interval 90 3.29 1.46 24 23 (20.1%) Whole Interval 87 3.28 1.44 27 21 (18.4%) Momentary Time 86 3.29 1.46 28 22 (19.2%) Duration 86 3.34 1.49 28 25 (21.9%) Latency 86 3.27 1.45 28 21 (18.4%) Note. Only 104-105 participants answered this question Relationship Between Demographic Characteristics and Educator Perspectives. A statistically significant (i.e., p<.001, p<.01, or p<.05) difference was found in the way educators from Head Start versus those in the other Head Start settings rated the effectiveness of event/frequency, partial, whole, duration, and latency sampling (Table 8). When comparing teachers with more than ten years of experience to those with less than ten years, there was no significant difference in how educators rated the effectiveness of informal assessment methods. 16 CLASSEN, KANG & CHEATHAM Table 8. T-Test Results for Systematic Informal Assessment Effectiveness Rating (N=114) HEAD START OTHER MISSING n M SD n M SD t df Anecdotal Notes 63 5.10 1.05 35 5.09 1.17 .041 96 16 Event/Frequency 63 3.27 1.57 30 4.57 1.19 -3.98*** 91 21 Partial Interval 63 2.94 1.41 27 4.11 1.25 -3.73* 88 24 Whole Interval 62 3.02 1.44 25 3.92 1.25 -2.74* 85 27 Momentary Time 62 2.98 1.40 24 4.08 1.31 -3.30 84 28 Duration 62 3.03 1.48 24 4.13 1.22 -3.20** 84 28 Latency 62 3.00 1.42 24 3.96 1.30 -2.86* 84 28 Note. Only 86-98 participants answered this question; ***p<.001, **p<.01, *p<.05 Training Needs Identified by Early Educators Limited training was reported as the largest barrier to EEs using informal assessment methods (n=64, 56.1%). In addition, 21 participants (18.4%) reported not being sure how to select the right methods, 15 participants (13.2%) were not comfortable using the methods, and 13 participants (11.4%) said they were too busy teaching to informally assess children. About 61 participants (53.5%) said they learned about the assessment methods through in-service training while 40 participants (35.1 %) through university course work, 34 (29.8%) learned through mentoring on the job, and 29 (25.4%) were self-taught. In addition, 12 participants (10.5%) reported no training related to informal assessment methods. For self-ratings of training level needed, 52 participants (45.6%) reported needing some to extreme training regarding informal assessment methods (see Table 9). EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 17 Table 9. Systematic Informal Assessment Level of Training Needs (N=114) n M Median SD Missing Anecdotal Notes 105 1.74 1.60 .92 9 Event/Frequency 106 2.27 2.22 .92 8 Partial Interval 106 2.40 2.33 .88 8 Whole Interval 106 2.41 2.36 .90 8 Momentary Time 106 2.41 2.35 .92 8 Duration 106 2.37 2.32 .92 8 Latency 106 2.43 2.38 .93 8 Note. Only 105-106 participants answered this question Influences of Educators’ Demographic Characteristics. A statistically significant (i.e., p<.01, or p<.05) difference was found in the way educators from Head Start versus those in setting other than Head Start rated their level of training need for event/frequency, partial interval, whole interval, and duration (Table 10). Specifically, there was a statistically significant difference in the scores for event/frequency training need in Head Start (M=2.43, SD=1.00) and event/frequency training need in the other than Head Start settings (M=2.03, SD=.73) conditions; t(2.22)=p<.01. Statistically significant results for partial interval training need in Head Start (M=2.40, SD=.98) and partial interval training need in settings other than Head Start (M=2.38, SD=.70) conditions; t(.140)=p<.05. Finally statistically significant results for whole interval training need in Head Start (M=2.43, SD=.98) and whole interval training need in settings other than Head Start (M=2.35, SD=.77) conditions; t(.442)=p<.05. When comparing teachers with more than ten years of experience to those with less than ten years, there was no significant difference in how educators rated their level of training needed for each informal assessment method. 18 CLASSEN, KANG & CHEATHAM Table 10. T-Test Results for Systematic Informal Assessment Level of Training Needs (N=114) HEAD START OTHER MISSING n M SD n M SD t df Anecdotal Notes 65 1.77 .98 39 1.72 .82 .273 102 10 Event/Frequency 65 2.43 1.00 40 2.03 .73 2.22** 103 9 Partial Interval 65 2.40 .98 40 2.38 .70 .140* 103 9 Whole Interval 65 2.43 .98 40 2.35 .77 .442* 103 9 Momentary Time 65 2.43 1.00 40 2.38 .80 .298 103 9 Duration 65 2.38 1.02 40 2.33 .76 .317* 103 9 Latency 65 2.43 1.00 40 2.43 .84 .030 103 9 Note. Only 104-105 participants answered this question; **p<.01, *p<.05 DISCUSSION Ongoing, SIA is essential to EEs making instructional decisions. Using ongoing assessment for planning and progress monitoring is considered a fundamental indicator of quality programs for all young children, including children with disabilities (DEC, 2014; NAEYC & NAECS/SDE, 2003). This study’s purpose was to explore EE’s knowledge, current practice, and PD needs regarding informal assessment. Early Educators’ Current Informal Assessment Practice Results indicate that EEs working in Head Start typically use two informal assessment methods (i.e., anecdotal notes, event/frequency) and EEs working in settings other than Head Start used event/frequency, partial interval, and duration sampling less often than educators working in Head Start settings. These results suggest that EEs’ limited informal assessment knowledge impacted their varied use of SIA. Educator reports of consistently using few data collection methods suggest that EEs may not be adequately prepared to meet the call for choosing multiple data sources for planning instruction and making decisions about services (DEC, 2014; Wolraich, Gurwitch, Bruder, & Knight, 2005). Educators’ limited use of varied SIAs may be due to educators’ self- EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 19 reports of limited informal assessment knowledge and training which are further discussed in the next sections. Early Educators’ Informal Assessment Knowledge and Effectiveness Ratings Another objective of the current study was to determine teachers’ informal assessment knowledge. Results indicate that most teachers’ self-rating of their knowledge regarding anecdotal notes positively matched their reported frequency of anecdotal assessment use. However, 37-55% stated they never used any informal assessment method except anecdotal notes. Of those using anecdotal notes, only 33% (n=51) reported collecting data daily or several times daily, and only 28% (n=30) collected anecdotal data two to three times per month. This is of particular concern considering 53%-80% of participants (range 56-84) reported never using any other informal assessment data. These data reveal that ongoing SIA is not occurring for the majority of participants. These results are similar to previous research indicating EEs are still grappling with how to collect ongoing data through embedded natural routines and instruction (Banerjee & Luckner, 2013; NRC, 2008). Professional Development Needs The analyses revealed three significant findings related to informal assessment PD needs. Implementation science suggests that “teachers’ buy-in” related to using a specific instructional approach greatly impacts their frequency and quality of use (Fixsen et al., 2005). Similarly, Wilkins (2008) explained that PD integrating educators’ beliefs and building confidence will improve learning outcomes that can be sustained for longer periods of time. The current results indicate EEs’ beliefs about the effectiveness of each informal assessment method impacted their current practice and use. For example, EEs rated anecdotal notes as the most effective, event/frequency sampling as somewhat effective, and latency sampling as somewhat ineffective. Participants rated anecdotal notes and event/frequency most effective and most commonly use. Limitations As with most studies, this investigation had limitations. Findings should be interpreted with caution due to the small sample size and the convenient, snowball sampling method. Furthermore, the other than Head Start group consisting of preschool special education only educators, infant and toddler educators, ECSE teachers, and licensed child care educators may not have adequately represented the entire EC workforce. Self- report measures may also contain discrepancies between reported knowledge and current use and actual knowledge and classroom practice. However, the teacher’s self-ratings of their informal assessment use were strengthened by questions forcing them to choose the best descriptor of their informal assessment use and frequency, rather than simply asking if they did or did not use each informal assessment method, which limited bias and provided more detailed data. Additionally, participants across the two states did not work in similar settings. However, the differences between states more closely represent various EC settings in the United States. These limitations should be considered if this study is to be replicated on a larger scale. 20 CLASSEN, KANG & CHEATHAM Future Research Future research can include a one day informal assessment training and follow-up classroom observations of actual teachers using assessment methods to verify and validate teachers’ perceptions. In addition, the survey instrument should be further developed to include specific questions related to how teachers choose assessment methods, analyze data, and collaborate with families to understand children’s skills and develop instructional plans. Furthermore, studies in the future should explore how EEs’ meet the assessment needs of children from diverse cultural, linguistic, and family backgrounds (e.g., family beliefs, composition, SES) across various states. Implications for Practice These data have important implications for pre-service and in-service PD. As indicated by participants, pre-service teacher preparation programs should enhance how they address informal assessment methods in course work, practicum experiences, and assignments. Specifically, pre- service and in-service training should include multiple opportunities to observe teachers using SIA and practice administering SIA methods with various children for various purposes (DEC, 2014; NAEYC & NAECS/SDE, 2003). In addition, assessment PD should occur over time using mentoring or coaching supports to explicitly connect effective curriculum design to the SIAs role in the instruction process. Thus, researchers and teacher education programs designing PD training should consider how they may address EEs’ buy-in and beliefs regarding the effectiveness of SIA methods. Moreover, researchers and PD facilitators should recognize implementation of newly gained informal assessment skills will require time and ongoing support. Second, most EEs indicated their informal assessment knowledge was gained through in- service training while less than a third of participants gained informal assessment knowledge from their pre-service training. The data from the current study signifies the need for pre-service teacher education programs to review and improve their assessment courses so future educators can become proficient in implementing various informal assessment methods. Similarly, Banerjee and Luckner (2013) suggested that teacher education programs need to provide multiple opportunities for teacher candidates to practice using formal and informal assessments for children with varied characteristics in real-life situations and different settings. Furthermore, participants in Banerjee’s and Luckner’s study reported needing more training on how assessment relates to planning instruction, progress monitoring, and writing meaningful goals. Another study reported EEs’ desire for video examples of assessments occurring in natural settings and daily routines (Susman- Stillman et al., 2014). Third, EEs in the current study reported needing additional training related to all informal assessment methods except for anecdotal notes. In addition, barriers such as not being sure how to select the “most effective” methods, not being comfortable using the methods, having a busy schedule or limited time to use other methods, and not preferring particular assessment methods were identified. These survey outcomes may be related to participants’ limited informal assessment knowledge and could be addressed in a pre-service or in-service training. PD related to selecting informal assessment methods and knowing each methods’ time commitment may improve the use of whole interval, momentary, partial interval, duration, and latency sampling. In addition, PD delivery should include coaching, online modules, (Waitoller & Artiles, 2013) and communities of practice. Other studies revealed similar findings regarding participants’ EARLY EDUCATOR INFORMAL ASSESSMENT SURVEY 21 preferred PD delivery methods (e.g., Howes, James, & Ritchie, 2003; Ramey & Ramey, 2005; Susman-Stillman et al., 2014). Specifically, participants reported that the most beneficial delivery and support method for PD to be onsite mentoring and coaching (Susman-Stillman et al., 2014). Moreover, other research suggested ongoing PD (i.e., monitoring, mentoring, and supervising) provided by administrators supports EEs’ use of evidence-based practices (Howes, James, & Ritchie, 2003; Ramey & Ramey, 2005). REFERENCES Bagnato, S. J. (2005). The authentic alternative for assessment in early intervention: An emerging evidence-based practice. Journal of Early Intervention, 28(1), 17-22. doi:10.1177/105381510502800102 Bagnato, S. J., McLean, M., Macy, M., & Neisworth, J. T. (2011). Identifying instructional targets for early childhood via authentic assessment: Alignment of professional standards and practice-based evidence. Journal of Early Intervention, 33(4), 243-253. doi:10.1177/1053815111427565 Bagnato, S. J., Neisworth, J. T., & Pretti-Frontczak, K. (2010). LINKing authentic assessment and early childhood intervention: Best measures for best practice. Baltimore, MD: Paul H. Brookes Publishing Co. Banerjee, R., & Luckner, J. L. (2013). Assessment practices and training needs of early childhood professionals. Journal of Early Childhood Teacher Education, 34, 231-248. doi:10.1080/10901027.2013.816808 Catalino, T. A., & Meyer, L. E. (2015). Environment: Improving access and participation. In R. M. Santos (Ed.), DEC recommended practices: Enhancing services for young children with disabilities and their families (pp. 53- 63). Los Angeles, CA: Division for Early Childhood of the Council for Exceptional Children. Classen, A., & Cheatham, G. A. (2015). Systematic monitoring of young children’s social- emotional competence and challenging behaviors. Young Exceptional Children, 18, 1-19. doi:10.1177/1096250614523970 Codding, R. S., Skowron, J., & Pace, G. M. (2005). Back to basics: Training teachers to interpret curriculum‐based measurement data and create observable and measurable objectives. Behavioral Interventions: Theory & Practice in Residential & Community‐Based Clinical Programs, 20(3), 165-176. doi:10.1002/bin.194 Copple, C., & Bredekamp, S. (2009). Developmentally appropriate practice in early childhood programs serving children from birth through age 8. National Association for the Education of Young Children. 1313 L Street NW Suite 500, Washington, DC 22205-4101. Division for Early Childhood (2014). DEC recommended practices in early intervention/earlychildhood special education 2014. Division for Early Childhood (DEC), (2014). DEC recommended practices in early intervention/early childhood special education 2014. Early, D. M., Maxwell, K. L., Burchinal, M., Alva, S., Bender, R. H., Bryant, D., ... & Henry, G. T. (2007). Teachers' education, classroom quality, and young children's academic skills: Results from seven studies of preschool programs. Child Development, 78(2), 558-580. doi: 4139245 Fixsen, D. L., Blase, K. A., Naoom, S. F., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19(5), 531-540. doi:10.1177/1049731509335549 Howes, C., James, J., & Ritchie, S. (2003). Pathways to effective teaching. Early Childhood Research Quarterly, 18, 104-120. doi:10.1016/S0885-2006(03)00008-5 Krasch, D., & Carter, D. R. (2009). Monitoring classroom behavior in early childhood: Using group observation data to make decisions. Early Childhood Education Journal, 36, 475-482. doi:10.1007%2Fs10643-009-0316-1 Macy, M., Bricker, D., & Squires, J. (2005). An examination of the validity and reliability of a curriculum-based assessment approach used to determine eligibility for Part C early intervention services. Journal of Early Intervention, 28, 295-310. doi:10.1177/105381510502800101 Madaus, J., Rinaldi, C., Bigaj, S., & Chafouleas, S. M. (2009). An examination of current assessment practices in northeastern school districts. Assessment for Effective Intervention, 34, 86-93. doi:10.1177/1534508408325882 Moiduddin, E., Bush, C., Manley, M., Aikens, N., Tarullo, L., Malone, L., and Lukashanets, S., (2017). A Portrait of Head Start Classrooms and Programs in Spring 2015: FACES 2014-2015 Data Tables and Study Design. OPRE Report 2017-101. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. 22 CLASSEN, KANG & CHEATHAM National Association for the Education of Young Children (NAEYC) & National Association of Early Childhood Specialists in State Departments of Education (NAECS/SDE) (2003). Early Childhood Curriculum, Assessment, and Program Evaluation: Building an Effective, Accountable System in Programs for Children Birth through Age 8. National Research Council. (2008). Early childhood assessment: Why what and how. Committee on developmental outcomes and assessments for young children. Board on children, youth, and families, board on testing and assessment, division of behavioral and social sciences and education, 145-177. Neisworth, J. T., & Bagnato, S. J. (2004). The mismeasure of young children: The authentic assessment alternative. Infants & Young Children, 17, 198-212. doi:2004/07000 Pretti-Frontczak, K., Kowalski, K., & Brown, R. D. (2002). Preschool teachers' use of assessments and curricula: A statewide examination. Exceptional Children, 69, 109-123. doi:10.1177/001440290206900108 Ramey, S. L., & Ramey, C. T. (2005). Creating and sustaining a high-quality workforce in child care, early intervention, and school readiness programs. In M. Zaslow & I. Martinez- Beck (Eds.), Critical issues in early childhood professional development (pp. 355-368).Baltimore, MD: Paul H. Brookes Publishing Co. Saluja, G., Early, D. M., & Clifford, R. M. (2002). Demographic characteristics of early childhood teachers and structural elements of early care and education in the United States. Early Childhood Research & Practice, 4(1), n1. Schultz, T., Kagan, S., & Shore, R. (2009). Taking stock: Assessing and improving early childhood learning and program quality. Washington, DC: National Early Childhood Accountability Task Force. Retrieved from http://www.eec.state.ma.us/docs/Sharon%20Lynn%20Kagan%20Presentation%20to%20EEC%20Board.pd f Susman-Stillman, A., Bailey, A. E., & Webb, C. (2014). The state of early childhood assessment: Practices and professional development in Minnesota. Center for Early Education and Development, University of Minnesota. Retrieved from http://ceed.umn.edu/wp-content/uploads/2017/04/The-state-of-early-childhood- assessment-practices-and-professional-development-in-minnesota.pdf U.S. Department of Education, (2016). Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service, The State of Racial Diversity in the Educator Workforce, Washington, D.C. 2016. Retrieved from https://www2.ed.gov/rschstat/eval/highered/racial-diversity/state-racial-diversity-workforce.pdf Waitoller, F. R., & Artiles, A. J. (2013). A decade of professional development research for inclusive education: A critical review and notes for a research program. Review of Educational Research, 83, 319-356. doi:10.3102/0034654313483905 Whitebook, M., McLean, C., Austin, L.J.E., & Edwards, B. (2018). Early Childhood Workforce Index – 2018. Berkeley, CA: Center for the Study of Child Care Employment, University of California, Berkeley. Retrieved on February 18, 2019 from http://cscce.berkeley.edu/topic/early-childhood-workforce-index/2018/. Wilkins, J. L. (2008). The relationship among elementary teachers’ content knowledge, attitudes, beliefs, and practices. Journal of Mathematics Teacher Education, 11(2), 139–164. doi:10.1007/s10857-007-9068-2 Wolery, M., &b Ledford, J. (2014). Monitoring intervention and children’s progress. In M. E. McLean., M. L. Hemmeter., & Snyder, P. (Eds.), Essential elementsfor assessing infants and preschoolers with special needs. (pp.383-400). Upper Saddle River, NJ: Pearson Publishing. Wolraich, M. L., Gurwitch, R. H., Bruder, M. B., & Knight, L. A. (2005). The role of comprehensive interdisciplinary assessments in the early intervention system. In M. J. Guralnick (Ed.), A developmental systems approach to early intervention (pp. 133–149). Baltimore, MD: Paul H. Brookes. Wortham, S. C., & Hardin, B. J. (2016). Assessment in early childhood education. Merrill/Prentice Hall.