Chapter 1 Dialog, 17(2), 1-16 Copyright © 2014, ISSN: 1930-9325 RESEARCH ARTICLE Efficiency in Assessing Emergent Literacy Skills in Students Attending Head Start Karen L. Gischlar Rider University Edward S. Shapiro Lehigh University Research has indicated that development of reading skills begins very early in a child's life, suggesting that skill development should be closely monitored during the preschool years. However, assessment with the young child presents unique challenges, as results can be influenced by the child's emotional state and the conditions of testing, including lengthy administration. Assessments should be efficient and economical, producing the best results with the expenditure of minimum time and resources. The purpose of this study was to examine three measures of emergent literacy that were administered to 4- year-old students attending a Head Start program. It was hypothesized that tasks purporting to assess the same skills would evidence redundancy, suggesting that not all of the tasks were necessary for administration. Results indicated that although correlations between the tasks from the three measures were low to moderate, there was overlap among the larger batteries. Implications for practice and avenues for future research are offered Keywords: early literacy, assessment The ability to read proficiently is essential to academic and later life success (Cunningham & Stanovich, 1997; Snow, Burns, & Griffin, 1998). Consistently, research has demonstrated that children who read well experience more exposure to print and subsequent growth in numerous knowledge domains. In contrast, children who are poor readers tend to experience difficulty across academic subjects, as other content areas increasingly depend on reading ability (Lonigan, Burgess, & Anthony, 2000). As reading is related to a child’s success in school, academic success is related to future occupational and economic outcomes. According to the National Center for Education Statistics (2001), students who are successful in attaining high levels of education are more likely to be employed and to earn higher salaries than their counterparts who do not fare as well in school. 2 GISCHLAR & SHAPIRO The Relationship between School Performance and Poverty Numerous studies have shown that children living in poverty are much more likely to perform below standards on reading and writing tasks than their peers from economically stable homes (Snow & Páez, 2004). Differences in the skills associated with literacy are already visible during the preschool years when children from underprivileged homes lag behind their peers in vocabulary, extended discourse skills, and familiarity with the functions and uses of print. This lower early performance is of concern because myriad research has suggested that a good beginning in literacy during the preschool years is associated with successful conventional reading later in elementary school (Juel, 1988; Snow & Páez, 2004). Thus, young children living in poverty can be considered at high-risk for the development of future reading and learning problems. The relationship between poverty and poor academic performance has been long- recognized. As private preschools began to proliferate in the United States in the 1960s, President Johnson initiated Project Head Start, a federally-funded program aimed at preparing young children from socio-economically disadvantaged families for success in school (van Kleek & Schuele, 2010). Although only about half of all eligible children are served through Head Start, the program remains in existence with the same goal of preparing children of poverty for school entry (van Kleek & Schuele, 2010). Given the relationship between success in school and reading skill, Snow & Páez (2004) suggested that Head Start classrooms should be evaluated primarily from the perspective of how well they foster emergent literacy skills and language development. Emergent Literacy Skill Acquisition Traditionally, approaches to teaching reading have considered the starting point to be the child’s entry into formal schooling. However, a growing body of literature suggests that the acquisition of literacy skills occurs on a developmental continuum that begins early in a child’s life (Lonigan, Burgess, & Anthony, 2000), with the preschool years viewed as a critical developmental period (Lonigan, 2006b). Emergent literacy skills can be considered developmental precursors to conventional reading and writing skills (Whitehurst, Zevenbergen, Crone, Schultz, Velting, & Fischel, 1999) and include: a) oral language (Dickinson, McCabe, Anastasopoulos, Peisner-Feinberg, & Poe, 2003; National Institute of Child Health and Development, 2005; Roberts, 2005), specifically the ability to understand and use vocabulary (Landry, Swank, Smith, Assel, Gunnewig, 2006); b) letter name knowledge (Lonigan et al., 2000; Roberts, 2003); c) phonological awareness, the ability to detect and manipulate the sound structure of oral language (Kirby, Parrila, & Pfeiffer, 2003; Lonigan et al.); d) naming speed (Kirby et al.); and e) print awareness, which is the ability to name letters and the knowledge that written letters are associated with sounds (Landry et al., 2006). Assessment of Emergent Literacy Skills Mounting evidence has demonstrated that reading problems can be ameliorated through early intervention. In fact, statistics suggest that if children receive training in foundational skills EMERGENT LITERACY 3 during the preschool years, as few as 5% may experience later reading problems, compared to the current level of 20% to 30% (Landry et al., 2006). Because of the relationship between emergent literacy skills and future reading performance (Dickinson et al., 2003) and reading and academic success (Cunningham & Stanovich, 1997), these skills represent a crucial area for early assessment-driven instruction and intervention. As important as it is to monitor a young child’s growth in the emergent literacy skill areas, caution should be exercised in assessment. Early development is rapid and episodic and test results can be influenced by experience, the child’s emotional state, and conditions of the assessment (Epstein, Schweinhart, DeBruin-Parecki, & Robin, 2004). Given these concerns, it is vital that an efficient, economical, and valid means for measuring growth in the emergent literacy areas be identified to inform early instruction and intervention. Both “efficient” and “economical” imply the expenditure of minimum time and resources in producing the best results (Dictionary.com, 2008). Inappropriate assessment practices, including lengthy administration, can produce results that are less than valid, incorrectly influencing a student’s placement. Furthermore, instructional time can be wasted in inappropriate assessment, which limits the classroom learning experience (Schappe, 2005). For example, a child exposed to a lengthy test, or many different measures, may tire, leading to results that misrepresent the child’s skill development. Thus, it is of great importance to identify a test of emergent literacy that efficiently and economically assesses skills of interest. A more efficient and economical measure would increase the reliability and validity of the results and also reduce administration time, resulting in more time for classroom instruction. Currently, there are a number of standardized assessments of emergent literacy available for the preschool population. However, it can be challenging for the early childhood educator to decide which to use, as many contain tasks whose titles suggest that they measure similar skills, yet varying methods of testing are utilized. For example, many assessments include a rhyming task, but the task is presented in various ways across the larger measures. In some cases, the ability to rhyme is assessed through picture matching (e.g., Individual Growth and Development Indicators (IGDI), Early Childhood Research Institute on Measuring Growth and Development (ECRI-MGD), 1998; Phonological Awareness Literacy Screening PreK (PALS-PreK), Invernizzi, Sullivan, Meier, & Swank, 2004); and in others through auditory presentation (e.g., Center for Improving the Readiness of Children for Learning and Education—Phonological Awareness, Language, and Literacy System (CIRCLE: C-PALLS),CIRCLE Group, 2004). Some tests include rhyming tasks that are timed (e.g., IGDI), while others contain tasks whose timing is left to the administrator (e.g., PALS, CIRCLE). It is possible that these differing formats impact child performance. The challenge is to identify the most economical, non-redundant set of measures that is most likely to provide a clear picture of the child’s key areas of strength and deficit. Purpose of the Current Investigation The purpose of the current study was to determine the relationship among three different measures of emergent literacy skills administered to preschool students from a Head Start program, including the relationship between tasks from the different measures that purported to assess similar skills. A further purpose of the investigation was to determine whether there was 4 GISCHLAR & SHAPIRO redundancy, or overlap, among measures, in an effort to identify the most efficient means for testing preschoolers. The expected outcome was that analyses would suggest that the measures were assessing similar skills, indicating that it wasn’t necessary to administer all three. Specifically, the following research questions were addressed: a) Do tasks that assess different skills evidence significantly lower correlations than those purporting to measure the same skills? Further, will patterns emerge among the correlations in terms of the trait being measured versus the method of assessment (e.g., timed versus untimed, or visual versus auditory presentation)?; and b) What combination of measures demonstrated the greatest amount of redundancy? In other words, would the measures demonstrate overlap, thereby suggesting that the administration of all three was not necessary? METHODS Participants and Setting The study participants included 4-year-old primary English-speaking students who attended a Head Start program in an urban area in the northeastern United States. All students received services through the same seven classrooms. These classrooms were funded by a U.S. Department of Education Early Reading First (ERF) grant and utilized Opening the World of Learning (OWL; Schickedanz, Dickinson, & Charlotte-Mecklenberg Schools, 2004), a comprehensive early literacy program, as their core curriculum. The participants included 30 English-speaking 4-year-old students (13 males) in cohort one, 22 students (12 males) in cohort two, and 27 students (13 males) in cohort three. This subsample of 79 participants was drawn from a larger sample that included approximately 70 students per year receiving services in the ERF classrooms. The decision to include only 4-year-old primary English speakers in the current study was made to limit the effects of variables, specifically age and primary language, that were not a focus of the research questions. Further, this subsample of students was the only group assessed with the PALS (Invernizzi, Sullivan, Meier, & Swank, 2004) measure and this assessment was of primary interest. Measures Phonological Awareness Literacy Screening-PreK (PALS PreK; Invernizzi, Sullivan, Meier, & Swank, 2004). The PALS PreK (Invernizzi et al., 2004) is a measure of the young child’s knowledge of emergent literacy foundation skills including: a) name writing; b) alphabet knowledge; c) beginning sound awareness; d) print and word awareness; e) rhyme awareness; and f) nursery rhyme awareness, all of which have been deemed developmentally appropriate skills for 4-year-olds. The test is an untimed measure that is administered to the individual student through use of a flip chart, picture cards, and storybook twice per year. Typical administration time per student is 25 to 30 minutes and the child's score is the number of correct responses in each skill area. After administration, raw scores for each task are compared to developmental ranges. Students falling below the low end of any range should be considered for intervention in that skill area (Invernizzi et al.). The PALS PreK has demonstrated reliability and validity. All internal consistency estimates across tasks were in the acceptable range. Coefficients for Guttman split-half ranged EMERGENT LITERACY 5 from .71 to .94 and Cronbach’s alpha levels ranged from .75 to .93. Interrater reliability was accomplished by having two separate and independent raters score a child’s performance. These estimates, expressed as Pearson correlation coefficients, were consistently .99 across measures (Invernizzi et al., 2004). Furthermore, the PALS PreK measure evidenced moderate to high correlations with other measures across concurrent and predictive validity studies (Invernizzi, 2004). Center for Improving the Readiness of Children for Learning and Education— Phonological Awareness, Language, and Literacy System (CIRCLE: C-PALLS; CIRCLE Group, 2004). The CIRCLE (CIRCLE Group, 2004) includes measures of rapid letter and vocabulary naming and seven measures of phonological awareness (i.e., listening, rhyme matching, rhyme production, alliteration, counting words in a sentence, syllabication, and onset rime) that are administered during three benchmark periods across the school year—fall, winter, and spring. The letter naming and vocabulary measures are timed for 1-minute, whereas all other measures are untimed (CIRCLE Group, 2004). Because the CIRCLE measure is relatively new, limited psychometric data were available. To assess reliability of the CIRCLE measures, Cronbach alphas were calculated to measure internal consistency of the set of seven phonological awareness items. For the 3- and 4- year-old students, alpha was .91, for the 5-year-olds was .92, and for the total sample was .93, evidencing strong reliability. Furthermore, the CIRCLE was compared with the Expressive One Word Picture Vocabulary Test (EOWPVT; Brownell, 2000), the PCTOPP (Wagner, Torgesen, & Rashotte, 1999), and the Developmental Skills Checklist (DSC; CTB-McGraw-Hill, 1990) and demonstrated convergent and discriminate validity with estimates ranging from .17 (CIRCLE vocabulary with the DSC phonological awareness measures, discriminate validity) to .80 (CIRCLE letter fluency with the PCTOPP print awareness measure, convergent validity). Individual Growth and Development Indicators (IGDI; ECRI-MGD, 1998). The IGDI (ECRI-MGD, 1998), a progress monitoring tool for preschool aged children, includes three measures, Picture Naming, Alliteration, and Rhyming. Picture Naming, an assessment of expressive language development, is administered through the presentation of color pictures of objects typically found in a preschooler’s environment, including home (e.g., “cake” and “sink”), classroom (e.g., “glue” and “book”), and community (e.g., “rabbit” and “train”). The child is instructed to name the pictures as quickly as possible and the number of pictures named correctly in 1-minute is the child’s score. For the Rhyming and Alliteration tasks, the student is presented with a card containing a cue picture at the top and three pictures below it. The student is instructed either to choose the rhyming match or picture with the same beginning sound, depending on the task. The child’s score is the number of correct identifications in a 2-minute period. Like Picture Naming, after demonstration with the standard samples, the Rhyming and Alliteration cards can be presented in any order (Missall & McConnell, 2004). The three tasks were validated in a longitudinal study with a sample of 90 preschool children whose age range was from 36 to 60 months. Picture Naming was found to correlate with other standardized measures of language development including the Peabody Picture Vocabulary Test: Third Edition (PPVT-3; r=.56 to .75, p<.001) and the Preschool Language Scale-3 (PLS-3; r=.63 to .79, p<.001). Rhyming demonstrated moderate correlations with other measures of phonological awareness and early literacy development including the PPVT-3 (r=.56 to .62, p<.05), Concepts About Print (CAP; r=.54 to .64, p<.01), the Test of Phonological 6 GISCHLAR & SHAPIRO Awareness (TOPA; r=.44 to .62) and the DIBELS (Good & Kaminski, 2002) Letter Naming Fluency (r=.48 to .59) and Onset Recognition Fluency (r=.44 to .68) measures. Finally, Alliteration demonstrated moderate correlations with the PPVT -3 (r=.40 to .57, p<.01), the CAP (r=.34 to .55, p<.05), the TOPA (r=.75 to .79, p<.01), and the Letter Naming Fluency Measure of the DIBELS (r=.39 to .71, p<.05). Test-retest coefficients for the three tasks across a 3-week time period ranged from .44 to .89 (Missall & McConnell, 2004). Procedure Graduate students enrolled in school psychology, psychology, special education, and counseling programs were trained by the first author in the administration and scoring of the PALS PreK (Invernizzi et al., 2004) and CIRCLE (CIRCLE Group, 2004) measures and were required to meet a minimum of 95% accuracy before assessing students. For the IGDI measure, teachers and assistants were trained in administration, scoring, and interpretation. After the IGDI training, participants were required to demonstrate performance in administration through use of the checklists developed by the ECRI-MGD (1996-2002). Teachers and assistants were provided immediate feedback and were required to satisfy 100% of checklist indicators before assessing students. The PALS-PreK (Invernizzi et al., 2004) was given at the beginning and end of the school year, whereas the CIRCLE (CIRCLE Group, 2004) measure was administered at three benchmark periods, fall, winter and spring. The IGDI (ECRI-MGD, 1998) measures were given on a monthly basis. Inter-observer agreement (IOA) for the graduate students administering tests was collected across 25% to 30% of sessions through the use of audio recorders and a second rater. For the Head Start teachers administering IGDI, IOA was collected through use of the IGDI accurate administration checklists (ECRI-MGD, 1996-2002) and through a second rater who scored simultaneously for approximately 25% of the sessions. The IOA was calculated using the percentage of agreement method, wherein the number of possible agreements is divided by the number of scoring agreements and disagreements. The average IOA across the measures was 96.6% (r = 92.4% to 100%). RESULTS Skewness and kurtosis statistics were calculated for each of the tasks administered to students. Fabrigar et al. (1999) suggested that skewness above 2.0 and kurtosis above 7.0 may indicate that data are non-linear and abnormally distributed. Results presented in Table 1 demonstrated that all tasks across the emergent literacy measures were within reasonable skewness and kurtosis limits. EMERGENT LITERACY 7 TABLE 1 Skewness and Kurtosis of Emergent Literacy Tasks Task Skewness Kurtosis PName -1.37 0.63 PUpLet -1.12 0.07 PLowLet -0.57 -1.28 PSounds 0.82 -0.47 PBegSounds -0.56 -1.16 PPrint -0.53 -0.29 PRhy -0.76 -0.40 PNursery 0.07 -0.85 CLet -0.27 -0.72 CVocab -0.24 -0.06 CListening -1.10 -0.11 CRhy1 -0.23 -1.22 CRhy2 -0.36 -1.41 CAllit 0.20 0.62 CSent -0.09 -1.15 CSyll -0.20 -0.09 COnset 0.49 -1.07 IGDIPN -0.54 0.39 IGDIAllit 0.98 0.47 IGDIRhy 0.41 -0.77 As displayed in Table 2, a multitrait-multimethod matrix (Campbell & Fiske, 1959) was created to assess the construct validity of the three emergent literacy measures used in this study (i.e., PALS, CIRCLE, and IGDI). The matrix presented all of the correlations that resulted across each of the tasks included in the larger measures, as assessed through a variety of methods (Campbell & Fiske). For example, all three of the tests included measures of rhyme, but each employed a different method of assessment. The PALS and IGDI both presented visual stimuli with rhyming tasks, but the IGDI was a timed measure, whereas the PALS was not. The CIRCLE included two measures of rhyme, but both were presented verbally by the administrator, with no picture prompt. The multitrait-multimethod matrix allowed for examination of the relationships between traits as measured through different methods and, likewise, different traits as measured with similar methods. The matrix indicated that the tasks across the larger tests purporting to assess letterform identification correlated highly (r=.82 – .92), despite method, while the tasks purporting to assess phonological awareness skills had low to moderate correlations (r=.04 –.60). Finally, the two tasks measuring vocabulary had a moderate relationship (.42). 8 GISCHLAR & SHAPIRO TABLE 2 Multi-Trait/Multi-Method Matrix Method 1: PALS Method 2: CIRCLE Method 3: IGDI P N am e P U p L et P L o w L et P S o u n d s P B eg S o u n d s P P ri n t P R h y P N u rs er y C L et C V o ca b C L is te n in g C R h y 1 C R h y 2 C A ll it C S en t C S y ll C O n se t IG D IP N IG D IA ll it IG D IR h y M et h o d 1 : P A L S PName -- PUpLet .56 -- PLowLet .46 .92 -- PSounds .41 .65 .73 -- PBegSounds .43 .55 .56 .58 -- PPrint .37 .48 .49 .53 .65 -- PRhy .58 .35 .31 .42 .50 .55 -- PNursery .42 .35 .35 .41 .48 .65 .40 -- M et h o d 2 : C IR C L E CLet .47 .85 .82 .64 .45 .43 .31 .34 -- CVocab .31 .37 .35 .28 .41 .57 .38 .55 .49 -- CListening .27 .25 .30 .20 .32 .42 .31 .42 .24 .31 -- CRhy1 .51 .50 .48 .38 .40 .48 .49 .50 .36 .33 .36 -- CRhy2 .40 .46 .41 .35 .45 .44 .44 .51 .42 .37 .34 .60 -- CAllit .36 .47 .48 .43 .32 .34 .33 .40 .40 .19 .09 .44 .22 -- CSent .37 .43 .39 .40 .55 .54 .37 .43 .38 .45 .42 .41 .44 .04 -- CSyll .29 .19 .17 .18 .22 .28 .17 .34 .10 .27 .25 .27 .22 .06 .51 -- COnset .20 .21 .23 .37 .42 .54 .43 .46 .21 .47 .26 .38 .48 .27 .33 .31 -- M et h o d 3 : I G D I IGDIPN .23 .27 .32 .34 .41 .54 .42 .35 .27 .42 .24 .35 .32 .30 .29 .08 .23 -- IGDIAllit .24 .32 .35 .41 .49 .47 .32 .38 .25 .25 .18 .39 .44 .32 .37 .26 .38 .30 -- IGDIRhy .49 .48 .43 .46 .56 .61 .58 .58 .40 .51 .29 .59 .58 .38 .43 .33 .54 .40 .55 -- EMERGENT LITERACY 9 Canonical correlation analyses were calculated for the PALS-PreK (Invernizzi et al., 2004) and the CIRCLE (CIRCLE Group, 2004), the PALS-PreK and the IGDI (ECRI-MGD, 1998), and the CIRCLE and the IGDI to determine the degree of redundancy, or overlap, across the three larger measures. When the canonical correlation analysis results for the PALS and IGDI measures were considered, the full model, which evaluated shared variance between the dependent variable set (PALS) and the covariate set (IGDI) across all canonical functions, was statistically significant (Wilks’ λ=.32, F=3.93, p<.001). Further examination revealed one significant canonical correlation (canonical R=.78) that accounted for 88.8% of the variance in the model. As regards redundancy, or overlap, analysis demonstrated that the IGDI accounted for 51.3% of the variance in the PALS, indicating overlap. All correlations between the dependent variables and covariates with the significant canonical correlation were negative (See table 3.). TABLE 3 Canonical Correlation Analysis for PALS and IGDI Canonical Variate 1 PALS Name Writing -0.59 Uppercase Letters -0.62 Lowercase Letters -0.60 Sounds -0.65 Beginning Sounds -0.79 Print Awareness -0.88 Rhyme Awareness -0.78 Nursery Rhyme Awareness -0.75 IGDI Picture Naming -0.68 Alliteration -0.65 Rhyming -0.93 When the canonical correlation analysis results for the CIRCLE and IGDI were examined, the full model, which evaluated shared variance between the dependent variable set (CIRCLE) and the covariate set (IGDI) across all canonical functions, was statistically significant (Wilks’ λ=.34, F=3.28, p<.001). Similarly to the canonical correlation analysis conducted with the PALS and IGDI, there was one significant canonical correlation (canonical R= .77) that accounted for 88.7% of the variance in the model. Analysis demonstrated that the IGDI accounted for 38.2% of the variance in the CIRCLE, which suggested redundancy, or overlap, across the measures. All correlations between the dependent variables and covariates with the significant canonical correlation were negative (See table 4). 10 GISCHLAR & SHAPIRO TABLE 4 Canonical Correlation Analysis for CIRCLE and IGDI Canonical Variate 1 CIRCLE Letters -0.53 Vocabulary -0.70 Listening -0.40 Rhyming 1 -0.78 Rhyming 2 -0.77 Alliteration -0.55 Sentences -0.60 Syllabication -0.41 Onset Rime -0.69 IGDI Picture Naming -0.61 Alliteration -0.67 Rhyming -0.96 Finally, a canonical correlation analysis was conducted for the PALS and CIRCLE. The full model, which evaluated shared variance between the dependent variable set (PALS) and the covariate set (CIRCLE) across all canonical functions, was statistically significant (Wilks’ λ=.04, F=3.75, p<.001). There were two significant canonical correlations. The first (canonical R=.91) accounted for 73.4% of the variance in the model and the second (canonical R=.75) accounted for 18.3% of the overall variance, with a cumulative percentage of 91.7% for the two significant canonical correlations. Further analysis indicated that the CIRCLE accounted for 50.9% of the variance in the PALS for the first canonical variate. For the second variate, CIRCLE accounted for 16.4% of the variance in the PALS. Examination of canonical loadings on variates associated with the first significant canonical correlation revealed that all were negative and moderate to strong with the exceptions of CIRCLE Listening, Syllabication and Onset Rime, which were negative and low. Examination of canonical loadings on variates associated with the second significant canonical correlation revealed that PALS Upper Case Letters, Lower Case Letters, and Sounds and CIRCLE Letters loaded negatively. Tasks that loaded positively included PALS Print Awareness, Rhyme Awareness, and Nursery Rhyme Awareness and CIRCLE Vocabulary, Listening, Rhyming 1, Rhyming 2, Sentences and Onset Rime. This pattern of loadings suggested that this variate involved phonological awareness (positive loaders) and not letters tasks (negative loaders). It should be noted that the PALS Sounds task involves letters because the student needs to look at the written letter and make the associated sound (See Table 5). EMERGENT LITERACY 11 TABLE 5 Canonical Correlation Analysis for PALS and CIRCLE Canonical Variate 1 2 PALS Name Writing -.65 .13 Uppercase Letters -.96 -.21 Lowercase Letters -.93 -.19 Letter Sounds -.75 -.05 Beginning Sounds -.61 .39 Print Awareness -.62 .60 Rhyme Awareness -.49 .48 Nursery Rhyme Awareness -.56 .68 CIRCLE Letters -.92 -.25 Vocabulary -.51 .56 Listening -.37 .46 Rhyming 1 -.64 .35 Rhyming 2 -.58 .40 Alliteration -.58 .11 Sentences -.53 .40 Syllabication -.28 .28 Onset Rime -.37 .58 DISCUSSION The purpose of this study was to determine the relationship among different measures of emergent literacy skills administered to preschool students, including the relationship between tasks from the different measures that purported to assess similar skills. A further purpose of the investigation was to determine whether there was redundancy, or overlap, among measures in an effort to identify the most efficient and economical means for testing preschoolers. The analyses conducted to address these areas included creation of a multitrait-multimethod matrix and canonical correlations. The multitrait-multimethod matrix (Campbell & Fiske, 1959) was created to assess the construct validity of the three emergent literacy measures used in this study (i.e., PALS, CIRCLE, and IGDI) and it included all of the correlations that resulted across each of the skills, as assessed through various methods (Campbell & Fiske). When correlations across measures were considered by skill, the letter naming tasks correlated highly with one another (range = .82 to .92), although they were administered by different methods, including timed versus untimed and mixed presentation versus upper and lower case letters presented singularly. Considering the time element in administration, it may be more efficient for a preschool teacher to administer a 1-minute timed measure with each student, like the CIRCLE Letters task. Two of the measures, CIRCLE and IGDI, also included vocabulary tasks. For both tasks, the student was presented with a picture of an object commonly found in a preschooler’s school, home, or community environment. A student’s score on each measure was the number of pictures identified correctly in 1-minute. Despite the fact that the two measures were 12 GISCHLAR & SHAPIRO administered in a similar fashion, the correlation between the two was moderate at .42. This suggests that there may have been differences in the types of pictures presented to students, implying that experience and exposure affect performance on measures of preschool vocabulary. Correlations also were calculated for phonological awareness tasks purporting to assess similar skill areas. According to Lonigan (2006a), studies of phonological awareness in young children have demonstrated consistently that regardless of linguistic complexity, differing tasks are indicators of the same underlying ability. This finding might suggest that tasks across the three larger measures assessing phonological awareness should highly correlate. However, the range of correlations was low to moderate (r=.04 – .60). It is unknown to what extent complexity of skill and method of administration (e.g., timed versus untimed, picture versus auditory only presentation) affected student performance. Furthermore, results may have been impacted by factors outside of testing, including how much emphasis was given in the classroom to any particular skill. In addition to the multitrait-multimethod matrix, canonical correlation analyses were conducted. The first set of canonical correlations examined the relationship between the PALS and the IGDI. One significant canonical correlation (canonical R=.78) accounted for 88.8% of the variance in the model. Further, the IGDI accounted for 51.3% of the variance in the PALS, indicating overlap. All tasks across the two measures loaded moderately to strongly on the first, and only significant, variate. The overlap, or redundancy, between the measures suggested that administration of both measures may not have been necessary. When deciding which of the measures to administer, there are several considerations. First, the PALS assesses uppercase and lowercase letters, print awareness, and name writing. Although print awareness has not been found to be a strong predictor of future reading ability (Lonigan et al., 2000), administration of the measure may provide the teacher with additional knowledge about the student’s emergent reading skills and motivation. Further, the PALS includes measures of letter identification and this area has demonstrated substantial predictive relationships with later reading skills (Lonigan et al., 2000). The IGDI does not include a measure of letter naming and, thus, the PALS may prove to be a more comprehensive measure. However, the PALS has some shortcomings that can be addressed by administration of the IGDI. First, administration of the PALS can take 30 minutes or so per student, depending on the length of time the examiner allows for response; the measure is untimed. Because timing is left to the discretion of the administrator, differing amounts of time may be afforded across administration periods and students. This variable may make time and student comparisons difficult, given the lack of uniformity. The IGDI, on the other hand, consists of one 1-minute measure and two 2-minute measures. With time allotted for the sample and practice items, total administration time for the IGDI is approximately 10 minutes and is consistent across students. Also, the IGDI can be used for frequent progress monitoring and each administration is essentially an alternate form, as the teacher needs only to shuffle the cards to change the test. Conversely, the PALS is administered only at the beginning and end of the school year and both assessment periods employ the same form of the test. In deciding whether to use the PALS or IGDI, the professional will have to weigh the costs and benefits of each and choose the measure that best fits his/her needs and available resources. The IGDI also was compared with the CIRCLE measure through canonical correlation analysis. Similar to the previous analysis, one significant canonical correlation (canonical R=.77) was found that accounted for 88.8% of the variance in the model. The IGDI accounted for 38.2% of the variance in the CIRCLE, indicating overlap between the two measures. All tasks across the EMERGENT LITERACY 13 two measures loaded moderately to strongly on the first, and only significant, variate. The overlap, or redundancy, between the measures suggested that administration of both measures may not have been necessary. There also are pros and cons when these two measures are compared. As noted, the IGDI is a progress monitoring tool which can be administered on a frequent basis, so that the teacher can track student progress. The CIRCLE, on the other hand, is administered only at three benchmark periods, fall, winter and spring. Further, whereas the average administration time for the IGDI is 10-minutes per student, the CIRCLE can take as long as 30-minutes; the CIRCLE Letters and Vocabulary measures are timed, but the timing of the seven phonological awareness tasks is left to the discretion of the examiner. Finally, the CIRCLE system, which requires test kits, personal digital assistants (PDAs), and a yearly subscription (Wireless Generation mClass, 2000-2009), may be too costly for preschools to purchase. Despite its shortcomings, the CIRCLE does have a number of benefits to use. First, the PDA administration and scoring helps to limit human error, as the teacher only needs to input student responses. Second, data for each student are directly uploaded to a website that interprets performance and groups students by areas of need and suggests interventions. Finally, the CIRCLE website generates progress reports in a format that is easy for teachers to understand and share with parents and other professionals. It should be noted that the IGDI also has a website for teachers to use (Get It, Got It, Go; http://ggg.umn.edu/) that graphs student progress, but it is up to the professional to interpret results and decide upon and design intervention. The final canonical correlation analysis examined the redundancy between the PALS and CIRCLE measures. In this case, two significant canonical correlations resulted. The first variate (canonical R=.91) accounted for 73.4% of the variance in the model and the second (canonical R=.75) accounted for 18.3% of the overall variance; the cumulative percentage was 91.67%. The CIRCLE accounted for 50.9% of the variance in the PALS for the first canonical variate and 16.4% of the variance for the second variate. This redundancy, or overlap, between the measures suggested that administration of just one of the measures may have been sufficient in assessing student performance. Arguments for and against administering each of these measures have been made. To summarize, the PALS is more economical financially than the CIRCLE, but takes longer to administer. Both measures offer online data interpretation and suggested activities for instruction and intervention, which make them both teacher-friendly. One advantage of the CIRCLE measure is the winter assessment period that is absent from the PALS. This mid-year test aids the teacher in deciding whether instruction and intervention are working for a particular student, rather than waiting until the end of the year, as with PALS, when it is too late to make adjustments. Although the multitrait-multimethod matrix evidenced low to moderate correlations among tasks purporting to measure similar skills, the canonical correlations indicated redundancy across the larger measures. These findings suggested that not all of the larger measures employed were necessary to administer. The high correlations for the letter naming tasks indicated that method of administration was not a factor that affected student outcomes. Similar results should be obtained whether the student is presented with all letters or a sampling of uppercase and lowercase forms in a 1-minute timed assessment. However, the low to moderate correlations among the PA tasks suggested that the method of testing may be important to student performance outcomes. Certainly, further research is needed to determine exactly which combination of the tasks would be the most efficient and economical. Any new 14 GISCHLAR & SHAPIRO combination would need to be evaluated for reliability and validity of results and utility of administration. Although this investigation adds to the emergent literacy assessment literature base, it does have several limitations that should be noted. First, the relatively small sample size (i.e., n=79) limited the types of analyses that could be conducted. Although the canonical correlation analyses suggested redundancy, or overlap, among the three measures, a technique such as structural equation modeling (SEM) may have provided more information about the relationships between the larger tests and their component tasks. This increased knowledge would have been helpful in understanding whether the three different measures truly assessed similar constructs, as suggested by task names, and would be important to future test development. Furthermore, all participants attended the same Head Start program, which had been the recipient of an Early Reading First grant. Students in these classrooms may have received more intensive instruction in emergent literacy areas than typically found in preschool classrooms, given the focus of the grant. It is unknown to what extent the findings would generalize to other preschool programs, or even to other Head Start programs. Certainly, further research is needed with a larger, more diverse population to allow for more sophisticated analyses with SEM. A further limitation of this study was that it included only primary English-speaking 4- year-old students. The Head Start classrooms that participated also provided services for 3-year- olds, but the decision was made to exclude them from analyses, as they were not assessed with all measures and, thus, comparisons would have been limited. However, with today’s increasing focus on early monitoring and intervention, questions must be raised about how inclusion of the younger students may have impacted results; this is an area ripe for future study. Finally, this study was limited to the examination of just three emergent literacy measures and one of those measures, the CIRCLE (The CIRCLE Group, 2004) had limited psychometric reports available for review. Certainly, there are other measures of emergent literacy and also “readiness” assessments that include literacy scales that should be investigated. The examination of other groups of measures through the analyses used in this study may result in different findings. More exploration in this area is needed. Despite its limitations, this study addressed an area that begs further exploration. Because reading is crucial to one’s academic and life success (Cunningham & Stanovich, 1997; Snow, Burns, & Griffin, 1998), effort should be focused on early assessment and intervention. Future studies should focus on which PA tasks and methods of assessment will provide the early childhood educator with the most information about a student to enable data-based decision making. Certainly it is important to identify the most economical and efficient means for assessing literacy skills in young children to promote sound instruction and intervention without wasting valuable classroom time in assessment that may not provide adequate information about the child for decision making purposes. REFERENCES Brownell, R. (Ed.) (2000). Expressive One-Word Picture Vocabulary Test. Novato, CA:Academic Therapy Publications. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. CIRCLE Group. (2004). mClass CIRCLE. Houston, TX: The University of Texas Health Science Center at Houston. EMERGENT LITERACY 15 CTB/McGraw-Hill. (1990). Developing Skills Checklist. Monterey, CA: Author. Cunningham, A. E., & Stanovich, K. E. (1997). Early reading acquisition and its relation to reading experience and ability 10 years later. Developmental Psychology, 33, 934-945. doi: 10.1037/0012-1649.33.6.934. Dickinson, D. K., McCabe, A., Anastasopoulos, L., Peisner-Feinberg, E. S., & Poe, M. D. (2003). The comprehensive language approach to early literacy: The interrelationships among vocabulary, phonological sensitivity, and print knowledge among pre-school aged children. Journal of Educational Psychology, 95, 465-481.doi: 10.1037/0022-0663.95.3.465. Dictionary.com (2008). Dictionary. Retrieved August 12, 2008 from http://www.dictionary. reference.com/browse. Early Childhood Research Institute on Measuring Growth and Development (1996-2002).Checklist for Accurate Administration. Downloaded January 3, 2006 from http://ggg.umn.edu/get/procedures_and_materials/Checklist.pdf Early Childhood Research Institute on Measuring Growth and Development (1998). Technical Report #2: Selection of General Growth Outcomes for Children Between Birth and Age Eight. Minneapolis, MN: University of Minnesota. Epstein, A. S., Schweinhart, L. J., DeBruin-Parecki, A., & Robin, K. B. (2004). Preschool assessment: A guide to developing a balanced approach. Retrieved February 2, 2007 from http://nieer.org/resources/policybriefs/7.pdf Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299. doi: 10.1037/1082-989X.4.3.272. Good, R. H., III, & Kaminski, R. A. (2002). Dynamic Indicators of Basic Early Literacy Skills (6 th ed.). Eugene, OR: Institute for the Development of Educational Achievement. Ivernizzi, M., Sullivan, A., Meier, J., & Swank, L. (2004). Phonological Awareness Literacy Screening: PreK. Charlottesville, VA: University of Virginia. Juel, C. (1988). Learning to read and write: A longitudinal study of 54 children from first through fourth grades. Journal of Educational Psychology, 80, 437-447. doi: 10.1037/0022-0663.80.4.437. Kirby, J. R., Parrila, R. K., & Pfeiffer, S. L. (2003). Naming speed and phonological awareness as predictors of reading development. Journal of Educational Psychology, 95, 453-464. doi: 10.1037/0022-0663.95.3.453. Landry, S. H., Swank, P. R., Smith, K. E. Assel, M. A., & Gunnewig, S. B. (2006). Enhancing early literacy skills for preschool children: Bringing a professional development model to scale. Journal of Learning Disabilities, 39, 306-324. Lonigan, C. J. (2006a). Conceptualizing phonological processing skills in prereaders. In D. K. Dickinson & S. B. Neuman (Eds.), Handbook of early literacy research (Vol. 2, pp. 77- 89). New York, NY: The Guilford Press. Lonigan, C. J. (2006b). Development, assessment, and promotion of preliteracy skills. Early Education and Development, 17, 91-114. doi: 10.1207/s15566935eed1701_5. Lonigan, C. J., Burgess, S. R., & Anthony, J. L. (2000). Development of emergent literacy and early reading skills in preschool children: Evidence from a latent-variable longitudinal study. Developmental Psychology, 36, 596-613. doi: 10.1037//OOI2-1649.36.5.596. Missall, K. N. & McConnell, S. R. (2004). Psychometric Characteristics of Individual Growth and Development Indicators: Picture Naming, Rhyming, and Alliteration. Minneapolis, MN: University of Minnesota. National Center for Education Statistics. (2001). Chapter 5: Outcomes of education. Digest of Education Statistics. Retrieved February 26, 2007 from http://nces.ed.gov// pubs2002/ digest2001/ch5.asp. National Institute of Child Health and Development Early Child Care Research Network. (2005). Pathways to Reading: The role of oral language in the transition to reading. Developmental Psychology, 41, 428-442. Roberts, T. A. (2003). Effects of alphabet-letter instruction on young children’s word recognition. Journal of Educational Psychology, 95, 41-51. doi: 10.1037/0022-0663.95.1.41. Roberts, T. A. (2005). Articulation accuracy and vocabulary size contributes to phonemic awareness and word reading in English language learners. Journal of Educational Psychology, 97, 601-616. doi: 10.1037/0022- 0663.97.4.601. Schappe, J. F. (2005). Early childhood assessment: A correlational study of the relationships among student performance, student feelings, and teacher perceptions. Early Childhood Education Journal, 33, 187-193. doi: 10.1007/s10643-005-0046-y. Schickedanz, J. A., Dickinson, D. K. & Charlotte-Mecklenberg Schools. (2004). Opening the World of Learning: A Comprehensive Early Literacy Program. Pearson Early Learning Group. Snow, C. E., Burns, M. S., & Griffin, P. (Eds.) (1998). Preventing Reading Difficulties in Young Children. Washington, D. C.: National Academy Press. 16 GISCHLAR & SHAPIRO Snow, C. E., & & Páez, M. M. (2004). The Head Start classroom as an oral language environment: What should the performance standards be? In E. Zigler and S. J. Styfco (Eds.) The Head Start Debates. Baltimore, MD: Paul H. Brookes Publishing Co. van Kleek, A., & Schuele, C. M. (2010). Historical perspectives on literacy in early childhood. American Journal of Speech-Language Pathology, 19, 341-355. doi: 10.1044/1058-0360(2010/09-0038. Wagner, R. K., Torgesen, J. K., Rashotte, C. A. (1999). Comprehensive Test of Phonological Processing. Austin, TX: Pro-Ed. Whitehurst, G. J., Zevenbergen, A. A., Crone, D. A., Schultz, M. D., Velting, O. N., & Fischel, J. E. (1999). Outcomes of an emergent literacy intervention from Head Start through second grade. Journal of Educational Psychology, 91, 261-272.