EPAAv20n17


 

Manuscript received: 05/23/2011 
Revisions received: 11/30/2011 

Accepted: 12/04/2011 

 

education policy 
analysis archives 
A peer-reviewed, independent,  
open access, multilingual journal  
 
 
 
 
 
  

 Arizona State University 
 

Volume 20  Number 17  June 20, 2012 ISSN 1068-2341 
 

 
An Empirically-based Statewide 

System for Identifying Quality Pre-Kindergarten Programs 
Jeffrey M. Williams 

Susan H. Landry 
Jason L. Anthony 

Paul R. Swank 
April D. Crawford 

 University of Texas Health Science Center Houston 
 

Citation: Williams, J. M., Landry, S. H., Anthony, J. L., Swank, P. R., & Crawford, A. D. (2012). An 
Empirically-based Statewide System for Identifying Quality Pre-Kindergarten Programs.  Education 
Policy Analysis Archives, 20 (17). Retrieved [date], from http://epaa.asu.edu/ojs/article/view/1014  
 
Abstract: This study presents an empirically-based statewide system that links information about 
pre-kindergarten programs with children’s school readiness scores to certify pre-kindergarten 
classrooms as promoting school readiness. Over 8,000 children from 1,255 pre-kindergarten 
classrooms were followed longitudinally for one year. Pre-kindergarten quality indicators of 
intentional instruction, an early literacy focus, and professional development were key predictors 
of kindergarten outcomes. A latent profile analysis identified pre-kindergarten classrooms that 
were high on pre-kindergarten quality indicators and high on kindergarten outcomes (67.3%), 
low on pre-kindergarten quality and kindergarten outcomes (21.3%), or low on quality but high 
on outcomes (11.4%). The last group of classrooms was likely to serve middle-class children and 
not use the state program model. This project demonstrates how a scientific approach can 
inform stakeholders and parents about the effectiveness of early childhood programs.  
Keywords:  preschool; kindergarten; emergent literacy; social development; school readiness; 
quality rating system; state policy. 



Identifying Quality Pre-Kindergarten Programs  2 

 

 

 
Sistema estatal basado en datos empíricos para identificar programas de educación pre-
escolar de calidad  
Resumen: El presente estudio presenta un sistema estatal de bases empíricas que vincula la información 
de los programas de pre-kínder con los resultados de la preparación preescolar para certificar que los 
salones de pre-kínder promueven las habilidades básicas para la escuela primaria. Más de 8,000 niños de 
1,255 salones de pre-kínder fueron monitoreados longitudinalmente por un año. Tres indicadores de la 
calidad en pre-kínder: instrucción intencional, foco en alfabetización temprana y desarrollo profesional-, 
fueron claves para predecir los resultados en kínder. Un análisis de perfil latente identificó los salones de 
pre-kínder con buenos resultados en los indicadores de calidad y en los resultados de kínder (67.3%), a 
los salones bajos en la calidad de pre-kínder y en los resultados de kínder (21.3%), y a los salones  bajos 
en calidad pero altos en resultados (11.4%). Este último grupo de salones eran los más probables de 
atender a niños de clase media y de no usar el programa modelo del estado. Este proyecto demuestra 
cómo un enfoque científico puede dar información a los padres y a las partes interesadas sobre la 
efectividad de los programas de educación preescolar. 
Palabras clave: pre-escolar; kindergarten; alfabetización emergente; desarrollo social; adecuación 
para escolaridad; sistema de medición de calidad; política de estado. 
 
Sistema estatal com base em dados empíricos para identificar programas de qualidade pré-
escolar 
Resumo: Este trabalho apresenta um sistema de base empírica estatal que liga informações de 
programas de pré-escola  com os resultados de preparação para a escola para certificar que as salas 
de aula do jardim de infância promovem as competências básicas para a escola primária. Mais de 
8.000 crianças de pré-jardim em 1.255 salas de aula foram monitorados longitudinalmente por um 
ano. Três indicadores de qualidade da instrução pré-K: instrução intencional, foco em alfabetização, 
e desenvolvimento profissional, foram fundamentais para prever os resultados no jardim de infância. 
A análise do perfil latente identificados pré-escola salas de aula com bons resultados nos indicadores 
de qualidade e resultados na educação infantil (67,3%), as salas de resultados de baixa qualidade pré-
jardim de infância e creche (21,3 %) e baixa salas de aula de qualidade, mas alta no desempenho 
(11,4%). Este último grupo de salas tinham maior probabilidade de servir as crianças de classe média 
e não usar o programa do estado do modelo. Este projeto demonstra como uma abordagem 
científica pode fornecer informações para os pais e as partes interessadas sobre a eficácia dos 
programas pré-escolares. 
Palavras-chave: infância, pré-escolar, desenvolvimento da alfabetização, o desenvolvimento social, 
aptidão para o ensino, sistema de medição de qualidade, política de estado. 

Introduction 

There is increasing recognition of the importance of the early childhood period as a critical 
time for promoting children’s learning in order to assure readiness for success in school (Campbell, 
Ramey, Pungello, Sparling, & Miller-Johnson, 2002; DiPietro, 2000; National Research Council, 
2001). To reach this goal early childhood programs are provided through a variety of service delivery 
models. Public school pre-kindergarten in most states is funded primarily for children from lower 
income backgrounds and federally funded Head Start programs also primarily target very low-
income families. Childcare programs serve children from all income levels and typically provide 
longer hours of service to meet the needs of working families. Given heightened public awareness of 
the importance of early childhood programs in preparing children to be school ready and the 
multiple service delivery models in which this occurs, systems are needed to determine if, in fact, 
children are school ready after leaving these programs. Recent reports describe how quality rating 



Educational Policy Analysis Archives Vol. 20 No. 17  3 

 

systems may need to target the performance of programs to advance children’s learning so that 
school ready goals are assured (e.g., foundation skills necessary for reading and social competence; 
Gallagher, Clifford, & Maxwell, 2004; Kagan & Rigby, 2003; Kauerz, 2008).  

States are faced with a number of challenges as they strive to develop early childhood 
education systems that meet the social and academic needs of children, particularly those from low 
income backgrounds, using current funding streams (Kauerz, 2008). As children from low income 
backgrounds are known to have less exposure to rich vocabulary, fewer experiences with books, and 
less parental responsiveness to children’s interests (Evans, 2004; Hart & Risley, 1995; Landry, Smith, 
Swank, Assel & Vellet, 2001), there is increasing belief in the United States that high quality 
preschool programs should be utilized as a means of assuring that children from low socioeconomic 
(SES) backgrounds are prepared to start school in kindergarten. In addition, irrespective of how 
early childhood programs are funded, parents are in need of evidenced-based information regarding 
the extent to which an early childhood program prepares a child to be school ready.  

Shortcomings of Current Quality Rating Systems 

National organizations like the National Association for the Education of Young Children 
(NAEYC, 2008) and the National Association of Early Childcare Professionals (NAECP, 2008), 
conduct evaluations of childcare programs to determine if they meet a broad range of quality 
indicators that, in turn, result in accreditation. Many of the indicators included in these evaluation 
systems are conceptually derived and/or based on long-standing beliefs in the field of early 
childhood, such as the quality of conversations between teachers and children and availability of 
materials for children’s exploration. Accreditation from these national organizations can provide 
important information to families about quality indicators of a program. However, many preschool 
programs are not able to apply for accreditation because of the costs involved. For example in many 
states, only a small percentage of childcare and Head Start programs seek these accreditations. 
Additionally, few public school programs seek accreditation by national organizations. In light of 
this, state education leaders and policy makers are looking for cost-effective approaches to 
document whether early childhood programs from various service delivery models are adequately 
preparing children for kindergarten.  

Several states across the nation (Colorado, Washington, Pennsylvania, Oklahoma, North 
Carolina, and Ohio; Zellman & Perlman, 2008) have implemented quality rating systems for early 
childhood education. (e.g., Zellman, Perlman, Le, & Setodji, 2008) However, although recent reports 
emphasize the need for outcome-based accountability and information about program effectiveness 
(Kaurtz, 2008; Zellman, Brandon, Boller, & Kreader, 2011), only two systems (Colorado’s Qualistar 
Rating System and Washington’s Seeds to Success) directly link accreditation to the extent to which 
a preschool program has actually provided children with the skills necessary for success in 
kindergarten. To do that, an evaluation system must assess characteristics of program quality and 
relate those characteristics to children’s learning. Current accreditation procedures that do not 
consider this link may involve a great deal of time and financial resources in the process of capturing 
program, classroom, and teacher characteristics that are not meaningfully related to children’s school 
readiness. In the initial development phase of the Texas School Ready system described here, policy 
makers prohibited the use of child-level outcome data during the preschool year due to concerns 
about high-stakes testing of preschool children. However, schools in Texas are required to 
administer an approved literacy screener within the first six weeks of children’s entry into 
kindergarten. No current evaluation system considers the link between preschool quality and 
children’s school readiness when they are actually entering school.  



Identifying Quality Pre-Kindergarten Programs  4 

 

 

Rationale for Development of a New Quality Rating System  

 There are many reasons for identifying which early childhood programs are effective in 
preparing young children for school. While recent large scale reports indicate that high quality early 
childhood centers can be as cognitively stimulating as home care in middle-class families, working 
parents of lower income have less access to such high quality centers (Phillips, Voran, Kisker, 
Howes, & Whitebook, 1994). Moreover, it is high quality, early childhood programs that have 
demonstrated a positive influence on children’s school readiness (e.g., Bierman, Domitrovich, Nix, 
Gest, Welsch, Greenberg, et al., 2008; Hindson, Byne, Fielding-Barnsley, Newman, Hine, & 
Shankweiler, 2005; Reynolds, Ou, & Topitzes, 2004). Thus, one of the most important reasons for 
identifying effective programs is to provide parents with information that they can use to guide their 
decisions regarding selection of the best program for their child. Of concern is that families at 
poverty levels are more likely to rely on center-based programs that accept federal subsidy (Center 
for the Childcare Workforce, 2004) and such programs tend to have very limited funding earmarked 
for quality enhancement. Thus, programs that accept federal subsidy tend to suffer from underpaid 
and poorly trained staff and a dearth of appropriate learning materials (e.g., books, puzzles, blocks). 
As a result, young children from low SES homes, who are at greatest risk for developmental 
difficulties and scholastic unpreparedness, are spending time in early childhood centers that may be 
less likely to provide high quality stimulation and responsiveness as part of their instructional 
practices.  
 A second compelling reason to develop quality rating systems for early childhood programs 
that are linked to child outcomes is for accountability and program development. Legislators and 
others responsible for funding are beginning to insist on evaluative information in order to sustain 
or increase program funding. Additionally, those responsible for program management and 
implementation (e.g., director, principal, teacher) could use evaluative information to inform their 
educational practices. While self-assessment is often conducted, the extent to which this is used to 
enhance program quality is not well established. This often occurs because it is not collected in a 
form that can be used to inform and guide programmatic changes. An optimal quality rating system 
would provide programs that are in need of improvement with an individualized training and 
technical assistance plan. 

While there are numerous reasons for using quality rating systems, there are also many 
challenges. For example, funding for rating systems on a statewide basis is often restricted, and 
therefore, approaches need to be efficient. Efficiency means finding methods that are reliable and 
valid but at the same time cost-effective. For example, whereas direct observations of preschool 
programs and classrooms over multiple visits are currently the norm, these methods are costly with 
respect to travel and personnel time. If programs must bear these costs then the expense is often not 
feasible for small programs and those whose operations are dependent upon federal subsidy. Also, 
rating systems that evaluate program quality need to be equally applicable across all types of service 
delivery systems (e.g., public school, Head Start, childcare). For example, a good quality rating 
system must not be biased for or against programmatic differences that exist institutionally as a 
function of service delivery systems. Otherwise, the quality rating system would not be fully 
endorsed or implemented, which would negate the potential benefit of being able to compare 
preschool programs that function under different auspices.  

Sociopolitical Context of the Present Study 

The goal of the present study is to describe the development of an empirically-based quality 
rating system for early childhood programs, an effort directed by a state legislated request to first 
integrate delivery of early childhood education services for preschool-age children from 
economically disadvantaged families. This 2003 request first encouraged early childhood education 



Educational Policy Analysis Archives Vol. 20 No. 17  5 

 

programs (i.e., child care, school districts, Head Start) to come together in a seamless mixed delivery 
model and adopt similar standards and practices, and a second legislative request (2005) was to 
develop quality rating systems. Approximately 1000 pre-kindergarten classrooms in Texas voluntarily 
participated in the integrated service delivery model during the time of the present study. The three 
types of early childhood programs have differences in policies about service focus in areas such as 
education, health, and social services (Gallagher et al., 2004). However, the emphasis in this new 
integrated service delivery model, called the Texas Early Education Model (TEEM; Landry, Swank 
Anthony & Assel, 2011), is on preparing children to be school ready upon entrance into 
kindergarten by providing them with the foundational language, literacy, mathematics, and social 
skills necessary to succeed in kindergarten. Programs enrolled in TEEM share a common set of 
supports, including professional development, provision of curricular materials, training in a 
research-based curriculum, a system for assessing children’s progress in cognitive and social skills, a 
set of state learning standards, and classroom mentors. A recent paper described this integration 
model and its efficacy in terms of both changes in preschool teachers’ instructional practices and 
children’s school readiness (Landry et al, 2011). 

Purpose of the Present Study 

The primary objective of this report is to describe the development of a scientifically-based 
and empirically-derived system that is being used across Texas to rate the quality of pre-kindergarten 
classrooms. The School Readiness Certification System (SRCS) is not intended to replace other 
accreditation standards, such as those offered by NAEYC or state licensing bodies. Instead, it was 
designed to provide specific information about how programs prepare pre-kindergarten children for 
school so that parents can make informed decisions about early childhood education programs. The 
SRCS also was designed to provide information to guide technical assistance for those programs that 
do not meet certification criteria. In addition, the SRCS was designed to have the advantage of being 
equally available to all types of early childhood program providers, adding to its utility and provision 
of greater choice to parents. Finally, the SRCS was developed under the guiding notions that the 
rating system should be objective and evidence-driven and that it should maintain a concentrated 
focus on indicators that predict school readiness.  

The theoretical, conceptual, and practical underpinnings of the SRCS were developed in 
conjunction with state educational leadership, national experts in early childhood, and community-
based stakeholders (e.g., directors, teachers, parents). The agreed upon objective was that the SRCS 
should be a statewide system, applicable across service delivery systems, that incorporates 
information about the quality of pre-kindergarten programs in combination with children’s early 
reading and social competencies, which are assessed by the public schools when children are in 
kindergarten. For the SRCS to be feasible for implementation across a large state with funding 
constraints, it had to be cost-effective yet still reliable and valid. These aims and constraints required 
that the development of the SRCS involve multiple steps, including (a) development of a web-based 
application system for pre-kindergarten programs, (b) documentation of the reliability and validity of 
data provided by early childhood program staff, (c) determining the feasibility of various procedures 
for tracking children from pre-kindergarten into kindergarten, (d) determination of which pre-
kindergarten variables reliably predict kindergarten school readiness in this population, and (e) use of 
statistical models to ascertain the best criteria for certifying classrooms as “School Ready”. The 
present report briefly summarizes the development stage and two pilot studies for the sake of 
completeness, while it fully details the scientific underpinnings used to establish the certification 
criteria.  



Identifying Quality Pre-Kindergarten Programs  6 

 

 

Methods 

Pilot Studies 

Development and validation of the Teacher Self-Report survey. The first pilot study was conducted 
during the 2004/2005 academic school year for the purpose of developing and validating a set of 
survey questions that teachers could answer as part of a web-based application for certification. A 
pool of 74 items was generated based on current research and best practices in the pedagogy of 
promoting early childhood learning across social, language, literacy, and math domains. Included 
were questions about responsive teaching practices, classroom arrangement and organization, daily 
routines, lesson planning, monitoring progress of children’s learning, use of small vs. large group 
activities, and classroom curriculum and materials. The broad item content required individual items 
to vary in scaling metric, and no scale structure was presumed because of the newness of the survey. 
Ninety-nine teachers completed the survey online. 

Given that certification would be based, in part, on teachers’ self-report of their teaching 
practices and characteristics of their classrooms, it was imperative to validate data gathered from the 
Teacher Self-Report survey with independent observations. Trained research assistants observed the 
99 teachers who completed the Teacher Self-Report. Following the standardized procedures 
prescribed for the Teacher Behavior Rating Scale (TBRS; Landry, Crawford, Gunnewig & Swank, 2002), 
observations lasted approximately two and one half hours during the time of the day that was 
devoted to instruction aimed at cognitive and school readiness skill acquisition. The TBRS is a 101 
item, standardized classroom observation system that has been used in a number of state and 
national program evaluation projects (e.g., national evaluation of Early Reading First). Observers 
rate the quality and frequency of occurrence of specific teaching behaviors known to predict 
children’s school readiness. The teacher behaviors that are rated fall into twelve conceptual 
categories, namely oral language, print & letter knowledge, phonological awareness, written 
expression, book reading, centers, mathematics, lesson planning, team teaching, progress 
monitoring, portfolios, and classroom management and responsive teaching practices that are not 
content specific. Observers rate the frequency of occurrence of each target behavior (i.e., 0-1 
occurrence, 2-3 occurrences, or 4 or more occurrences). Observers also rate the quality of each 
target behavior as low, medium, or high. Thus, there are separate quality and frequency summary 
scores for each of the twelve scales of the TBRS. Inter-rater reliability using generalizability 
coefficients (Mitchell, 1979) range from .80 to .98, and internal consistency, for the measure as a 
whole, using Cronbach’s alpha is = .96 (Landry et al., 2002).  

Zero-order correlations of TBRS scales with questions from the Teacher Self-Report were 
tested for significance. In general, questions from the Teacher Self-Report were more highly 
correlated with conceptually similar TBRS scales than with conceptually dissimilar TBRS scales. 
Those survey questions that had three or more significant correlations above r = .25 with 
conceptually overlapping TBRS scales were retained and left unchanged. Survey questions that 
yielded significant but small correlations with conceptually overlapping TBRS scales were reviewed 
for possible sources of confusion and if the source was obvious then they were reworded and 
retained. Survey items not reliably correlated with any like TBRS scale were dropped. Most 
constructs measured by the Teacher Self-Report were indexed by a number of survey items that did 
demonstrate good convergent validity with similar TBRS scales. There were two exceptions. None 
of the Teacher Self-Report items that inquired about responsiveness to children’s needs (learning, 
emotional, or otherwise) or behavioral management of the classroom demonstrated convergent 
validity with like TBRS scales. Thus, in general, results provided preliminary validation that teachers 
were accurately reporting their instructional and classroom practices in ways that could be 
corroborated by an independent observer. 



Educational Policy Analysis Archives Vol. 20 No. 17  7 

 

Feasibility of procedures for tracking children and obtaining assessment data. Feasibility studies were 
conducted during the 2005/2006 school year. Given the many possible challenges to linking pre-
kindergarten school data with children’s kindergarten outcome data, it was necessary to examine the 
practical and fiscal feasibility of various procedures. For example, a variety of methods of tracking 
children through their pre-kindergarten and kindergarten years were tested. This often involved 
children transferring among multiple schools. To accomplish tracking of children, coordination of 
efforts were required among the Texas Education Agency, regional education service centers, 
independent school districts, early childhood education agencies, and Head Start grantees. We also 
piloted a variety of methods aimed at collecting kindergarten assessment data from kindergarten 
teachers, elementary schools, independent school districts, and third party vendors. Examples of 
different methods included sharing of electronic data, development and sharing of protected Excel 
templates, testing of individual children by research staff, and a third party secured web-based 
application. Ultimately, kindergarten assessment data were gathered from 774 children who attended 
20 public school districts across the state.  

The feasibility studies revealed that state legislation was needed to increase schools’ 
compliance with Texas Education Agency’s recommendation to provide kindergarten assessment 
data. It was also determined that a web-based application was optimal for gathering kindergarten 
assessment data. For example, restrictions could be programmed in a web-based system to require 
uniform reporting of assessment data, which was determined necessary given that schools around 
the state were recording the assessment data in different ways (e.g., item level scores, subtest raw 
scores, categories of competency across the entire screening measure). Finally, the feasibility studies 
demonstrated that it would be more cost effective to have the state education agency perform a large 
scale matching of pre-kindergarten children’s demographic data with those of enrolled 
kindergarteners the next year than to have regional education service centers explore matches 
individually. 

Participants in the Main Demonstration Project 

Participants included 538 preschool sites, 1,326 pre-kindergarten classrooms, and 12,585 
preschool-age children. Preschools were distributed across three types of service delivery systems; 
258 childcare, 147 public school and 133 Head Start schools. In each community partnership there 
was representation of classrooms from each type of delivery system. The lead agency in the 
community partnership (i.e., public school, Head Start) held regular meetings to inform the 
participating programs about the application process. The Head Start programs represented all types 
of federally funded grantees, including Community Action, school district, and State Region Service 
Center programs. The childcare programs included for-profit and not-for-profit and ranged from 
small centers to large YMCA programs. Of the 1,255 classrooms from which data were gathered, 
1,024 were participating in the voluntary, state integration program model that focused on programs 
serving low-income children. The 231 classrooms that did not participate in this model were invited 
to participate in the current study to increase the heterogeneity of the sample, as these 231 pre-
kindergarten classrooms served children from middle-income families.  

In the spring of 2006, the preschool programs submitted an on-line application that included 
information detailed in the Procedures section. Programs were aware that though this was a 
demonstration year, their participation would result in a decision about their certification status. In 
the fall of 2006, school readiness scores (reading and social screening measures) were obtained from 
kindergarten classrooms for approximately 8,000 (73%1) of the children from these preschool 
                                                
1 Many children attended kindergarten in districts where an assessment other than the Texas Primary Reading 
Inventory (TPRI) or Tejas LEE (TJL) was administered so these children’s data could not be used, and some  
children’s data did not contain enough information for a definitive match. 



Identifying Quality Pre-Kindergarten Programs  8 

 

 

classrooms. Fifty-one percent of the children followed were boys. Children’s average age on the first 
day of kindergarten was 5.5 years (SD = .30). The number of days children attended preschool 
ranged from 1 to 250, with an average of 156 days (SD = 35.5). The sample of children was 
ethnically diverse; 69% Hispanic, 17% Caucasian, 13% African American, and 1% American Indian, 
Asian, or Pacific Islander. A large percentage of children (i.e., 67%) were eligible for free and 
reduced lunch and 48% had limited proficiency in English.  

Procedures 

e-School Plus  

The web-based application had three components: (a) preschool facility report, (b) manage 
my school and students, and (c) teacher self-report. Information gathered in the three components 
included characteristics of the preschool program and preschool classroom, teaching practices, and 
teaching beliefs, based on empirical research that has identified components associated with high 
quality programs (National Early Literacy Panel, 2008). For example, information was gathered 
concerning aspects of the classroom environment (e.g., availability of small group learning areas, 
space for children to gather for a large group activities, literacy richness)(Crosser, 1992; Dunn, 
Beach, & Kontos, 1994), instructional practices (e.g., intentional language and literacy instruction as 
well as time for children to explore and practice for mastery; National Research Council, 2001; 
Raver & Knitzer, 2002), curriculum (e.g., availability of research based scope and sequence for 
cognitive and social learning; Assel, Landry, Swank, & Gunnewig, 2007; Whitehurst, Epstein, Angell, 
Payne, Crone, & Fischel, 1994), assessment approaches (reliable and valid information about 
children’s learning across the year with feedback to teachers to direct instruction; Phaneuf & 
Silberglitt, 2003), and professional development (e.g., small group, learning across time, support for 
opportunities to practice in the classroom; Elmore, 2002; National Commission on Teaching & 
America’s Future, 1996). The application included the following components.  

Preschool facility report. The director of a child care or Head Start program or principal of a 
public school pre-kindergarten program provided information on key areas of their program 
including: (a) accreditation/ licensing, staffing patterns (i.e., teacher education, number of teachers 
and teaching assistants), (b) staff credentials (i.e., state certified, bilingual, early childhood, English as 
a second language, reading specialist, child development associate-CDA), (c) program/classroom 
characteristics (i.e., full vs. half day, language of instruction, number of books), (d) curriculum used 
for literacy, math and social/emotional development, (e) methods and frequency of assessing 
children’s academic knowledge (e.g., standardized, work sampling, checklist, Personal Digital 
Assistance), (f) professional development practices, and (g) community integration. This component 
required about 15 minutes to complete.  

Manage my school and students. Each preschool program was asked to provide information on 
their classroom teachers and those children who would be going to kindergarten the following year 
(e.g., number of preschool classrooms, length of program day, and language of instruction). 
Demographic information was entered for each child (e.g., age, gender, home language) as well as 
information related to the educational experience (e.g., eligibility for free/reduced lunch, special 
education, English language learners). 

Teacher Self-Report survey. Each pre-kindergarten teacher was asked to provide information on 
highest level of education and credentials as well as respond to a series of questions regarding 
practices related to the following: (a) best practices, (b) classroom arrangement, (c) lesson planning, 
(d) child monitoring and assessments, (e) social skills development, (f) book reading, (g) 
phonological awareness, (h) print and letter knowledge, (i) written expression, (j) math, and (k) oral 
language instruction. For example, the teacher was asked to respond to a group of questions 



Educational Policy Analysis Archives Vol. 20 No. 17  9 

 

regarding techniques for monitoring children’s progress and teaching practices for book reading, 
phonological awareness, letter knowledge, and written expression. Other questions required the 
teacher to choose from an array of pictures those that were similar to activities they used for 
promoting early literacy skills, early mathematics skills, and classroom organization. In addition, 
teachers’ beliefs regarding children’s needs for standardized classroom practices versus 
individualized instruction were included. Such questions inquired about the extent to which 
individualization of lesson plans and behavior management techniques were used. Completion of 
the Teacher Self-Report required about 20 minutes. 

Teachers also submitted examples of their daily schedules and lesson plans via US Postal 
Services. The daily schedules and lesson plans that teachers submitted varied tremendously in 
amount of detail and intelligibility. As could be expected, teachers approached lesson planning and 
labeling of daily schedules very differently. For example, while some teachers specified learning 
objectives and correlated activities in their lesson plans, other teachers’ lesson plans resembled daily 
schedules that simply listed a time of day and vague classroom activities (e.g., circle time). In short, 
our coding team concluded these data were not amenable to coding for potential inclusion in the 
certification criteria. 

Each teacher submitted 12 photographs of their classroom, either via uploading of digital 
photographs or via mailing of disposable cameras. Specifically, teachers submitted photographs of 4 
walls, 4 corners, and 4 areas of their choice. A 15-item coding system was developed based on the 
Preschool Classroom Environment Checklist (CIRCLE, 2001). Using two- or three-point rating 
scales, research assistants determined the presence, amount, and quality of the classroom 
arrangement, centers, themes, children’s books, writing materials, labels, children’s names and work 
samples, samples of modeled writing, letter wall, daily schedule, helper chart, and props for group 
instruction (a description and scoring examples can be found in Appendix A). Research assistants 
received two days of training, and were certified for coding once an average agreement with an 
expert coder exceeded 92% for each item. Raters then attended bi-weekly coding meetings to 
discuss scoring questions and prevent coder drift. Ten percent of the pictures were randomly 
selected for recoding by a different rater to ensure inter-rater reliability throughout the coding period 
with reliability exceeding 0.80 (Mitchell, 1979).  

Kindergarten School Readiness Outcomes 

To identify each child’s kindergarten teacher, the first step was to fund the 22 state regional 
education service centers to perform a series of searches in which children’s information from e-
School Plus was matched to children’s information included in the Texas Education Agency’s 
centralized databases. Information used to match children’s data included social security number, 
when available, and children’s demographic data, including name, gender, ethnicity, birthday, 
free/reduced lunch status, etc. When this matching process yielded a partial match, decisions were 
made on a case-by-case basis. When a student’s information matched two records in the TEA’s 
database equally well (i.e., partial match with both records), neither was used. Matches yielded the 
name of a school that a given child was believed to attend. Next, a unique identifier was assigned to 
each “matched” child who attended public or charter schools in the state.  

Schools that included children “matched” through this process were sent a letter and asked 
to provide the names and contact information of the child’s kindergarten teacher. These teachers 
were contacted via the US Postal Service, email, and telephone and provided with a password and 
instructions for how to login to e-School Plus and how to enter children’s school readiness data. 
School readiness data included scores from the district administered reading screenings (see below) 
and kindergarten teachers’ responses to a social competence screener (see below). Kindergarten 
teachers’ access to e-School Plus was restricted in such a way that they only saw a list of names of 



Identifying Quality Pre-Kindergarten Programs  10 

 

 

children who attended their particular kindergarten classroom. Finally, because preschool classroom 
was the unit of analysis for school readiness certification (see below), the kindergarten outcome 
scores for all children within a given preschool classroom were averaged to create classroom-level 
pass-rates (Snijders & Bosker, 1999).  

Texas Primary Reading Inventory. About 90% of school districts and charter schools in the state 
of Texas administer the Texas Primary Reading Inventory (TPRI; Texas Educational Agency, 1998) 
two to three times per school year during kindergarten, first grade, and second grade. This measure 
has strong reliability and validity and has been demonstrated to predict end of second grade reading 
(Texas Educational Agency, 2002). For each progress monitoring wave, the inventory includes a 
screener and a more comprehensive inventory. The kindergarten screener consists of two subtests 
that permit rapid determination of whether or not students are at-risk for reading problems at the 
end of first grade. Specifically, the Graphophonemic Knowledge subtest requires children to provide 
the sounds associated with each of ten letters. The Phonemic Awareness subtest requires children to 
(a) blend onset and rime units into real words and to (b) blend up to three individual phonemes into 
real words. Standard administration procedures dictate that if a child scores “Developed” on the 
Graphophonemic Knowledge subtest, then the child is classified as “Developed” on the screener 
and the Phonemic Awareness subtest is not administered. However if a child does not score 
“Developed” on the Graphophonemic Knowledge subtest, then he or she is administered the 
Phonemic Awareness subtest, and if the child achieves a score of “Developed” on the Phonemic 
Awareness subtest then he or she is classified as “Developed” on the screener as a whole. In the 
present study, children’s classification status on the screener was used as the reading outcome 
because only the screener is administered to all children. Because some schools in the state did not 
administer the TPRI as part of their standard operating procedures, research staff individually 
administered the TPRI to approximately 600 children. In total, 76% of the sample followed took the 
TPRI, and the average preschool classroom pass-rate was 65%. 

Tejas Lee. Most children in Texas who speak Spanish in their homes and who are provided 
classroom instruction in Spanish are administered the Tejas Lee (TJL; Texas Educational Agency, 
2002) instead of the TPRI. The TJL, like the TPRI, is used to monitor children’s learning of literacy 
skills two to three times per school year during kindergarten, first grade, second grade and third 
grade. The TJL is not a Spanish translation of the TPRI, however it does assess similar literacy 
component skills. Additionally, the TJL was developed following procedures that closely resembled 
those used to develop the TPRI. Because the 2006/2007 TJL did not include a screener, a scoring 
routine that conceptually paralleled that of the TPRI was used to classify children. Specifically, 
priority was given to the letter sounds subtest such that if children scored “Developed” on this 
subtest then children were classified as “Developed”. If not, then children were still classified as 
“Developed” if they achieved a score of “Developed” on the subtests that involved blending of 
syllables and segmentation of syllables. In total, 24% of the sample followed took the TJL, and the 
average preschool classroom pass-rate was 82%.  

From the TPRI and TJL scores, a single classroom-level Literacy pass-rate was constructed 
by first taking each child’s score, regardless of measure, and then calculating the classroom average. 
Although there are difficulties with aggregating scores from different measures, this was necessary to 
avoid running two sets of certification analyses, which could have potentially created different 
certification criteria for the classrooms serving English- and Spanish-speaking populations. In 
addition, because children from the same preschool classroom could be assessed with different 
reading screeners in Kindergarten, conducting separate analyses could have resulted in different 
certification decisions for the same classroom. 

Social competence and behavior screener. The social competence subscale from the Social 
Competence and Behavior Evaluation (SCBE-30; Dumas, Martinez, & LeFreniere, 1998; Kotler & 



Educational Policy Analysis Archives Vol. 20 No. 17  11 

 

McMahon, 2002; LaFreniere & Dumas, 1996) was used to evaluate children’s social skills with peers 
and teachers in early kindergarten. Specifically, this 10-item, teacher-report checklist measures 
children’s social behavior, such as social integration, tolerance, and cooperation (e.g., comforts or 
assists other children in difficulty, works easily in a group). Each item utilizes a six-point scale 
(1=almost never occurs to 6=almost always occurs). The SCBE-30 has demonstrated high inter-
rater reliability (.83-.87) and good test-retest reliability (.82). The SCBE-30 has been successfully 
validated and used in numerous studies for monolingual as well as for children learning English as a 
second language (e.g., LaFreniere & Dumas, 1996; LaFreniere, Masataka, Butovskaya, Chen, Dessen, 
Atwanger, et al., 2002).  

Results 

 Analysis proceeded in three stages. Stage One involved reducing the number of variables 
from the various components of e-School Plus. This was necessary because the facility report, 
teacher self-report, and classroom picture data comprised over 100 variables. Exploratory and 
confirmatory factor analyses were performed to yield a more manageable number of variables. In 
Stage Two, the resultant factors were combined with the kindergarten outcome data (i.e., classroom-
level reading and social screener averages) and a latent profile analysis was conducted to identify 
classroom characteristics that were associated with children’s school readiness as assessed in 
kindergarten. Because not all classrooms in a facility entered data into the system, classroom was 
deemed to be the most appropriate unit of analysis in order to prevent certifying a school based on 
partial information. Finally in Stage Three, follow-up analyses were conducted to examine the 
composition of the latent profiles in terms of service delivery systems (i.e., Head Start, public school, 
child care) and participation in the state’s integration model (i.e., TEEM).  

Data Reduction 

Overall strategy. Items from e-School Plus and items from the coding system that described 
the photograph data were subjected to separate exploratory factor analyses. Prior to factor analysis, 
all variables were examined for evidence of non-normality, and in cases in which only mild to 
moderate skewness was observed, model parameters were estimated using maximum likelihood with 
robust standard errors. In some instances, values of one or more of the variables in the analysis had 
to be collapsed due to low response frequency. These variables were modeled as ordinal, rather than 
interval, and parameters were estimated using mean- and variance-adjusted weighted least squares 
estimation (Muthén, du Toit, & Spisic, in press). Varimax or Promax rotation was used, based on the 
size of factor correlations. Once the best exploratory model had been determined, confirmatory 
factor analysis was used to generate factor scores for each factor retained in the EFAs. This 
procedure was used to optimize interpretability of factor scores, by permitting no crossloadings.  

Teacher Self-Report survey. Eight items from the teacher survey were dropped because they did 
not correlate with any other variables (four items about book & print awareness, one about 
classroom arrangement, one about lesson planning, one about print and letter knowledge, and one 
about the number of teacher aides) and an additional four items were dropped because they failed to 
produce unique loadings that exceeded .30 on any factor. Exploratory factor analysis2 of the 
remaining 37 items yielded eight correlated factors with eigenvalues greater than 1.0. The scree plot 
was consistent with either a three factor solution or a six factor solution. The six factor solution was 

                                                
2 For these and all other analyses of teacher and classroom variables, multilevel analyses (teacher/classroom 
nested within school) were attempted.  However, due to the preponderance of schools for which only one 
teacher or classroom existed in the data (49%), these models consistently failed to converge or had problems 
with the parameter estimation.  Therefore, traditional, single-level analyses were conducted.   



Identifying Quality Pre-Kindergarten Programs  12 

 

 

judged interpretable, and yielded a root mean square error of approximation (RMSEA) of .04, which 
reflects a good fit when considering the complexity of the model (see Table 1 for factor reliabilities 
and Table 2 for description of the six survey factors).  
 
Table 1 
Factor Reliabilities 
Section Factor Number of 

items 
Cronbach's 
alpha 

Teacher Survey Shared reading 4 0.53 
Teacher Survey Math instruction 7 0.87 
Teacher Survey Assessment 3 0.49 
Teacher Survey Intentional instruction 9 0.57 
Teacher Survey Early literacy 8 0.76 
Teacher Survey Lesson planning 6 0.41 
Classroom pictures Center quality 6 0.59 
Classroom pictures Visual supports 4 0.50 
Professional development Core training 4 0.76 
Professional development Health and safety 5 0.85 
Professional development Teaching techniques 3 0.64 
Assessment Checklists 3 0.88 
Assessment Organization-developed 3 0.93 
Assessment Curriculum-based 3 0.91 
Assessment Handheld / PDA 3 0.74 
Assessment Standardized tests 3 0.87 
Assessment Work sampling 3 0.84 
 

Classroom pictures. Three items from the photograph-based Preschool Environment Checklist 
were dropped because of low inter-rater reliabilities (classroom labels, presence of educational 
posters/wall materials, and organized classroom theme). Two additional items were dropped 
because they failed to produce unique loadings in excess of .30 on any factor (children’s work 
displayed and samples of modeled writing). Kappa statistics for the remaining 10 items ranged from 
.56 to .94. Exploratory factor analysis yielded three correlated factors with eigenvalues greater than 
1.0. However, the scree plot was consistent with a 2-factor solution, the 3-factor solution included a 
one-item factor, and the two-factor solution yielded a reasonable fit (RMSEA=.06). Therefore, two 
factors were retained. 

The six items that comprised Factor 1 inquired about the presence of labels, children’s 
names, and other visual supports used for group instruction or classroom management (e.g., 
calendar, weather chart, daily news, letter wall, helper chart, daily schedule). Therefore, Factor 1 was 
labeled Visual Supports. The four items that formed Factor 2 asked about the quality of centers and 
how well supplied these were with literacy materials (e.g., books, writing supplies). Factor 2 was 
therefore labeled Center Quality. The two factors were correlated at .45.  

Facility-report - Professional development. The Facility Report of e-School Plus included fourteen 
items that inquired about the kinds of professional development provided at a given facility. One of 
these items was dropped because it failed to produce unique loadings in excess of .30 on any factor. 
Exploratory factor analysis of the 13 remaining items yielded three correlated factors with 
eigenvalues over 1.0. This three-factor solution was the most interpretable among alternative 
models, and it fit the data well (RMSEA = .06).  



Educational Policy Analysis Archives Vol. 20 No. 17  13 

 

The first factor included five items that inquired about professional development that was 
focused on teaching of core subjects (e.g., language, math). Factor 1, therefore, was labeled 
Professional Development in Core Subjects. The second factor was comprised of three items that 
inquired about professional development that was focused on teaching methods/techniques (e.g., 
classroom management) and was labeled Teaching Techniques. Finally, the five remaining items 
(e.g., CPR training, nutrition, health & safety) formed the third factor, labeled Health & Safety 
Training. Inter-factor correlations ranged from .49 to .59. 

Facility-report - Assessment. The Facility Report of e-School Plus also included 18 items that 
inquired about the frequency that organizations administered different types of assessments to assess 
children’s language, literacy, mathematics, and social/emotional/behavioral competence. 
Exploratory factor analysis of all 18 items yielded six factors with eigenvalues greater than 1.0. Each 
factor was comprised of three items. The six factor structure mapped perfectly onto the six 
assessment methods. Thus, there were separate factors for Standardized Tests, Progress Monitoring 
with PDA, Work Sampling, Checklists, Curriculum-Based Measures, and Organization Developed 
Measures. Substantively, these results suggest that organizations tended to use the same assessment 
method across domains of school readiness and across the age range of preschool children. 
Correlations among the six assessment factors ranged from .04 to .58 (median = .34). Although the 
six-factor solution was readily interpretable, residual variances were substantial, as indicated by a 
relatively high RMSEA (.16). This is likely due to the very high correlations among assessment 
methods increasing the power of the analysis. However, the Root Mean Square Residual was 
reasonable (.04).  

Correlation of factor scores with kindergarten outcomes. Because kindergarten school readiness was 
considered a primary outcome of interest that should inform the SRCS, the final step in reducing the 
number of variables for the latent profile analysis was to check the zero-order correlations of the 
school readiness outcomes with all the factor scores generated above and any remaining variables 
from e-School Plus that were not part of the data reduction thus far (e.g., average number of books 
in the classroom). Any preschool variable that was not correlated with at least one of the 
kindergarten school readiness outcomes was excluded from subsequent analyses. School readiness 
outcomes were based on children’s social screener scores and their classification status obtained on 
the literacy screeners. Children’s school readiness outcomes were aggregated to the level of 
preschool classroom so that they could be correlated with classroom level indicators of preschool 
quality. Thus, kindergarten variables were the percent of children in a given pre-kindergarten 
classroom who scored “Developed” on literacy screeners and the average social screener score 
among children from a given pre-kindergarten classroom. Three indicators of preschool quality were 
significantly correlated with at least one indicator of school readiness: Intentional Instruction from 
the Teacher Self-Report, Early Literacy Instruction from the Teacher Self-Report, and Professional 
Development in Core Subjects from the Facility Report.  

Identification of Latent Profiles of Classrooms 

Our ultimate goal was to determine criteria for certifying or not certifying classrooms that 
had entered the SRCS. In other words, based on the results of the analyses, each classroom would 
have to be assigned to either a certified group or a not-certified group. Therefore, we decided that a 
clustering approach, in which classrooms were grouped into different categories based on the 
pattern of their responses, was most appropriate. Specifically, latent profile analysis (LPA) was used. 
Similar to factor analysis, LPA is a latent variable approach in which the relations among observed 
variables are explained in terms of an unobserved (latent) variable. Unlike factor analysis, the latent 
variable in LPA is discrete, rather than continuous, and the focus of the analysis is the classification 



Identifying Quality Pre-Kindergarten Programs  14 

 

 

of cases (in this case, classrooms) into latent profiles based on their pattern of means on the 
dependent (endogenous) variables (Magidson & Vermunt, 2004).  
 
Table 2 
Summary of Factors from the Teacher Self-Report Survey 
Factor Type of items Number of Items Factor label 
1 Strategies used to promote oral language 

development during shared book reading. 
4 Shared Reading 

2 Materials used to teach mathematics concepts. 7 Math Concepts 
3 Frequency and kinds of child assessments                 

administered and how they are used. 
3 Assessment 

4 Intentional teaching strategies that crossed                 
content areas. 

9 Intentional 
Instruction 

5 Frequency that teachers employed various 
strategies that promote early literacy (e.g. 
writing and print awareness. 

8 Early Literacy 
Instruction 

6 Relative importance of teacher directed 
instruction/lesson planning versus child 
initiated play/following children’s interests; 
frequency that teachers used their own ideas, 
district standards, agency guidelines, and state 
standards to guide their lesson planning. 

6 Lesson Planning 

 
The three classroom-level factor scores and the two kindergarten outcome variables, which 

were aggregated to classrooms, were entered into the latent profile analysis. Of the original 1,225 
classrooms in the system, 1,185 classrooms had data on at least one of these factors or kindergarten 
outcome variables, and 447 were missing at least one data point. The most common pattern of 
missing data (69%) was missing both the facility report and kindergarten outcome data. As long as 
the data are missing at random (MAR), maximum likelihood estimation techniques using full 
information will produce unbiased estimates (Baraldi & Enders, 2010; Little & Rubin, 1987). For the 
kindergarten outcome data, the state of Texas allows a variety of measures to be used for the 
kindergarten reading screen (e.g., DIBELS) so data were missing because we only collected data for 
the two most common assessments (i.e., TPRI and TejasLee). For the facility report data, it is 
possible that the pattern of missingness is related to the school’s training practices (i.e., not missing 
at random, MNAR). However, because those same schools completed the rest of the application 
and required their teachers to complete the teacher self-report, we considered it unlikely. Therefore, 
in order to retain the variability in the teacher self-report, we decided to include the observations 
with missing data in our final analyses. However, to be eligible for certification, complete data was 
required. Of the 1,185 classrooms that had at least partial data, 738 had complete data and were 
eligible for certification.  

In order to avoid convergence to local maxima, the model was run with 100 random initial 
starts, and parameters were estimated using maximum likelihood with robust standard errors due to 
some mild to moderate non-normality among the five indicators. Alternative models were compared 
using two criteria: the BIC (Schwarz, 1978), which is a measure of absolute model fit, and the Lo-
Mendell-Rubin (LMR) adjusted likelihood ratio test, which tests the hypothesis of k vs. k-1 latent 
profiles (Lo, Mendell, & Rubin, 2001). A significant LMR test indicates that the current model, with 
k latent profiles, fits the data better than the more parsimonious model with one less latent profile 
estimated.  



Educational Policy Analysis Archives Vol. 20 No. 17  15 

 

Two-class model results. The first model examined was a two-class model, analogous to a 
straight pass-fail criterion. For this model, the LMR test was used to assess the necessity of modeling 
two latent profiles versus one latent profile. A non-significant LMR test in this case would indicate 
that there was no variation in the pattern of means among the five indicators. However, the LMR 
test was highly significant, LMR = 248.1, p < .0001 (see Table 3), and examination of the model-
estimated means demonstrated that the profiles of the two classes were not parallel, confirming the 
necessity of a classified structure. Standardized estimated means for the two classes are plotted in 
Figure 1. Although the profiles for the preschool variables are roughly parallel in the two classes, the 
mean differences on these variables are accompanied by substantially different average outcomes in 
kindergarten. The first class, with above-average scores on both the preschool and kindergarten 
indicators, composed 78% of the sample, whereas 22% of the sample demonstrated below-average 
scores on the preschool indicators and much lower scores on the kindergarten indicators, especially 
the literacy outcome. Although no strict criterion was set a priori for the number or percentage of 
classrooms that would pass or fail certification, the 78% of classrooms that fell into the higher-
performing class was considered an acceptable pass rate.  
 
Table 3 
Model information and fit statistics for Latent Profile Analyses. 
   Conditional probabilities of class membership 

Model 
 

BIC LMR 
Class 
1 

Class 
2 

Class 
3 

Class 4 

2-class  11546 248.1** 75% 25% --- --- 
3-class  11523 144.0** 61% 21% 18% --- 
4-class  11486 77.5* 60% 17% 13% 10% 

Note: BIC=Bayesian information criterion,  
LMR =Lo-Mendell-Rubin adjusted likelihood ratio test. 
* p < .05, ** p < .001 
 

 Three class model results. To ensure that we had adequately captured the complexity of the data, 
we also explored a three-class model. The LMR for the three class model was highly significant, 
LMR = 144.0, p < .0001, indicating that the three-class model fit better than the two-class model, 
and comparison of the BICs was consistent with this assessment, BICthree-class = 11523; BICtwo-class = 
11546 (see Table 3). Inspection of the profiles of the three classes showed that the pattern found in 
the two-class model was almost exactly replicated by two latent profiles in the three-class model (see 
Figure 2). However, the three-class model included a new latent profile with a very different pattern. 
Specifically, whereas means on the kindergarten indicators in this third class resembled those of the 
high-performing class, means on two of the preschool quality indicators were substantially lower 
than those in the low-performing class. This new, mixed-performing group comprised 11% of the 
classrooms and was drawn primarily from the high-performing group in the two-class analysis, 
whose proportion dropped from 78% of classrooms to 67% of classrooms in the three class 
solution. The percentage of classrooms in the low-performing group dropped from 22% to 21%. 
Table 4 shows classification status and average kindergarten outcomes for the 3-class model. 
 



Identifying Quality Pre-Kindergarten Programs  16 

 

 

Table  4 
Latent Profile Classification and Average Kindergarten Outcome Scores (in original units) from 3-class model. 
Class n Classroom Reading Category 

Percent "Developed" 
Classroom  
Social Screener 

High 497 (67.3%) 80% 4.2 
Mixed 84  (11.4%) 77% 3.9 
Low 157 (21.3%) 25% 3.5 

 
Four class model results. The final model explored was a four-class model. This model fit the 

data better than the three-class model, BIC = 11486, LMR = 77.5, p = .034 (see Table 3). The 
differences between this model and the three-class model centered around distinctions among the 
newly-created mixed-performance group. In other words, the percentages of classrooms assigned to 
the high- and low-performing groups did not change with the addition of a fourth class, and the 
mixed-performance group was about evenly divided into two new groups in the four class solution. 
However, since this finer-grained distinction did not add to our ability to dichotomize classrooms 
into certified or not certified groups, we opted for the three-class model in the interest of parsimony. 

Issues regarding mixed-performing classrooms certification. Because certification by definition requires 
a pass-fail criterion or criteria, we had to decide whether or not to certify the mixed-performing 
group. On the one hand, the kindergarten performances of the children in these preschool 
classrooms resembled those of the high-performing group. In other words, the children from these 
preschool classrooms were school ready based on their kindergarten literacy scores and kindergarten 
social scores. On the other hand, these preschool classrooms did not possess the characteristics that 
are generally associated with positive kindergarten outcomes. Given the common goal of the Texas 
Education Agency and the State Center for Early Childhood Development to promote both school 
readiness and high quality early childhood education, only those classrooms empirically assigned to 
the group with high scores on all indicators were ultimately certified as “School Ready”. Thus, in this 
first year of the School Readiness Certification System, school ready certification was awarded to 497 
preschool classrooms across Texas, which represented 67% of classrooms eligible for consideration 
for certification (i.e., with complete data). 

Validity of the three class model results. In order to examine the validity of the three-class model, 
several follow-up analyses were performed. Common practice dictates that follow-up analyses aimed 
at identifying differences among latent classifications be performed in an effort  to validate the 
model selected. First, ANOVA was used to compare the three groups’ means on the indicators of 
the latent profiles. Because this involved three simple contrasts for each of the five indicators, a 
Bonferroni correction was made to the alpha, p = .05 / 3, so that p = .0167, was considered 
significant. With this conservative alpha, twelve of the fifteen contrasts were still statistically 
significant, t’s = 2.99-34.83, p’s < .01, indicating that the three latent profiles indeed reliably differed 
on almost every indicator. However, the mixed performance group’s mean on the Professional 
Development in Core Subjects indicator did not differ significantly from either the low-performing 
group, t = 1.90, p = .057, or the high-performing group, t = 2.02, p = .044. The low- and high-
performing groups did differ significantly on Professional Development in Core Subjects, t = 5.36, p 
< .0001. Additionally, as suspected, the means on the literacy outcome did not differ between the 
mixed-performing group and the high-performing group, t = 1.90, p = .094. We also compared the 
average attendance of the three groups. The low-performing group averaged significantly fewer days 
than both the high- or mixed-performing groups (147 vs. 156 and 154, respectively, ps < .05). 

Finally, we extended our examination of the validity of the classification scheme to include 
comparing classrooms that were ultimately certified (i.e., the high-performing group) to those 
classrooms that were ultimately not certified (i.e., the mixed- and low-performing groups).  



Educational Policy Analysis Archives Vol. 20 No. 17  17 

 

Classrooms certified as “School Ready” had 55% more classmates score “developed” on the state 
mandated kindergarten reading screeners than classrooms that were not awarded certification, t = 
18.92, p < .001. Similarly, children who attended “School Ready” classrooms had significantly higher 
scores on the measure of social adjustment, t = 9.82 , p < .001. Finally, children in “School Ready” 
certified classrooms had higher attendance than children in non-certified classrooms, t = 3.24, p < 
.01. 

Follow-up Analyses Addressing the Composition of the Latent Profiles 

Because the goal of the certification system was to identify characteristics of programs that 
promoted school readiness, independent of the type of service delivery model (i.e., public school 
pre-kindergarten, Head Start, childcare), service delivery model and variables confounded with 
service delivery model were explicitly precluded from the analysis used to inform the certification 
criteria. That is, systematically making certification more or less difficult to obtain for particular 
types of programs would undermine the objective of a statewide certification system. Nonetheless, it 
was of interest to the Texas Education Agency to discover how likely classrooms from different 
service delivery models were to become certified. In addition, the Texas Education Agency was 
interested in examining the rate of certification of classrooms in the state’s integration model. 
Therefore, two sets of follow-up analyses were conducted that examined the relation of latent 
classification with facility type and the relation of latent classification with whether or not a program 
was enrolled in the Texas Early Education Model (TEEM). Finally, two analyses were conducted in 
order to examine the distribution of at-risk children into certified classrooms.  

Association between certification and service delivery system. For analyses that examined the extent to 
which latent classification was associated with type of preschool facility (i.e., public school based 
pre-kindergarten, Head Start, or childcare; see Table 5), a 3 x 3 contingency table analysis was 
conducted and found significant, χ2 (4, n = 684) = 113.7, p < .001.  
 
Table 5 
Frequency (%) of Latent Profiles by Facility Type. 
 Latent profile  
Facility type High Low Mixed 
Childcare 87 (52%) 30 (18%) 50 (30%) 
Head Start 111 (59%) 64 (34%) 13 ( 7%) 
Public School 273 (83%) 40 (12%) 16 ( 5%) 
 

Examination of cell frequencies revealed that public school pre-kindergarten classrooms 
were overrepresented in the high-performing group and were underrepresented in the low-
performing and mixed-performing groups. Head Start classrooms were overrepresented in the low-
performing group. Finally, childcare classrooms were overrepresented in the mixed-performing 
group and were underrepresented in the high-performing group.  

Because service delivery system only varied among classrooms that were enrolled in the state 
integration program, a restricted analysis was performed in which we examined the relation of 
facility type and latent classification only among classrooms enrolled in the state integration 
program. The results of this restricted analysis were largely the same as those performed with the 
entire sample of classrooms. The only exception was that when childcare classrooms that served 
higher income children (i.e. classrooms that were not enrolled in the state integration program) were 
removed from the analysis, childcare classrooms were no longer underrepresented in the high-
performance group.  



Identifying Quality Pre-Kindergarten Programs  18 

 

 

Association between certification and participation in state integration model. Two sets of follow-up 
analyses were conducted to examine the extent to which latent classification was associated with 
enrollment in the state integration program. First, a 3 (Latent profile: High, Low, Mixed) x 2 (State 
program model: Participating or Non-participating) contingency table analysis was conducted on the 
full sample. However, in light of the fact that all of the classrooms in our sample that did not 
participate in the state integration model were classrooms in childcare centers, we conducted a 
second, restricted analysis on a sample that only included childcare settings.  

For the full analysis, there was a significant association between classification into the three 
types of latent classrooms and enrollment status in the state integration program, χ2 (2, n = 731) = 
80.2, p < .001 (see Table 6). Specifically, TEEM classrooms were slightly underrepresented in the 
mixed-performing group. The opposite was true of classrooms that did not participate in TEEM, 
which were overrepresented in the mixed group and were underrepresented in both the high- and 
low-performing groups.  
 
Table 6 
Frequency (%) of Latent Profiles by Program Status. 

 Latent profile 

Program Status High Low Mixed 

Not State Program 
21 (42%) 4 ( 8%) 25 (50%) 

State Program 472 (69%) 151 (22%) 58 ( 9%) 
 

For the restricted analyses, the only significant deviation from expected cell frequencies was 
an overrepresentation of classrooms that were not enrolled in TEEM in the mixed-performing 
group, χ2 (2, n = 167) = 13.8, p < .01. These results indicate that for-profit child care programs that 
served middle income families were more likely than childcare programs enrolled in TEEM to 
evidence poor quality preschool programming but still graduate children who were ready for 
kindergarten in terms of academic and social preparedness.   

Association between certification and at-risk populations. We also examined the association between 
classroom certification and children’s free/reduced lunch status, an indicator of economic 
disadvantage. No differences were found in the distribution of disadvantaged children into certified 
classrooms, χ2 (1, n = 10,137) = .86, ns (see Table 7), indicating that economically disadvantaged 
children were equally likely to be in a certified classroom as their more affluent peers. The second 
analysis compared certification status between native English speakers and English-language 
learners. Results indicate that English-language learners were more likely than expected to be in a 
not-certified classroom, whereas native English speakers were more likely than expected to be in a 
certified classroom, χ2 (1, n = 5,376) = 75.0, p < .001.  
 
Table  7 
Percentage of free/reduced lunch and English language learners by certification status. 
Class n Free or reduced lunch  

percent 
English language learner 
percent 

Certified 7,435 (73.4%) 74.3% 44.3% 
Not certified 2,702 (26.6%) 75.2% 59.4% 

 



Educational Policy Analysis Archives Vol. 20 No. 17  19 

 

Discussion 

This project was initiated by a state mandate to develop a system that would inform parents 
about which early childhood programs were preparing preschool-age children for school. Key 
stakeholders agreed that the resultant quality rating system should take into account both 
scientifically-based indicators of quality pre-kindergarten programs and indicators of children’s 
school readiness. To our knowledge, this is a unique approach that has not been implemented by 
any other state or national quality rating systems, as most quality rating systems only consider 
characteristics of the pre-kindergarten program and classroom environment without linking these to 
kindergarten outcomes. A goal of the system was to ensure applicability of the criteria to the full 
range of early childhood service delivery types and thus, characteristics that were unique to a specific 
type of program were not included in the certification criteria.  

Advantages and disadvantages of the certification system 

Working within the fiscal constraints of Texas, this approach was only possible through 
capitalizing on the situation that the public schools already performed kindergarten screenings to 
identify children potentially at risk for reading problems. Constraints on development of the 
certification system included: (a) a need for cost-effectiveness, (b) reliability and validity of self-
report data, (c) inclusion of indicators of quality of pre-kindergarten program and indicators of 
children’s school readiness, and (d) development of an unbiased system that could be used by all 
early childhood education service delivery systems. Thus, the School Readiness Certification System 
(SRCS) strives for a balance between good science and the financial and political realities faced by a 
new state-wide program. 
 Advantages of the SRCS are that it is a web-based application that allows input from the 
teachers, program directors, and/or principal. While input on a web-based system is cost-effective, a 
challenge is assuring that the information collected is valid in that it reflects actual classroom 
practices and the classroom environment. In a pilot study, reliability and validity of teacher-self-
report data was corroborated through observations of classrooms by external evaluators. Whereas 
annual on-site visits to all programs is cost prohibitive in a large state with thousands of early 
childhood education programs, randomly selected observations by external evaluators to verify 
validity of data in applications is feasible for future years. Additional support for the validity of the 
data entered into the web-based application comes from the fact that these data were related to 
children’s school readiness in ways expected from prior research. 

Determination of the preschool factors. Decisions about items to include in the web-based 
application were based on indicators of quality obtained from research studies that include 
classroom environment, instructional practices in social and cognitive skill domains, curriculum, 
assessment approaches used to monitor children’s progress and professional development (e.g., 
National Research Council, 2001; Whitehurst et al., 1994). With this broad range of information in 
combination with children’s outcomes in early kindergarten, it was possible to determine the most 
important pre-kindergarten factors to emphasize in a quality rating system. The results show that it 
was possible to refine the information such that a set of unique indicators from the pre-kindergarten 
period could be identified as most important in terms of relations with reading and social 
competence in early kindergarten. However, it is recognized that the development of this system 
does not allow for causal inferences as there were too many unobserved intervening variables that 
cannot be ruled out.  

Results of the latent profile analyses showed that three constructs in the pre-kindergarten 
environment provided the best information for discriminating classrooms into distinct groups. 
These indicators were: (a) teacher professional development, particularly core training in school-



Identifying Quality Pre-Kindergarten Programs  20 

 

 

readiness (e.g., rich language input, early literacy, social competence); (b) teacher intentional 
instructional approaches that promote achievement; and (c) literacy oriented instructional activities 
especially early writing activities. Together these aspects of the pre-kindergarten environment 
discriminated, or predicted, those classrooms whose children had higher vs. lower early kindergarten 
social and reading outcomes. What is particularly interesting is that kindergarten social and reading 
scores and the three pre-kindergarten quality indicators were positively correlated. These findings 
have implications for efforts to understand how intentional cognitive activities during the pre-
kindergarten years may support a range of school readiness skills including social competence 
(Duncan, Dowsett, Claessens, Magnuson, Huston, Klebanov, et al., 2007; Reynolds, Mavrogenes, 
Bezruczko, & Hagemann, 1996). They suggest that classrooms that provide planful, cognitive 
activities such as early literacy carried out in intentional ways may also promote children’s social 
competence. While the indicators have a focus on building learning in language and literacy areas, 
the professional development and intentional instruction areas included responsive interaction styles 
between teacher and children as well as support for children’s efforts. This included rich language 
input such as the use of rare vocabulary words, providing explanations and expanding children’s 
utterances. Taken together, these indicators suggest that classroom teachers can incorporate 
practices that promote both social and cognitive competence and that the two areas of development 
do not involve a tradeoff. 

Validity of the certification system. Another key finding was that children from poverty 
backgrounds who were in well implemented integration classrooms got similar reading and social 
pass rates to children from higher income backgrounds in terms of their early kindergarten screening 
scores. Moreover, they demonstrated better social competence, on average, including the ability to 
regulate their behavior in social situations. One explanation for improved social regulation for these 
children could be the emphasis on TEEM teachers providing predictable routines and engaging 
learning activities. This may provide children with the support they need to learn to more 
independently organize their behavior in activities that require cooperating with others and taking 
initiative. The formula for certification provide support that children living in poverty, whether 
enrolled in child care, Head Start or public school classrooms can enter kindergarten with the 
reading and social skills necessary to learn what is required in kindergarten. This finding is consistent 
with a conclusion reached regarding twins raised in different environments, where 60% of the 
variance in cognitive skill was related to environment with genetic influence being essentially zero 
(Turkheimer, Haley, Waldron, D’Onofrio, & Gottesman, 2003). These results counter the arguments 
that children from poverty backgrounds cannot reach similar levels of achievement with that of 
more affluent peers when given the right environment for learning. It is encouraging that exposure 
to high quality classroom-based program that emphasize cognitive readiness (e.g., language, literacy 
building) in ways that supported social competence may be able to place children from low SES 
backgrounds on an equal footing with more economically advantaged children. However, it is also 
possible that the classrooms in the mixed class that mostly served middle-income children were not 
supporting these higher SES children’s learning adequately. 

It is important to note that classrooms from each of the service delivery programs were 
classified as high quality, albeit classrooms from public school programs showed the highest rate of 
certification. This might be expected as public school requires a college degree though the results 
show that this alone did not assure 100% certification status. The results of the certification of the 
Head Start classrooms was somewhat lower than that of public school classrooms, and this may be 
due, in part, to the diversity of performance standards, or goals to which such programs attend as 
well as a lower income criteria that often means children in Head Start programs are from 
particularly disadvantaged backgrounds. This said, subsidized childcare and Head Start classrooms 
showed greater than 50% certification rates in spite of serving very low income children. 



Educational Policy Analysis Archives Vol. 20 No. 17  21 

 

Secondary benefits of the certification system. While the goal of the School Readiness Certification 
System was to identify classrooms that prepare children for entering kindergarten, it also has the 
potential to serve as a guided self-assessment. Technical assistance for programs that did not meet 
the certification criteria was available and considered a key component of the system. Thus, it serves 
a supportive rather than a punitive role. For those not certified in the present study, a profile of 
strengths and weaknesses was developed from the data base and after each program received this 
report, they were offered the opportunity to discuss a technical support plan with the State Center 
for Education and Child Development, who developed the SRCS. Specifically, state-funded early 
childhood support staff in each community partnership are trained to interpret this profile and to 
prioritize a sequence of steps for supporting the classroom to improve weak areas. The advantage of 
the technical assistance is that the support is tailored to the needs of individual programs so that 
resources have a high likelihood of being used effectively. This better assures that when a program 
reapplies, it is more likely to achieve certification.  
 Given the voluntary nature of the certification system, a question arises as to what the 
incentive is for programs to participate? For school districts, a strong advantage of a comprehensive 
quality rating system is that it has the potential through market forces to encourage more children to 
arrive at kindergarten ready to succeed in public school. This would allow for better use of resources 
in kindergarten and beyond, since fewer children would require referral to special education or need 
remedial services. Also, less of the initial part of the kindergarten year would be spent by regular 
education teachers getting lower achieving children up to speed with the rest of the class. For 
stakeholders, such as directors of large child care agencies or Head Start agencies, feedback provided 
through a comprehensive quality rating system would allow for an increased awareness of which 
individual programs were more or less effective. This information can serve functions of 
accountability as well as guide program development. For individual centers and programs, receipt 
of a high rating or certification at minimal investment cost would allow positive advertisement 
within a community of program effectiveness. For example, private for-profit childcare programs 
need to remain competitive for continued funding. One means to demonstrate a competitive edge 
would be to market oneself as a program that has been independently determined to promote school 
readiness.  

Limitations and future directions 

Practical considerations that need to be addressed through ongoing research with this system 
include the issue of how to classify programs with multiple classrooms where some reach 
certification criteria and others do not. Ideally, all classrooms at a given school would be assessed 
and certification would be awarded to schools rather than individual classrooms. Within the current 
context of classroom-level certification, a possible solution would be to certify a program as Texas 
School Ready if a majority of classrooms reach certified status. This could be combined with the 
criteria that the minority of non-certified classrooms within a program reach certification within a 
specified period of time. An important policy implication includes the need for states to find 
sustainable funding for such a system so that no program is excluded due to lack of financial 
resources.  

There are some important limitations to the current study that are noteworthy. Data from a 
group of classrooms could not be included in analyses that informed the certification criteria 
because these classrooms had incomplete applications. Additionally, it is important to note that if a 
group of children scored developed in reading and social skills in kindergarten, but the preschool 
classroom that they attended did not have scientifically-based best practices in place, then those 
classrooms were not certified. This practice allowed the system to stay true to its purpose; 
identifying programs that have teaching practices in place that scientific research has demonstrated 



Identifying Quality Pre-Kindergarten Programs  22 

 

 

promotes learning. To only base certification on kindergarten outcomes could introduce a SES bias 
into the certification process. That is, a large body of research shows that low-income children often 
score below middle income children on measures of academic achievement (e.g., Zill & West, 2001). 
This occurs, in part, because higher family income is strongly related to higher levels of parent 
education, a predictor of children’s academic achievement. Also, had we certified solely based on 
kindergarten outcomes then we would have certified classrooms in the mixed-group that did not 
provide children with the learning experiences that would have allowed their children to reach their 
optimal potential. That is, as many children in the mixed group were from middle class backgrounds, 
they might be expected to achieve even higher in kindergarten if they were provided with an optimal 
learning environment. In sum, we believe that the best approach to rating early childhood education 
classrooms or programs is one that takes into account both quality of preschool programming and 
children’s school readiness outcomes. 
 An important aspect of this system is that the items in the Teacher Self-Report represent 
constructs that teachers can report in a reliable manner. Contrary to expectations, two areas that did 
not show good reliability between observation and teacher self-report was responsiveness to 
children’s needs and behavioral management. While these are considered quite important in 
descriptions of best practices, the ability to reliably capture this information could not be 
demonstrated with this system. Given that major goals of the system were that criteria had to be 
captured reliably and empirical methods needed to be used rather than beliefs regarding best 
practices, it was necessary to drop these items. Future checks on the sensitivity of the system will 
allow for consideration of additional items and measurement methods.  

Given the emphasis on self-report from program administrators and teachers, the desire of a 
school to be certified could result in a “high stakes” climate that would promote responses based on 
presenting the classroom in the best possible light. There also was concern that incorporating end-
of-the-year testing in pre-kindergarten would be experienced by teachers as a “high stakes” type of 
testing, which may lead teachers to be overly concerned with the end-of-the-year assessment and 
they may “teach to the test” rather than attend to the whole child. This contributed to the decision 
to use the state kindergarten screening mechanism already in place rather than testing at the end of 
the pre-kindergarten year. The same concern may be applicable to specific self-reported teaching 
behaviors where, to ensure reliable reporting, the system may need to identify additional item sets 
that can be used in a random rotation approach to avoid response bias. Finally, the state’s ability to 
promote participation of pre-kindergarten programs in the SRCS will be dependent upon an 
emphasis of the usefulness of the system for program evaluation and improvement and avoidance of 
using the system in a punitive manner.  

Finally, at present the SRCS system lacks a more comprehensive set of school readiness 
outcomes. The combination of a limited set of school readiness outcomes and a data reduction 
process that excluded pre-kindergarten classroom characteristics that were unrelated to kindergarten 
outcomes may have oversimplified our characterizations of pre-kindergarten classrooms. For 
example, the Math Instruction factor was not used to help classify classrooms because it was 
unrelated to kindergarten outcomes, which was not surprising given the absence of a mathematics 
outcome. As early mathematics skills are receiving a great deal of attention from researchers and 
educators, the inclusion of a math screener as a kindergarten outcome in the system will be an 
important addition in the future. On the other hand, an asset to the SRCS is the inclusion of both an 
academic outcome (i.e., reading readiness) and a measure of social competence in its 
conceptualization of school readiness. Staying true to this whole-child conceptualization of school 
readiness was no small feat, as the social screener was not a state mandated kindergarten screener. 
Instead, collecting these data required enormous efforts and cooperation among school districts, 
principles, and teachers. While many districts were willing to have their kindergarten teachers 



Educational Policy Analysis Archives Vol. 20 No. 17  23 

 

complete the social competence measure, a more seamless approach would be to legislate a common 
social screener across the state.  

In summary, school readiness for young children is promoted through a classroom focus on 
literacy, language, and social skills through the implementation of a core curriculum, professional 
development for teachers specific to the area of school/cognitive readiness; and intentional high 
quality, school readiness instruction. The development of the SRCS demonstrates how an 
empirically based approach can result in an effective means for informing parents, state policy 
makers, and educators about the quality of early childhood programs. It will be important to 
remember that such a system is a dynamic process that will require ongoing attention and revisions 
that are responsive to new research findings, the continued sensitivity of the quality indicators, and 
expansion of early kindergarten outcomes (e.g., math). To assure knowledge about whether 
programs can sustain quality over time, the system will require re-certification on an annual or bi-
annual basis. 

 References  

Baraldi, A. N., & Enders, C. K. (2010). An introduction to modern missing data analyses. Journal of 
School Psychology, 48, 5-37. 

Bierman, K.L., Domitrovich, C.E., Nix, R.L., Gest, S.D., Welsh, J.A., Greenberg, M.T., et al.  
(2008). Promoting academic and social-emotional school readiness: The Head Start REDI 
program. Child Development, 79, 1802-1817. 

Campbell, F.A., Ramey, C., Pungello, E.P., Sparling, J.J., & Miller-Johnson, S. (2002).  
Early childhood education: Young adult outcomes from the Abecedarian project. Applied 
Developmental Science, 6, 42-57. 

Center for the Child Care Workforce. (2004). Current data on the salaries and benefits of the U.S. 
early childhood education workforce. Retrieved July 19, 2007, from 
http://www.ccw.org/pubs/2004Compendium.pdf 

CIRCLE (2001).  Preschool Classroom Environment Checklist.  Unpublished research instrument.  
Crosser, S. (1992). Managing the early childhood classroom. Young Children, 47, 23-29. 
DiPietro, J. A. (2000). Baby and the brain: Advances in child development. Annual  
  Review Public Health, 21, 455-471. 
Dumas, J. E., Martinez, A., & LaFreniere, P. J. (1998). The Spanish version of the 

Social Competence and Behavior Evaluation (SCBE) - Preschool edition: 
Translation and field testing. Hispanic Journal of Behavioral Sciences, 20, 255- 
269. 

Duncan, G.J., Dowsett, C.J., Claessens, A., Magnuson, K., Huston, A.C., Klebanov, P., et  
al. (2007). School readiness and later achievement. Development Psychology, 43,  
1428-1446 

Dunn, L., Beach, S.A., & Kontos, S. (1994). Quality of the literacy environment in day care and  
children’s development. Journal of Research in Childhood Education, 9, 24-34. 

Elmore, R.R. (2002). Bridging the gap between standards and achievement. The imperative for  
professional development in education. The Albert Shanker Institute. 

Evans, G.W. (2004). The environment of childhood poverty. American Psychologist, 59, 77-92. 
Gallagher, J.J., Clifford, R.M., & Maxwell, K. (2004). Getting from here to there: To an ideal  

early preschool system. Early Childhood Research & Practice, 6. Retrieved March 16, 2006 from 
www.ecrp.uiuc.edu. 

Hart, B. & Risley, T.R. (1995).  Meaningful differences in the everyday experiences of young American children.  
Baltimore: Paul H. Brookes Pub. Co. 

Hindson, B., Byrne, B., Fielding-Barnsley, R., Newman, C., Hine, D.W. & Shankweiler, D.  



Identifying Quality Pre-Kindergarten Programs  24 

 

 

(2005). Assessment and early instruction of preschool children at risk for reading disability. 
Journal of Educational Psychology, 97, 687-704. 

Kagan, S.L., & Rigby, E. (2003). Policy matters: Improving the readiness of children for school.  
Recommendations for state policy. Washington, DC: Center for the Study of Social Policy. 

Kaurtz, K. (2008). Learning from others: State efforts to expand services and build systems of  
early care and education. In A.R. Tarlov & M.C. Debbink (Eds.), Investing in early childhood 
development (pp. 85-112). NY: Palgrave Macmillan. 

Kotler, J. C. & McMahon, R. J. (2002). Differentiating anxious, aggressive and 
socially competent preschool children: Validation of the Social Competence and 
Behavior Evaluation -30 (parent version). Behavior Research and Therapy, 40, 
947-959. 

LaFreniere, P. J. & Dumas, J. E. (1996). Social competence and behavior evaluation in children  
ages 3 to 6 years: The short form (SCBE-30). Psychological Assessment, 4, 442-450. 

LaFreniere, P. J., Masataka, N., Butovskaya, M., Chen, Q., Dessen, M. A., Atwanger, K. et al.  
(2002). Cross-cultural analysis of social competence and behavior problems in preschoolers. 
Early Education and Development, 13, 201-219. 

Landry, S.H., Crawford, A., Gunnewig, S., & Swank, P.R. (2002). The CIRCLE-Teacher Behavior Rating 
Scale. Unpublished research instrument. 

Landry, S.H., Swank, P.R., Anthony, J.L, & Assel, M.A. (2011).  An experimental study evaluating 
professional development activities within a state funded pre-kindergarten program. Reading  
and Writing 24, 971-1010. 

Landry, S.H., Swank, P.R., Smith, K.E., Assel, M.A., & Gunnewig, S.B. (2006). Enhancing early 
literacy skills for preschool children: Bringing a professional development model to scale. 
Journal of Learning Disabilities, 39(4), 306-324. 

Landry, S.H., Smith, K.E., Swank, P.R., Assel, M.A., & Vellet, S. (2001). Does early responsive 
parenting have a special importance for children’s development or is consistency across early 
childhood necessary? Developmental Psychology, 37, 387-403. 

Little, R. J. A., & Rubin, D. B. (1987).  Statistical analysis with missing data. New York: J. Wiley & Sons. 
Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal 

mixture. Biometrika, 88(3), 767-778. 
Magidson, J., & Vermunt, J. K. (2004). Latent class models. In D. Kaplan (Ed.), The Sage handbook of 

quantitative methodology for the social sciences (pp. 175-198). Thousand Oakes: Sage Publications. 
Mitchell, F. (1979). Interobserver agreement, reliability, and generalizability data  

collected in observational studies. Psychological Bulletin, 86, 376-390. 
Muthén, B., du Toit, S. H. C., & Spisic, D. (in press). Robust inference using weighted least squares 

and quadratic estimating equations in latent variable modeling with categorical and 
continuous outcomes. Biometrika. 

National Association of Education of Young Children (2008). Promoting quality through  
accreditation. Retrieved on May, 2008 from http://naeyc.org/accreditation. 

National Association of Early Childcare Professionals (2008). National accreditation for early  
care and education programs. Retrieved on May, 2008 from http://www.naccp.org 

National Commission on Teaching and America's Future (1996). What matters most: Teaching  
  for America's future. New York: Teachers College, Columbia University. 
National Early Literacy Panel. (2008). Developing early literacy: Report of the National Early  
 Literacy Panel. Washington, DC: National Institute for Literacy. 
National Research Council (2001). Eager to learn: Educating our preschoolers. Washington,  
  DC: National Academy Press. 
Phaneuf, R., L., & Silberglitt, B. (2003). Tracking preschoolers' language and preliteracy  



Educational Policy Analysis Archives Vol. 20 No. 17  25 

 

   development using a general outcome measurement system: One education district's  
   experience. Topics in Early Childhood Special Education, 23, 114-123. 
Phillips, D. A., Voran, M., Kisker, E., Howes, C., & Whitebook, M. (1994). Child care for children in 

poverty: Opportunity of inequity. Child Development, 65, 472-492. 
Raver, C., & Knitzer, J. (2002). Ready to enter: What research tells policymakers about strategies to 

promote social and emotional school readiness among 3- and 4-year-old children. National 
Center for Children in Poverty, Mailmain School of Public Health, Columbia University. 

Reynolds, A.J., Mavrogenes, N.A., Bezruczko, N., & Hagemann, M. (1996). Cognitive and family-
support mediators of preschool effectiveness: A confirmatory analysis. Child Development, 
67, 1119-1140. 

Reynolds, A., Ou, S., & Topitzes, D. (2004). Paths of effects of early childhood  
 intervention on educational attainment and delinquency: A confirmatory analysis of  
  the Chicago Child-Parent Centers. Child Development, 75, 1299-1328.  
Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461-464. 
Snijders, T. A. B. & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel 

modeling. London: Sage Publishers. 
Texas Education Agency (1998). Texas Primary Reading Inventory, Technical Manual. Austin, TX: 

TEA.  
Texas Education Agengy (2002). Tejas LEE, Technical Manual. Austin, TX: TEA.  
Turkheimer, E., Haley, A., Waldron, M., D’Onofrio, B., & Gottesman, I.I. (2003).  

Socioeconomic status modifies heritability of IQ in young children. American Psychological 
Society, 14, 623-628. 

Whitehurst, G.J., Epstein, J.N., Angell, A.L., Payne, A.C., Crone, D.A., & Fischel, J.E. (1994).   
Outcomes of an emergent literacy intervention in Head Start. Journal of Educational Psychology, 
86, 542-555. 

Zellman, G. L., Brandon, R. N., Boller, K., & Kreader, J. L. (2011). Effective Evaluation of Quality 
Rating and Improvement Systems for Early Care and Education and School-Age Care, 
Research-to-Policy, Research-to-Practice Brief OPRE 2011-11a. Washington, DC: Office of 
Planning, Research and Evaluation, Administration for Children and Families, U.S. 
Department of Health and Human Services. 

Zill, N. & West, J. (2001). Entering kindergarten: A portrait of American children when they begin school. US 
Department of Education, OERI, NCES 2001-035. 



Identifying Quality Pre-Kindergarten Programs  26 

 

 

Appendix 

This section provides a description and photograph scoring examples for two classrooms. While 12 
photographs per classroom were considered when scoring the Preschool Environment Checklist, 
only 6 photographs that best represent key distinctions in scoring criteria were selected for the 
purposes of this illustration. Classroom 1 embodies many of the characteristics consistent with high 
scores, whereas Classroom 2 represents an environment receiving low scores.      
 

Classroom 1 
Central to the scoring system is the presence of several key visual supports, including a daily 

schedule, jobs chart, daily news, center management systems, rules chart, letter walls, and read aloud 
charts. With the exception of the read aloud chart, each of these elements is visible in Classroom 1. 
Once identified, coders consider the quality of these supports (e.g., at eye level, interactive, linked 
with learning objectives), which in this case fall into the high range. This classroom also receives 
high scores on a series of items connected with the richness of the learning environment, including 
access to a variety of books and writing materials, as well as a centers-based room arrangement that 
supports peer interaction and exploration in a variety of content areas. See Classroom 1 photographs 
below for specific examples.   
 


 




Educational Policy Analysis Archives Vol. 20 No. 17  27 

 









 
Classroom 1



Identifying Quality Pre-Kindergarten Programs  28 

 

 

 

Classroom 2 

This classroom is characterized by a lack of stimulating materials, print and interactive 
management systems. For example, several key visual supports captured with the measure are 
absent, including a daily schedule, rules chart, jobs chart, letter wall, and daily news. While evidence 
of a center management system can be seen (e.g., construction center label with space for children’s 
names), the management chart appears inaccessible in the dramatic play center.  In such cases the 
center management ratings would fall in the mid-range, allowing high scores to be reserved for those 
management systems that apply to several learning areas and appear to be used routinely. Classroom 
2 also receives low scores on the series of items reflecting the richness of the learning environment 
as evidenced by a lack of books, print, and variety in centers-based learning opportunities. See 
Classroom 2 photographs below for specific examples.   
 









Educational Policy Analysis Archives Vol. 20 No. 17  29 

 






Identifying Quality Pre-Kindergarten Programs  30 

 

 






Educational Policy Analysis Archives Vol. 20 No. 17  31 

 






 
Classroom 2 

 



Identifying Quality Pre-Kindergarten Programs  32 

 

 

About the Authors 

Jeffrey M. Williams 
University of Texas Health Science Center Houston 
 
Email: Jeffrey.Williams@uth.tmc.edu  
 
Dr. Jeffrey Williams is an assistant professor at the Children’s Learning Institute at the University 
of Texas Health Science Center in Houston.  He specializes in the application of multivariate 
statistical models to complex research questions, primarily in early childhood education. Specifically, 
he is interested in factor analysis and item response theory (IRT); structural equation modeling 
(SEM); and mediation and moderation, including their multilevel applications. 
 
Susan H. Landry 
University of Texas Health Science Center Houston 
 
Email: susan.landry@uth.tmc.edu 
 
Dr. Susan H. Landry is a Developmental Psychologist, the Albert & Margaret Alkek Chair in Early 
Childhood, and the Michael Matthew Knight Professor in the Department of Pediatrics at the 
University of Texas Health Science Center - Houston.  She is Director and Founder of the 
Children’s Learning Institute (CLI) at the University of Texas.  Funded by the National Institutes of 
Child Health and Development, The Institute for Education Sciences, U.S. Department of 
Education, Texas Education Agency, and Private Foundations, Dr. Landry has developed a 
significant research database. The research covers examinations of biological and environmental 
influences on children's development from infancy through adolescence as well as numerous 
federally funded parent-child and early childhood classroom intervention studies. This database and 
continued research makes use of new intervention approaches to provide parents and teachers with 
resources that are effective in supporting young children’s development.  
 
Jason L. Anthony 
University of Texas Health Science Center Houston 
 
Email: jason.l.anthony@uth.tmc.edu 
 
Dr. Jason Anthony is an Associate Professor at the Children’s Learning Institute and The 
University of Texas Medical School at Houston. He specializes in child psychopathology, 
developmental disabilities, interventions, assessment of children, learning disabilities, literacy 
acquisition, and family-and school-based interventions. 
 
Paul R. Swank 
University of Texas Health Science Center Houston 
 
Email: paul.r.swank@uth.tmc.edu 
 
Dr. Paul Swank is a full professor at the Children’s Learning Institute. He has 30 years of 
experience in teaching, mentoring and research. He is an expert in general linear mixed models, 
structural equation modeling, psychometrics and research design. He has published more than 



Educational Policy Analysis Archives Vol. 20 No. 17  33 

 

150 articles in the scientific literature. Dr. Swank’s recent work includes a collaborative study of 
traumatic brain injury in children and adults and the effects of stroke and epilepsy via brain 
imaging. 
 
April D. Crawford 
University of Texas Health Science Center Houston 
 
Email: april.crawford@uth.tmc.edu 
 
Dr. April Crawford is a program manager in the Children’s Learning Institute at the University of 
Texas Health Science Center Houston. Her interests include the development of observation and 
feedback systems to inform classroom practice and professional development, understanding how  
institutional settings shape children’s early learning experiences, and the extent to which programs 
aimed at quality improvement are differentially effective in a mixed delivery early care system. 

 

education policy analysis archives 
Volume 20  Number 17  June 20, 2012 ISSN 1068-2341 

 

 

Error! Reference source not found. Readers are free to copy, display, and distribute this article, 
as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is 
distributed for non-commercial purposes only, and no alteration or transformation is made in 
the work. More details of this Creative Commons license are available at 
http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the 
author(s) or EPAA. EPAA is published by the Mary Lou Fulton Institute and Graduate School 
of Education at Arizona State University Articles are indexed EBSCO Education Research 
Complete, DIALNET, Directory of Open Access Journals, ERIC, H.W. WILSON & Co, QUALIS 
– A 2 (CAPES, Brazil), SCOPUS, SOCOLAR-China.  

Please contribute commentaries at http://epaa.info/wordpress/ and send errata notes to 
Gustavo E. Fischman fischman@asu.edu  

 

 



Identifying Quality Pre-Kindergarten Programs  34 

 

 

education policy analysis archives 
editorial board  

Editor Gustavo E. Fischman (Arizona State University) 
Associate Editors: David R. Garcia & Jeanne M. Powers (Arizona State University) 

 
Jessica Allen University of Colorado, Boulder Christopher Lubienski University of Illinois, 

Urbana-Champaign 
Gary Anderson New York University  Sarah Lubienski University of Illinois, Urbana-

Champaign 
Michael W. Apple University of Wisconsin, 

Madison  
Samuel R. Lucas  University of California, 

Berkeley  
Angela Arzubiaga Arizona State University Maria Martinez-Coslo University of Texas, 

Arlington  
David C. Berliner  Arizona State University  William Mathis University of Colorado, Boulder 
Robert Bickel  Marshall University  Tristan McCowan  Institute of Education, London  
Henry Braun Boston College  Heinrich Mintrop University of California, 

Berkeley  
Eric Camburn  University of Wisconsin, Madison  Michele S. Moses University of Colorado, Boulder 
Wendy C. Chi* University of Colorado, Boulder Julianne Moss  University of Melbourne  
Casey Cobb  University of Connecticut  Sharon Nichols  University of Texas, San Antonio  
Arnold Danzig  Arizona State University  Noga O'Connor University of Iowa  
Antonia Darder  University of Illinois, Urbana-

Champaign 
João Paraskveva  University of Massachusetts, 

Dartmouth  
Linda Darling-Hammond Stanford University  Laurence Parker University of Illinois, Urbana-

Champaign 
Chad d'Entremont Strategies for Children Susan L. Robertson Bristol University 
John Diamond Harvard University  John Rogers University of California, Los Angeles 
Tara Donahue Learning Point Associates  A. G. Rud Purdue University 
Sherman Dorn University of South Florida  Felicia C. Sanders The Pennsylvania State 

University 
Christopher Joseph Frey Bowling Green State 

University  
Janelle Scott University of California, Berkeley  

Melissa Lynn Freeman* Adams State College Kimberly Scott Arizona State University  
Amy Garrett Dikkers University of Minnesota  Dorothy Shipps  Baruch College/CUNY  
Gene V Glass  Arizona State University  Maria Teresa Tatto Michigan State University  
Ronald Glass University of California, Santa Cruz  Larisa Warhol University of Connecticut  
Harvey Goldstein Bristol University  Cally Waite  Social Science Research Council  
Jacob P. K. Gross  Indiana University  John Weathers University of Colorado, Colorado 

Springs  
Eric M. Haas  WestEd  Kevin Welner University of Colorado, Boulder 
Kimberly Joy Howard* University of Southern 

California 
Ed Wiley  University of Colorado, Boulder 

Aimee Howley  Ohio University  Terrence G. Wiley Arizona State University  
Craig Howley  Ohio University  John Willinsky  Stanford University  
Steve Klees  University of Maryland  Kyo Yamashiro  University of California, Los Angeles 
Jaekyung Lee  SUNY Buffalo  * Members of the New Scholars Board 

 
 

 



Educational Policy Analysis Archives Vol. 20 No. 17  35 

 

archivos analíticos de políticas educativas 
consejo editorial 

Editor:  Gustavo E. Fischman (Arizona State University) 
Editores. Asociados Alejandro Canales (UNAM) y Jesús Romero Morante  (Universidad de Cantabria) 

 
Armando Alcántara Santuario Instituto de 

Investigaciones sobre la Universidad y la 
Educación, UNAM  México 

Fanni Muñoz  Pontificia Universidad Católica de 
Perú 

Claudio Almonacid  Universidad Metropolitana de 
Ciencias de la Educación, Chile 

Imanol Ordorika   Instituto de Investigaciones 
Economicas – UNAM, México 

Pilar Arnaiz Sánchez Universidad de Murcia, 
España 

Maria Cristina Parra Sandoval Universidad de 
Zulia, Venezuela 

Xavier Besalú  Costa Universitat de Girona, España Miguel A. Pereyra Universidad de Granada, España   
Jose Joaquin Brunner  Universidad Diego Portales, 

Chile 
Monica Pini Universidad Nacional de San Martín, 

Argentina 
Damián Canales Sánchez  Instituto Nacional para 

la Evaluación de la Educación, México 
Paula Razquin UNESCO, Francia   

María Caridad García  Universidad Católica del 
Norte, Chile 

Ignacio Rivas Flores Universidad de Málaga, 
España      

Raimundo Cuesta Fernández  IES Fray Luis de 
León, España 

Daniel Schugurensky Universidad de Toronto-
Ontario Institute of Studies in Education, Canadá   

Marco Antonio Delgado Fuentes Universidad 
Iberoamericana, México 

Orlando Pulido Chaves Universidad Pedagógica 
Nacional, Colombia 

Inés Dussel  FLACSO, Argentina José Gregorio Rodríguez Universidad Nacional de 
Colombia   

Rafael Feito Alonso Universidad Complutense de 
Madrid, España 

Miriam Rodríguez Vargas Universidad Autónoma 
de Tamaulipas, México 

Pedro Flores Crespo Universidad Iberoamericana, 
México 

Mario Rueda Beltrán Instituto de Investigaciones 
sobre la Universidad y la Educación, UNAM  
México   

Verónica García Martínez Universidad Juárez 
Autónoma de Tabasco, México 

José Luis San Fabián Maroto Universidad de 
Oviedo, España 

Francisco F. García Pérez Universidad de Sevilla, 
España 

Yengny Marisol Silva Laya Universidad 
Iberoamericana, México 

Edna Luna Serrano  Universidad Autónoma de Baja 
California, México 

Aida Terrón Bañuelos Universidad de Oviedo, 
España 

Alma Maldonado  Departamento de Investigaciones 
Educativas, Centro de Investigación y de 
Estudios Avanzados, México 

Jurjo Torres Santomé Universidad de la Coruña, 
España   

Alejandro Márquez Jiménez Instituto de 
Investigaciones sobre la Universidad y la 
Educación, UNAM  México 

Antoni Verger Planells University of Amsterdam, 
Holanda   

José Felipe Martínez Fernández  University of 
California Los Angeles, USA 

Mario Yapu Universidad Para la Investigación 
Estratégica, Bolivia   

 

 



Identifying Quality Pre-Kindergarten Programs  36 

 

 

arquivos analíticos de políticas educativas 
conselho editorial 

Editor:  Gustavo E. Fischman (Arizona State University) 
Editores Associados: Rosa Maria Bueno Fisher e Luis A. Gandin  

(Universidade Federal do Rio Grande do Sul) 
 

 
Dalila Andrade de Oliveira Universidade Federal de 

Minas Gerais, Brasil 
Jefferson Mainardes Universidade Estadual de 

Ponta Grossa, Brasil 
Paulo Carrano Universidade Federal Fluminense, 

Brasil 
Luciano Mendes de Faria Filho Universidade 

Federal de Minas Gerais, Brasil 
Alicia Maria Catalano de Bonamino Pontificia 

Universidade Católica-Rio, Brasil 
Lia Raquel Moreira Oliveira Universidade do 

Minho, Portugal 
Fabiana de Amorim Marcello Universidade 

Luterana do Brasil, Canoas, Brasil 
Belmira Oliveira Bueno Universidade de São Paulo, 

Brasil 
Alexandre Fernandez Vaz Universidade Federal de 

Santa Catarina, Brasil 
António Teodoro Universidade Lusófona, Portugal 

Gaudêncio Frigotto Universidade do Estado do Rio 
de Janeiro, Brasil 

Pia L. Wong California State University Sacramento, 
U.S.A 

Alfredo M Gomes Universidade Federal de 
Pernambuco, Brasil 

Sandra Regina Sales Universidade Federal Rural do 
Rio de Janeiro, Brasil 

Petronilha Beatriz Gonçalves e Silva Universidade 
Federal de São Carlos, Brasil 

Elba Siqueira Sá Barreto Fundação Carlos Chagas, 
Brasil 

Nadja Herman Pontificia Universidade Católica –
Rio Grande do Sul, Brasil 

Manuela Terrasêca Universidade do Porto, Portugal 

José Machado Pais Instituto de Ciências Sociais da 
Universidade de Lisboa, Portugal 

Robert Verhine Universidade Federal da Bahia, 
Brasil 

Wenceslao Machado de Oliveira Jr. Universidade 
Estadual de Campinas, Brasil 

Antônio A. S. Zuin Universidade Federal de São 
Carlos, Brasil 

  
 
 


	EPAAv20n17