Data-driven decision making in early education: Evidence From North Carolina’s Pre-K program Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 9/3/2018 Facebook: /EPAAA Revisions received: 1/20/2019 Twitter: @epaa_aape Accepted: 1/20/2019 education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 27 Number 18 March 4, 2019 ISSN 1068-2341 Data-Driven Decision Making in Early Education: Evidence From North Carolina’s Pre-K Program Michael Little American Institutes for Research Lora Cohen-Vogel James Sadler & Becca Merrill University of North Carolina at Chapel Hill United States Citation: Little, M., Cohen-Vogel, L., Sadler, J., & Merrill, B. (2019). Data-driven decision making in early education: Evidence From North Carolina’s Pre-K program. Education Policy Analysis Archives, 27(18). http://dx.doi.org/10.14507/epaa.27.4198 Abstract: The purpose of this study is to shed light on the use of data in early education settings—specifically, North Carolina’s Pre-K program. In this mixed-methods study, we draw upon in-depth interviews and survey data to examine (1) the types of data available to educators in Pre-K, (2) the ways in which data are intended to be used, (3) how data are reportedly used, and (4) the facilitators and inhibitors of effective data -driven decision making. Our findings reveal that Pre-K settings are data-rich environments, often with informal data collected through developmental screening tools and formative assessment systems. We find that engagement with and use of these data for instruction is variable. Finally, we find data sharing between grades is inconsistent, but an i mportant factor predicting data sharing is co-location of Pre-K programs within elementary school buildings. We consider our findings in the context of existing academic literature and http://epaa.asu.edu/ojs/ http://dx.doi.org/10.14507/epaa.27.4198 Education Policy Analysis Archives Vol. 27 No. 18 2 discuss the implications for policy and practice. Keywords: Education Policy; Data Use; Early Childhood Education; Pre-K Toma de decisiones basada en datos en educación de la primera infancia: Evidencia del programa Pre-K de North Carolina Resumen: Este estudio se centra en el uso de datos en entornos de educación in de la primera infancia, específicamente en el programa de Pre-K de North Carolina en los Estados Unidos. En este estudio de métodos mixtos, nos basamos en entrevistas en profundidad y datos de encuestas para examinar (1) los tipos de datos disponibles para educadores en Pre-K, (2) las formas en que se pretende utilizar los datos, (3) ) cómo se utilizan los datos, y (4) los facilitadores e inhibidores de la toma de decisiones efectiva basada en datos. Nuestros hallazgos revelan que los ajustes de Pre -K son entornos ricos en datos, a menudo con datos informales recopilados a través de evaluación de desarrollo y sistemas de evaluación formativa. Encontramos que la participación y el uso de estos datos para la instrucción es variable. Finalmente, encontramos que el in tercambio de datos entre los grados es inconsistente, pero un factor importante que predice el intercambio de datos es la ubicación de los programas de Pre-K dentro de los edificios de las escuelas primarias. Consideramos nuestros hallazgos en el contexto de la literatura académica existente y discutimos las implicaciones para la política y la práctica. Palabras-clave: Política educativa; Uso de datos; Educación de la primera infancia; Pre -K Tomada de decisão baseada em dados na educação infantil: Evidênc ia do programa Pre-K da North Carolina Resumo: Este estudo enfoca o uso de dados em ambientes de educação infantil - especificamente, o programa Pre-K da North Carolina nos EUA. Neste estudo de métodos mistos, utilizamos entrevistas em profundidade e dados de pesquisa para examinar (1) os tipos de dados disponíveis para os educadores no Pré -K, (2) as maneiras pelas quais os dados se destinam a ser usados, (3) ) como os dados são usados, e (4) os facilitadores e inibidores de tomadas de decisões eficazes baseadas em dados. Nossas descobertas revelam que as configurações pré-K são ambientes ricos em dados, muitas vezes com dados informais coletados por meio de triagem de desenvolvimento e por meio de sistemas de avaliação formativa. Achamos que o envolvimento e o uso desses dados para instrução é variável. Finalmente, nós achamos que o compartilhamento de dados entre as notas é inconsistente, mas um fator importante que prediz o compartilhamento de dados é a localização dos programas pré-K nos edifícios das escolas elementares. Consideramos nossas descobertas no contexto da literatura acadêmica existente e discutimos as implicações para políticas e práticas. Palavras-chave: Política Educacional; Uso de dados; Educação Infantil; Pré -K Data-Driven Decision Making in Early Educatio n 3 Intoduction Educational settings are increasingly becoming data-rich. Many studies of data use in schools have focused on K-12 (e.g., Anderson, Leithwood & Strauss, 2010; Horn, Kane & Wilson, 2015; Means et al., 2009; Murnane, Sharkey & Boudett, 2005; Sharkey & Murnane, 2003; Sutherland, 2004; Wohlstetter, Datnow & Park, 2008). But, early childhood education programs are increasingly expected to gather data about teachers and children and to use the data to drive decision making (Stein et al., 2013; Yazejian & Byant, 2013; Zweig et al., 2015). Advocates of data use in early education argue that data can be used to help understand the developmental needs of children, inform teachers about instructional modifications needed, and support effective development and administration of programs and policies (ChildTrends, n.d.). Further, the Race to the Top-Early Learning Challenge (RTT-ELC) encouraged states to integrate previously disparate data through the development of centralized, longitudinal data systems that track children into school and beyond (Data Quality Campaign, 2016). Despite these changes, little empirical research has examined how the proliferation of data is playing out in Pre-K settings—a limitation we address in the present study. The data for this article are drawn from a larger, five-year project called Early Education in Rural North Carolina, which is funded by the IES Early Learning Network. The project includes three studies that follow a cohort of children as they progress from Pre-K through third grade in North Carolina. The first study, of which the authors of this article represent, examines the policy context surrounding early learning in the state. The second and third studies focus on classroom environment and child outcomes, respectively. In the policy study, we use a conceptual framework, modified from Bryk and colleagues’ (2010) Organizing Schools for Improvement, to guide our inquiry that focuses on a range of essential supports that facilitate early learning. Examples of essential supports include family-school connections and professional capacity. We focus on the connections among the essential supports as well as the alignment of elements within essential supports, both horizontally and vertically. For example, in one essential support, instructional guidance, we analyze the extent to which standards, curricula, and assessments are aligned within grades as well as across them. Our conceptual framework posits that when these essential supports are aligned, children will experience positive, sustained learning gains in the early grades (Bryk, et al., 2010). Here, we focus only on the findings related to one of the essential supports—data-driven decision making (DDDM). To investigate DDDM, we draw upon semi-structured interviews, survey data from Pre-K teachers, and content analysis of policy documents. In this concurrent, mixed-methods study, we report on our first year of data collection, which focused on North Carolina’s Pre-K program. Our inquiry as it relates to DDDM was guided by the following research questions: What data are collected and available to educators in the NC Pre-K program? How are teachers, program site administrators, and county level administrators expected to use the data they collect or are provided? Do teachers, site administrators, and county administrators use data as intended? What are the key facilitators and inhibitors of effective DDDM in NC Pre-K? We begin with a review of the literature on DDDM, which provides a framework for our analysis. We then provide a contextual overview of early education in North Carolina and discuss the data collection and analytic methods employed in this study. After presenting our findings, arrayed by our research questions, we discuss them in relation to the existing literature and point to directions for future research into DDDM in early education. Education Policy Analysis Archives Vol. 27 No. 18 4 Data-Driven Decision Making in Education: A Review of the Literature This review is organized around a framework developed by Cohen-Vogel and Harrison (2013) that partitions the literature on DDDM into three parts: (1) data access and availability, (2) capacity for data use and action, and (3) cultures of data use. Access and Availability of Data Research confirms that an abundance of data is available in educational settings, including early educational settings (Firestone & González, 2007; Guskey, 2003; Halverson, Grigg, Prichett, & Thomas, 2007; Ingram, Louis, & Schroeder, 2004; Louis, Leithwood, Wahlstrom, & Anderson, 2010; Yazejian & Bryant, 2013). The data available are derived from several different sources. In the context of the accountability movement and increased mandates surrounding testing and data collection, one category of data is what Firestone and Gonzalez (2007) called externally derived. Externally derived data are data that are collected by actors other than classroom teachers. Examples of these data include summative assessment data and administrative data on attendance. These data often carry a sense of authority, due, in part, to their perceived objectivity, validity, and reliability within the testing and measurement communities (Anderson et al., 2010). While summative assessment data, collected externally, is not common in early childhood education, state and local Pre-K programs, as well as Head Start, are developing integrated data systems that store administrative data. These data systems include data such as attendance and special education status (Early Childhood Data Collaborative, n.d.). Another category of data is internally derived. Internally derived data are data that are informal and often collected by classroom teachers during the process of instruction. Both Black and Wiliam (1998) and Firestone and Gonzalez (2007) argued that internally generated performance data—such as formative assessments—are critical sources of information to educators. Firestone and Gonzalez (2007) further classified internal data as either informal, including teacher observations or anecdotal records, or more formal data, including homework assessments and end of unit examinations. As Black and Wiliam (1998) articulated, these forms of data are critical for providing teachers with the necessary information to drive instructional changes. They note that it is “important to look at or listen carefully to the talk, the writing, and the actions through which pupils develop and display the state of their understanding [...] for this will initiate the interaction through which formative assessment aids learning” (p. 7). A recent study by Farrell & Marsh (2016) found that informal sources of data, such as student work products, were identified by teachers as the most useful tools to inform changes in instructional delivery. In early education, many state and local Pre-K programs, as well as Head Start, require programs to collect ongoing formative assessment data and developmental screening data to track student progress (NIEER, n.d.). For example, Head Start program guidelines state, “a program must conduct standardized and structured assessments, which may be observation-based or direct, for each child that provide ongoing information to evaluate the child’s developmental level and progress…” (ACF, n.d.). Apart from classifying data in educational settings as externally or internally derived and formal or informal, the literature on DDDM also suggests that not all data are equally available or valued. Some research shows that teachers question the validity of data from standardized assessments; on occasion, school climates arise wherein “being dismissive of externally generated achievement data is a cultural trait that teachers learn and pass on to other teachers” (Ingram, Louis, & Schroeder, 2004, p. 1273; also see Guskey, 2007 and Cohen-Vogel & Harrison, 2013). Other educators—district and school administrators amongst them, however, have been found to view these kinds of data more favorably, primarily for evaluative and staffing purposes (e.g., Anderson, Data-Driven Decision Making in Early Educatio n 5 Leithwood & Strauss, 2010; Cohen-Vogel, 2011; Guskey, 2007). In early education settings, external data are less common. Under federal law, states are not required to administer standardized assessments until the third grade, for example. And while there is an increasing number of states that have some form of kindergarten entry assessments (Little & Cohen-Vogel, 2017; Little, Cohen- Vogel, & Curran, 2016), these assessments are commonly collected internally by classroom teachers. Finally, the literature has illuminated how teachers access the data that are available to them. Petrides and Guiney (2002) noted that data systems available to educators have become increasingly complex, replacing antiquated systems of the past where “offices in schools maintained independent sources of data with these sources rarely relat[ing] to each other” leading to “data redundancy and inaccuracies” (p. 7). Data centralization efforts have extended in recent years to include early education. One such effort was the RTT-ELC, in which applications were evaluated on the extent to which states developed coordinated early education data systems. For example, selection criterion B2, Supporting Effective Uses of Comprehensive Assessment Systems, stated the importance of, “Articulating an approach for aligning and integrating assessments and sharing assessment results, as appropriate, in order to avoid duplication of assessment and coordinate services for High-Need Children served by multiple Early Learning and Development Programs” (U.S. Department of Education, n.d.). Capacity for Data Use and Action Another dimension of the DDDM literature focuses on developing capacity for data use among educators (Datnow & Lea, 2016; Halverson et al., 2007; Murnane et al., 2005). For example, Louis and colleagues (2010) showed that in schools with higher levels of data use, “principals and teachers report increasing efforts to develop the capacity of teachers to engage collectively in data analysis for instructional decision making” (p. 192). This reflects an effort to develop analysis skills internally. The same authors find that schools reporting low levels of data use rely on external experts to facilitate data use. While schools and districts increasingly provide supports for data use, many educators, especially those under accountability pressures, do not feel prepared to use data effectively (Gallagher et al., 2008; Murnane et al., 2005; Sharkey & Murnane, 2003). Reflecting on a comprehensive review of the literature on teacher capacity for data use, Datnow and Lea (2016) argued that, “in order to be more successful, capacity building should directly address teachers’ beliefs, and data use must be decoupled from external accountability demands” (p. 7). In addition to capacity for data use, researchers have outlined the ways in which data can be acted upon. Firestone and Gonzalez (2007) provided a typology of data uses, which include data to guide instructional actions, data to enlighten teachers and school leaders, and data to mobilize support for actions. An example of data to guide instructional actions was provided by Cohen-Vogel (2011) who found that school leaders use student performance data make strategic staffing decisions and plan professional development. Using data to enlighten and mobilize support can take many different forms, such as communicating progress to parents, though these methods are infrequently linked to instructional modifications and actions. Louis et al. (2010) reported that schools leaders and teachers frequently used data to identify problems but rarely used data to solve problems. Finally, Cohen-Vogel and Harrison (2013) reported evidence to expand Firestone and Gonzalez’s (2007) conceptualization of guidance for instructional actions into two separate types: (1) using data to guide the structure of a school’s learning environment and (2) using data to guide instructional practice. Cultures of Data Use The third and more recent strand of inquiry surrounding DDDM relates to what Cohen- Vogel and Harrison (2013) called a culture of data use. Emblematic of having a culture of data use are Education Policy Analysis Archives Vol. 27 No. 18 6 teachers who value data and set coherent norms and expectations regarding the use of data and mutual accountability (Wohlstetter et al., 2008). Firestone and Gonzalez (2007) noted that such settings have an atmosphere of “organizational learning,” where a focus is placed on “improved instruction, problem solving, and an investment in the long term that incorporates teachers’ and principals’ voices” (p. 152). Wohlstetter and colleagues (2008) showed that in schools with positive cultures of data use, teachers “rely heavily on one another for support, new instructional strategies, and discussions about data” (p. 253). Additionally, teachers’ “structured time around data discussions was probably the most important scaffolding for continuous improvement” in the schools studied (p. 253). Others have focused on the role of school leaders in facilitating strong cultures of data use. Cohen-Vogel and Harrison (2013) found that leaders in schools with strong cultures of data use actively work to promote, “an atmosphere of learning that emphasized continuous improvement and long-term vision shared by teachers and principals” (p. 140). Marsh & Farrell (2015) posited a framework to help school leaders support positive cultures of data use. Their framework pushes leaders to assess teachers’ current capacity for data use, their needs for development, and how to cater supports to build data use capacity and a positive data use culture. As Cohen-Vogel and Harrison (2013) argued, the literature implicates three key parts involved in data-driven decision making: access to data, use of data to inform actions and build support for actions, and cultures of data use to foster and promote DDDM. This framework, developed from the literature on DDDM, guided the specification of our research questions. Our first research question examines the types of data that are available in the NC Pre-K program. Our second and third research questions explore the ways actors are expected to engage with and act upon data as well as how they do so. Finally, in the fourth research question, we explore cultures of data use by probing the facilitators and inhibitors of effective DDDM in the NC Pre-K program. Early Education in North Carolina North Carolina has often been heralded as a leader in early childhood policy. This recognition stems from the Smart Start initiative that began in 1993. The program began as a demonstration program in 18 of the state’s 100 counties with the goal of ensuring that all children ages 0-5 were healthy and prepared for school. Financed by state dollars and private donations, Smart Start provides children with quality child care, health care, and family support services at birth. In 1999, the program had expanded to all 100 counties (Ladd, Muschkin, & Dodge, 2014). Building on the early successes of Smart Start and a need to boost academic skills for at-risk 4-year-olds, a second program, More at Four, was adopted by the NC legislature in 2001. “At risk” is defined in the program as having a developmental delay, learning disability, a chronic health problem, limited English proficiency, or family income < 75% of the state’s median. Children whose parents are on active military-duty are granted automatic eligibility (NIEER, 2013). More at Four was renamed NC Pre-K in 2011 and moved from the state’s Department of Public Instruction (DPI) to the Department of Health and Human Services (DHHS). Most recently, in 2011, North Carolina received an approximately $70 million grant through RTT-ELC to increase the quality of its early education programs and provide links with health, nutrition, mental health, and family support services for its neediest children (Piker & Jewkes, 2014). Among other projects, the RTT-ELC dollars supported the revision of the state’s star-rated license system for early learning programs, the Tiered Quality Rating and Improvement System (TQRIS). The state also applied for but did not receive a Preschool Development Grant from the federal government in 2014 to expand its NC Pre-K program (USDOE, 2014). Data-Driven Decision Making in Early Educatio n 7 Despite the state’s past successes and innovation in early childhood education, the state has wavered in its support of these programs in recent years. For example, according to NIEER’s 2017 State of Preschool, North Carolina ranks 26th in preschool access for 4-year-olds, 18th in spending, and only meets eight out of the 10 quality standards (North Carolina previously met all 10). This change is largely a function of other states and localities increasing supports and surpassing North Carolina rather than declines in programming. For example, North Carolina’s per pupil spending on 4-year-olds has increased since 2002, when the NIEER State of Preschool yearbooks began, but has not increased as much as other states and localities (NIEER, 2017). The NC Pre-K Program As stated above, originally named More at Four, the NC Pre-K program was formed in 2001. The program currently serves about 29,000 4-year-olds each year, which is nearly 25% of the state’s 4-year-olds. NC Pre-K program classrooms can be located in private centers, Head Start centers, or public schools. Approximately 50% of program slots are provided in public school settings (Peisner-Feinberg et al., 2018). All NC Pre-K grantees must meet state-determined program standards and earn a four- or five-star rating under the state’s childcare star-rated licensing system (NIEER, 2018). Lead teachers in the NC Pre-K program are required to hold or be working toward a NC Birth through Kindergarten (B-K) license or the equivalent and assistant teachers are required to hold or be working toward an Associate Degree in early childhood education or child development or a Child Development Associate (CDA) credential (Peisner-Feinberg et al., 2018). The state’s Division of Child Development and Early Education (DCDEE) releases the official program requirements and guidance for NC Pre-K sites annually. As these requirements document for the 2016-2017 school year, NC Pre-K sites must operate for a minimum of 6.5 hours a day for 10 months of instructional time. Programs must adhere to the state content standards—the North Carolina Foundations for Early Learning and Development (Foundations)—which focus on five developmental domains of early childhood education: (1) approaches to play and learning, (2) emotional and social development, (3) health and physical development, (4) language development and communication, and (5) cognitive development (DCDEE, 2016). NC Pre-K guidelines also stipulate requirements regarding selection of curricula and use of developmental screeners and formative assessments. The program guidelines contain a list of 15 approved curricula that can be used in the classroom. The curricula are approved by the NC Child Care Commission, a 17-member body appointed by the governor and legislatures that includes parents, academics, and “general citizens.” Curricula must: (1) be comprehensive, meaning it addresses all five domains of the Foundations, (2) be evidence-based, meaning it includes a theoretical and/or research justification for content, and (3) align with the Foundations, meaning it describes intent or developmental goals of given experiences (DCDEE, 2016). NC Pre-K programs are required to conduct ongoing formative assessments to inform teacher’s instruction and to monitor children’s growth and development. Similar to curricula approval requirements, there is an approved list of 11 formative assessments that NC Pre-K sites can use. The state does not specify how many times a child should be assessed by the formative assessment. Sites must also use one of four approved developmental screeners to “identify children who should be referred for further evaluation and testing based on concerns in one or more developmental domains” (DCDEE, 2016). Every child in NC Pre-K, with the exception of those already in an Individualized Education Program (IEP), must be screened either six months before the school year begins or within 90 days after. Site administrators are also required to review all results and share them with families. Education Policy Analysis Archives Vol. 27 No. 18 8 The NC Pre-K Program meets all of the National Institute for Early Education Research’s (NIEER) recommended quality benchmarks (NIEER, 2013) and has shown moderate to large effect sizes (d = .34–1.14) on early reading and math skills (Peisner-Feinberg & Schaaf, 2011). Longitudinal findings on NC Pre-K indicate smaller but significant effect sizes (d = .12–.19) on NC third-grade reading and math standardized tests as compared to non-attenders (Peisner-Feinberg & Schaaf, 2010). NC Pre-K has also shown larger effects for dual-language learners compared to other children in the program (Peisner-Feinberg & Schaaf, 2009; Peisner-Feinberg et al., 2015). Having provided a contextual overview of early education in North Carolina, with a particular focus on the NC Pre-K program, we now turn to detail the methods used in our study. Methods Study Setting: Rural Counties in North Carolina The study is part of a larger project on early learning in rural North Carolina. That project follows a sample of children from six counties in the central part of the state. In Table 1, we provide basic demographic data for each of the participating counties along with the state as a whole. Counties have been assigned pseudonyms, and the values in this table are rounded to preserve the anonymity of participating counties. In general, the counties in our study are more racially diverse than the state average and have higher rates of poverty. The percentage of Black residents meets or exceeds the state average in five of the six sampled counties, and the percentage of Hispanic/Latino residents meets or exceeds it in all six. Finally, six of the six counties meet or exceeds the state estimate for percentage of people in the county living below the federal poverty line. Table 1 County Demographic Data Total Population in County % African American/Black % Hispanic/Latino Poverty Rate State 10,155,942 20 10 15 Allegro 150,000 20 10 20 Callenwood 70,000 10 15 15 Gia 60,000 30 10 15 Sundry 60,000 25 20 25 Virgil 45,000 50 10 25 Wyndfall 125,000 30 15 20 Note. Counties have been assigned pseudonyms. Data are from the North Carolina Office of Budget and Management. Retrieved from https://www.osbm.nc.gov/demog/county-estimates https://www.osbm.nc.gov/demog/county-estimates Data-Driven Decision Making in Early Educatio n 9 In total, the six counties serve approximately 2,000 children through their NC Pre-K programs as of the 2014-2015 school year. The children they serve are low-income and come from racially/ethnically diverse backgrounds. In Table 2, we provide NC Pre-K enrollment rates, by county, and by major racial subgroup. As illustrated in Table 2, Pre-K enrollment in these six counties is racially diverse, with most counties enrolling higher percentages of Black and Hispanic children than White children. Callenwood and Sundry counties enroll predominately Hispanic children while Virgil and Wyndfall enroll predominately Black children. Allegro and Gia counties serve roughly equal numbers of Black, Hispanic, and White children. Together, the Local Education Agencies (LEAs) or district administration in the six counties enroll almost 75,000 students in K-12. Several of the LEAs are among the least wealthy in the state. In 2016-17, 78 of the state’s 115 LEAs, including five of the six represented in this study, received supplemental funding from the state because their ability to generate local revenue is below the state average (NCDPI, 2017). In 2016-17, average per pupil spending (PPS) in the state was $8,296. Four of the six sampled districts had PPS below the state average; one county, Wyndfall, had more than a $1000 gap. Table 2 NC Pre-K Enrollment and Demographics by County Total County Enrollment Total African American/Black Enrollment Total Hispanic/ Latino Enrollment Total White Enrollment Allegro 480 170 170 140 Callenwood 240 40 150 50 Gia 140 50 40 50 Sundry 330 110 150 70 Virgil 140 90 30 20 Wyndfall 510 220 120 170 Note. North Carolina Early Childhood Integrated Data System. Data are from 2014-2015. Retrieved from https://www.ecids.nc.gov/ecids/ Data Collection Qualitative Data. We conducted interviews with county and state officials in the Fall and Spring of the 2016-2017 school year. In total, we had 35 participant interactions, which included 22 county or school district administrators and 13 state officials. Interviews were semi-structured and lasted approximately one hour each. Participants in the counties included Pre-K directors, LEA superintendents, and other officials with responsibilities related to the administration of NC Pre-K. Participants at the state level included representatives from the NC Department of Public Instruction’s Office of Early Learning and the Department of Health and Human Services’ Division of Child Development and Early Education. In addition to interview data, we also analyzed 97 documents collected from county and state websites. We downloaded all documents available on the state NC Pre-K website. In each county, we downloaded all documents associated with the NC Pre- K program on county-level Smart Start and LEA office websites. https://www.ecids.nc.gov/ecids/ Education Policy Analysis Archives Vol. 27 No. 18 10 We piloted the interview protocols with an administrator in a county not included in our sample and a program evaluator with intimate knowledge of the NC Pre-K program. Our interview protocol included a range of questions concerning DDDM. We asked participants questions such as “What data does the [county/district] require NC Pre-K programs to collect?” and “Does the state require Pre-K programs to use child-level data for specific purposes?” Some interview questions ask respondents about their perceptions of teachers’ use of data, for example. We did not interview any teachers as part of this study, so all reports of teacher practices are thus derived from perceptual reports from county and state administrators. Our quantitative data from the teacher survey, detailed below, is the only evidence directly collected from teachers. Quantitative Data. In addition to interview and document data, we also examined a subset of items from a survey of all participating Pre-K teachers in our study (N=59). The survey was administered in the Spring of 2017 as part of the broader Early Education in Rural North Carolina study. The broader study involved randomly selecting and recruiting 59 NC Pre-K classrooms in six rural counties in North Carolina and recruiting up to six children per classroom to follow from Pre- K to third grade. The number of classrooms recruited per county was selected in proportion to the number of NC Pre-K classrooms within that county. All teachers in our sample responded to the survey. All teachers were lead teachers of NC Pre-K classrooms. 65% of teachers worked in public school settings, 24% worked in private for-profits centers, 6% worked in Head Start centers, and 5% worked in private not-for-profit centers. Here, we include items that investigate two areas: (1) training for data use and (2) the ways in which teachers reported using assessment data. See Appendix A for the survey items. Each of the research questions is primarily answered via the qualitative interview data and the teacher survey data supplements the qualitative data, where applicable. Specifically, the teacher survey data helps to understand (1) the extent to which professional development is a facilitator of effective DDDM in NC Pre-K and (2) the ways teachers report using student assessment data. Data Analysis We analyzed the interview data and documents using a directed content analysis approach (Patton, 2002). We first analyzed the qualitative data categorically, assigning basic, descriptive codes for types of data, with subcodes that included the uses and applications of data. For example, our parent code “Uses of Assessment Data” included child codes for “Eligibility,” “Instructional Modification,” and “Needs Identification,” for example. We allowed themes to emerge from the data inductively (Miles & Huberman, 1994). As themes emerged, we added additional codes to reflect alignment between DDDM and other essential supports in our conceptual model. These include, for example, instructional supports (content standards, curricula, and assessments) and professional development. Two members of the research team coded each interview and document to promote reliability in the coding process. Additionally, coders sought out and identified disconfirming evidence (Corbin & Strauss, 2008). In the case of negative or disconfirming evidence, the team worked collaboratively to revise the coding framework—either modifying construct definitions or eliminating constructs, when appropriate. Our team met regularly to share and test emergent codes, work through coding inconsistencies, refine the coding framework and rubrics, and build reliability. Once the data were coded, members of the research team also wrote in-depth annotated memos to describe findings for each county, which cited qualitatively-coded evidence. Once all of the county- and state-level analytic memos were completed, we completed a final memo that summarized findings across all memos, highlighting areas of similarity or difference in findings across counties. Data-Driven Decision Making in Early Educatio n 11 Findings What data are collected and available to educators in the NC Pre-K program? Based on our interviews with state and county respondents and review of the NC Pre-K program guidelines, we learned that the data available to educators in the NC Pre-K program come from three principal sources: developmental screening tools, formative assessments, and state- and county-level administrative data systems. As mentioned previously, unless a child has an existing Individualized Education Program (IEP), NC Pre-K program guidelines stipulate that all children in NC Pre-K must be administered an approved developmental screener within 90 days after the first day of attendance in the program or within six months prior to the first day of attendance (DCDEE, 2016). As summarized in Table 3, the specific developmental screening tools used in the six counties are the Ages and Stages (N=1), Brigance (N=4), Dial-4 (N=2). County respondents in one county, Callenwood, reported using both the Brigance ad Dial-4 screener. In all other counties, a single screening tool was used across all Pre- K classrooms in the county. In Callenwood, selection of the screening tool differed between school- based and non-school-based Pre-K programs. The North Carolina Pre-K guidelines also state that the teacher or teacher assistant should “collect ongoing assessment data for each child by gathering information about what children know and do, how they interact with other children/adults and how they process information/solve problems” (NC Child Care Rule 10A NCAC 09.3008). Again, the state provides a pre-approved list of formative assessment tools that specific counties and/or programs can choose from. The specific formative assessment tools that each of the six counties in this study use are listed in Table 3. Aside from Virgil County, which did not report a specific formative assessment, county respondents from all other counties reported using the Teaching Strategies GOLD system. The Teaching Strategies GOLD system is aligned with the Creative Curriculum and uses an online platform where teachers score children’s development along 38 different objectives for development and learning. The frequency of teacher scoring of developmental progress varies and is determined locally (Teaching Strategies, n.d.). Table 3 Instructional Supports Used by County, Reported County Instructional Supports Used, Reported Curriculum Formative Assessment Developmental Screener Allegro Creative Curriculum TS Gold Brigance Callenwood Creative Curriculum TS Gold Brigance & Dial-4 Gia Creative Curriculum TS Gold Brigance Sundry Creative Curriculum TS Gold Ages and Stages Virgil Creative Curriculum Not reported Dial-4 Wyndfall Creative Curriculum TS Gold Brigance Education Policy Analysis Archives Vol. 27 No. 18 12 In addition to developmental screening and formative assessment data, educators in NC Pre- K have administrative data from the state available to them. There are four state-level data systems that serve as repositories for data in the NC Pre-K program: (1) NC Pre-K APP, which is a control system for Pre-K applications, prioritization, and placement; (2) NC Pre-K Plan, which is a system for monitoring teacher credential and qualifications, curricula, formative assessment, and developmental screener use, and hours of operation; (3) NC Pre-K Kids, which includes demographics of Pre-K children, attendance, and developmental screener evaluations; and (4) NC Educator Effectiveness System, which stores teacher evaluation data. Some of these systems, such as the NC Educator Effectiveness System are used more by county and state officials, while systems like the NC Pre-K Kids are used more at the program and classroom levels. How are teachers, program site administrators, and county level administrators expected to use the data they collect or are provided? Information about the intended uses of (1) developmental screener data, (2) formative assessment data, and (3) administrative data was provided through our analysis of policy documents, most notably, the NC Pre-K program guidelines, as well as interviews with state officials. In terms of developmental screeners, NC Pre-K program guidelines note, “screenings shall be used solely for the purpose of identifying children who should be referred for further evaluation and testing based on concerns in one or more developmental domains. The site-level administrator must review all developmental screening results and share the results with families” (North Carolina DCDEE, 2016). In terms of the intended use of formative assessment data, the NC Pre-K program guidelines note, “Classroom staff are required to conduct ongoing formative assessments to gather information about each child’s growth and skill development, as well as to inform instruction” (DCDEE, 2016). In terms of using the data to inform instruction, the guidelines are somewhat vague and state that teachers and teacher assistants should, “use the assessment information to tailor instruction to the individual needs of each child” and that the assessment data should be discussed with the teacher assistant to plan for each child. Finally, the guidelines state that teachers and teacher assistants should review children’s progress with his/her family. In terms of the administrative data systems, there is no guidance about how they should be used other than for reporting and accountability purposes. In other words, the guidelines are focused on what data should be entered in the systems and when they should be entered. There is no information about how the data stored in them should be used. Do teachers, site administrators, and county administrators use data as they are intended? What are the key facilitators and inhibitors of effective DDDM in NC Pre -K? Here, we present our findings for the third and fourth research questions together— weaving together the reported uses of data and the facilitators and inhibitors of using the data, where applicable. We begin by detailing findings on data use within the Pre-K year followed by information on how data are being shared vertically across grades. Use of data from developmental screeners. In general, our interviews with county-level administrators revealed reported uses of data from the developmental screener in ways that conform to the state guidelines. In all counties, county-level respondents highlighted how the developmental screener was used to facilitate needs identification for children. In Allegro county, for example, incoming NC Pre-K children are administered the DIAL-4 screening tool in the Spring before they enter the program so that specific services can be arranged. Referring to administration of the DIAL-4, a county-level respondent from Allegro noted: Data-Driven Decision Making in Early Educatio n 13 It’s required for all children to have one [developmental screening] within 90 days of starting into Pre-K. But again, we do ours in the Spring. The school system does the majority in the Spring before they’re placed in the Fall. It just can’t be done more than a year in advance but we do it in the Spring before they’re placed. And it just helps identify what services need to be put in place once they begin, if they need speech therapy, and what kind of referrals need to be made. While officials in all counties reported using a developmental screening tool to help with needs identification and arrangement of special education services, in particular, Allegro is the only county where respondents reported administering the screener during the six-month window before children enroll in NC Pre-K, rather than during the first 90 days of NC Pre-K. In Gia county, when asked about how the county uses data from the developmental screener, a county-level respondent noted that the developmental screener was used to help inform eligibility and placement decisions into NC Pre-K: Participant: It is one of the major things that we look at, the results of those developmental screeners to see whether they have an educational need. If [the child] scores below average then he does have an educational need. Interviewer: And does that mean that they have some sort of priority for placement? Participant: Yes. Although all counties must use income data to determine program eligibility for NC Pre-K, Gia County is the only county where administrators reported using the developmental screener as an additional element in the placement process. While the developmental screening tools are intended for use at the beginning of the NC Pre-K program (or, up to six months before) and primarily measure socio-behavioral outcomes that are predictors of developmental delays, an official from Virgil County noted that teachers in their county administer the screener at the beginning and end of the NC Pre-K year in order to measure growth. While this participant acknowledged that the initial screening is intended to alert teachers and parents about developmental delays, they also noted that the tool is used to “determine growth in academics”. Pre-K programs in Virgil County use the DIAL-4 developmental screener tool, which yields scores in four domains: Motor skills, concept skills, language skills, self-help skills, and social- emotional skills (Pearson, n.d). Use of data from formative assessments.Officials from all six counties reported that teachers (and themselves as program administrators) use formative assessment data to help drive instructional modifications and improvement. However, while it was clear to officials that this was the intended use of these assessments, fewer participants noted that such assessments are particularly helpful to teachers. In Callenwood County, officials reported that teachers saw the value of the ongoing assessments and that the assessments were a natural extension of effective professional practice. First, officials indicated a high level of buy-in and commitment to data use among teachers in the county. For example, one county official in Callenwood noted that: We use Curriculum Gold from Teaching Strategies and they [Pre-K teachers] enter all of the information on a child daily, weekly on how they’re achieving and at what end of the progression that [the child is] on. So, I think we do a great job of trying to make sure we stay with that. Beyond a commitment to using the assessment program with fidelity, county officials also highlighted its essential role in professional practice. As one official noted: Education Policy Analysis Archives Vol. 27 No. 18 14 Interviewer: Does the state require Callenwood County’s NC Pre-K programs to use child-level data for specific purposes? Participant: Require us? I’d have to say either no or I don’t know. But as a teacher is watching a child and recording this information, if you’re not using it to plan your instruction, there’s no use in having it. So, our teachers, I feel, are very good about using data and trying to figure out what would best meet the child’s needs in a classroom. And I can say that about our Pre-K teachers, I mean, that’s just how good they are. This excerpt highlights that teachers in this county, as perceived by the county official, do not view the formative assessments as a burdensome requirement from the State; rather, they embrace the process and use it to their advantage— a perspective not shared across officials from other counties. In Gia County, for example, county officials reported seeing less value in the formative assessment system. One official said the process is not easily integrated into the professional practice of experienced teachers: Most of our teachers are very experienced teachers. They’ve been doing it a long time and they have their ways of doing things. So, they will put stuff into Teaching Strategies GOLD and they will do the checkpoints [periodic points where teachers log a child’s progress on developmental continua], but they don’t see the value of it as of yet. They still think, “Oh, I can learn more from my checkpoints and my data sheets and that kind of thing that I did.” As this excerpt illustrates, teachers in Gia County, as reported by county respondents, use formative assessment procedures, but their engagement is largely for compliance purposes and is not deeply embedded in their professional practice. Another official from Gia County went further to suggest that teachers in the county do not engage with the TS GOLD data at all: Interviewer: Is there an expectation that they [teachers] use either that information or the Teaching Strategies GOLD information for specific purposes? Participant: Yes. We would love for them to pay attention to it. But they don’t necessarily (laughs) pay any attention to it. Using administrative data. In none of our interviews did any participant highlight how state administrative data systems could be used on an ongoing basis to help facilitate county - level decision making and program improvement. In one county, however, we did find a high level of engagement with a county-level system that combined formative assessment data from all Pre-K programs in the county. Administrators in Callenwood County reported regularly checking formative assessment data at the county level to promote high-quality programming. As one county administrator noted: Interviewer: How does the county ensure that programs are using the data? Participant: Actually [employee name] monitors all of [the county’s Pre-K teacher’s] GOLD entries [in a county-level data system]. Now, is somebody there in their classrooms every day? No. The best we can do at this point is monitor what they’re inputting and what’s going on in their classrooms because [the employee] can see all of it. If she senses problems, she lets me know and then I deal with it with the principals. But she’s pretty good about knowing and seeing. She’s in and out of those classrooms constantly. Data-Driven Decision Making in Early Educatio n 15 As the above quote demonstrates, officials from Callenwood County reported leveraging data systems to monitor instruction— monitoring that would not be feasible through direct observation of classrooms. In terms of data use for instructional improvement at the county administration level Callenwood County stood out. While all county officials reported that teachers used formative assessment data for instructional improvement, Callenwood was the only county where we found evidence of similar engagement at the county administration level. Vertical data sharing. One area of reported systematic data use that was common across the counties in our study related to sharing data on children as they progress from Pre -K into kindergarten, which we call vertical sharing. Officially, NC Pre-K program requirements state that all sites must have a data transition plan, meaning a “written transition plan showing how the needs of participating children will be implemented as they transition…into kindergarten” (NC Pre-K Program Requirements, 2018, pp. 2-6). However, the requirements do not specify the exact components that should be included (NC Pre-K Program Requirements, 2018). Counties, therefore, have autonomy in developing these policies and practices . Some county officials seemed unaware of the state requirement for an official transition plan and no officials reported an official transition policy or plan. However, some county officials reported informal vertical sharing of data from Pre-K to kindergarten. For example, a Sundry county official noted: Interviewer: Does North Carolina require NC Pre-K programs to share data on children with the elementary school that the child will attend? Participant: I don’t know that the State requires it, but we certainly do that. We want to make sure that there is a good transference of information from the Pre-K teacher to the kindergarten teacher. Also, the [Pre-K] teacher has a portfolio on all the children. We try and make sure that those portfolios get transferred into the proper kindergarten classroom as well. Despite some counties sharing data vertically, there is variability in terms of the depth of the specific data elements shared as well as in the setting types where data are likely to be shared. In Virgil County, for example, a rich portfolio of data—including assessment data and developmental screener data—are transferred from Pre-K teachers to kindergarten teachers. In contrast, officials in Wyndfall County reported that the data shared were administrative and demographic in nature, with very little information about children’s individual academic performance or needs. Even if information is transferred, there is little knowledge concerning how the information is received and applied. One official from Allegro County questioned whether data shared from Pre- K to kindergarten would be used in any meaningful way. She notes that “The information is passed on. I don’t know if it’s utilized.” Additionally, one official in Wyndfall County asserted that kindergarten teachers may like to make their own assessment of incoming children and would not see value in assessments made by the Pre-K teacher. She says of these teachers that, “they really like to catch a child as they’re coming in and they really want to form their own basis of knowledge on the child based on meeting the child and doing their own sets of assessments.” The county respondent claimed that kindergarten teachers were adamant about this approach and thus did not place much importance on vertical data sharing in her county. A significant factor mediating the prevalence of vertical data sharing is the setting of the Pre- K program—namely, whether or not the program is center-based or school-based. Officials in Allegro County, for example, noted that some administrative data are shared vertically from Pre-K to kindergarten, but this transfer most often occurs in school-based Pre-K settings and when the child’s Pre-K classroom and kindergarten classroom are in the same building. One official from this Education Policy Analysis Archives Vol. 27 No. 18 16 county noted that, “If you have a Pre-K in the [county public] school system, they're more apt to share with the kindergarten teacher because they're right there in the same building. If it was coming from a child care site, then it may not make that connection.” Officials in other counties reported similar disconnects in data sharing and stated that in cases where a child attended a center-based Pre-K, data are provided directly to the child’s parents, with the intent for them to share the data with their child’s kindergarten teacher. However, county officials do not have methods to ensure the data are shared and some expressed doubt that such information is regularly passed along to kindergarten teachers by parents. Beyond co-location of Pre-K and kindergarten classrooms facilitating the transmission of data vertically, setting type may also facilitate how actors engage with the data and make sense of it. For example, in Callenwood County, county-level administrators reported that most children who attend NC Pre-K located in an elementary school end up attending that elementary school for kindergarten. With both Pre-K and kindergarten teachers at the same site, vertical discussions about children occur along with data transmission, which may facilitate kindergarten teachers in utilizing the data. For example: Participant: So those Pre-K teachers [located in elementary schools] will probably have a PLC meeting with the kindergarten teachers at the end of the Spring (as they get ready for the new year) or the beginning of the new year, that they’ll share that data. Interviewer: And what data elements, in particular, would be shared? Participant: They will share the progression data that they have entered in Teaching Strategies. They actually will get a report, kind of a summary on a child and so they’ll share that information with those teachers. And any observations that may not be in there. Along with Callenwood County, we found similar engagement in vertical transitions and data use in Virgil County. In Wyndfall County, however, we found that there had been previous attempts at sharing data, but the attendance patterns of children inhibited effective transmission. A Wyndfall County administrator noted that: Years ago, we tried to do a data sharing process where information was transitioned to the elementary schools, but the way that Wyndfall County [Schools] are, there’s several different schools that a child could go to. So, information was shared with a school that the parents reported the child would go to and then parents moved and so information was lost. While officials from some counties expressed a commitment to sharing data vertically and helping educators to engage with the data, we note that our findings report county practices and not county policy. Our findings indicate that rather than formal policies, the counties in our sample have developed data sharing systems that are logistically feasible and follow whatever preferences district officials have on data sharing. Findings from Teacher Survey Besides receiving qualitative reports on how teachers and administrators were engaging with data, we also conducted surveys of Pre-K teachers about data use. Specifically, the survey items captured two domains of DDDM: (1) professional development supports for administering and using student assessments and (2) how teachers reported using data generated from assessments. Data-Driven Decision Making in Early Educatio n 17 As shown in Table 4, of the 59 teachers who responded to the survey, only 53% reported that they participated in training specific to administering student assessments in the past 12 months. Of those that did report receiving training, none reported that training was “Not Useful,” 16% reported that training was “Somewhat Useful,” 52% reported that training was “Useful,” and 32% reported that training was “Very Useful”. A similar percentage of teachers reported receiving training specific to how to use data from student assessments in the past 12 months. Specifically, 47% of survey respondents reported participating in these trainings. Of these 28 teachers who received training, none reported that these trainings were “Not Useful,” 29% reported that they were “Somewhat Useful,” 36% reported that they were “Useful,” and 36% also reported that they were “Very Useful.” In sum, approximately half of the teachers in our study reported professional development related to administering or using assessment data in the past 12 months and, in both cases, teachers reported by a strong majority that these trainings were useful or very useful to them. Table 4 Survey Items on Assessment and Data Trainings Yes No Participated in Training for Administering Assessments? 53% 47% Not Useful Somewhat Useful Useful Very Useful Usefulness of Training 0% 16% 52% 32% Training Focused on Using Student Assessment Data? Not Useful Somewhat Useful Useful Very Useful Usefulness of Training 0% 29% 36% 36% Note. N=59. The second question was only asked of the teachers who reported yes for the first question. As stated above, the NC Pre-K guidelines require that teachers use formative assessment data to inform instruction and monitor the growth and development of children. When NC Pre-K teachers in our study were asked what they used student assessment results for (see Table 5), most teachers reported that they used assessment results to serve these purposes. For example, when given the option to select the ways that a participant utilizes assessment data (not restricted to a single choice), 92% of respondents reported using assessment results to help them individualize instruction, and 80% of respondents reported using assessment results to identify individual learning needs. Additionally, 56% reported using results to identify children who may need additional testing, 37% reported using results to refer children for supplemental services, and 5% (n=3) reported using results to reassign a child to a more appropriate school or classroom placement (with two of these detailing that they used results for kindergarten placement). It is possible that some teachers may not consider TS GOLD to be an “assessment,” thus impacting the validity of responses to this item. Education Policy Analysis Archives Vol. 27 No. 18 18 Table 5 Uses of Assessment Data Assessment Data Used to… To identify children who may need additional testing. 56% To Refer children for supplemental services. 37% To identify individual learning needs. 80% To help me individualize instruction. 92% To reassign a child to a more appropriate school or classroom placement. 5% Notes. N=59 Discussion Through initiatives like the RTT-ELC, the era of “big data” has reached early education. This study highlights the characteristics of DDDM occurring in NC Pre-K programs in six rural counties in North Carolina. The DDDM framework introduced earlier in this article delineates three broad areas of focus: (1) access and availability of data, (2) capacity for data use and action, and (3) cultures of data use. In discussing our findings, we return to our research questions, which explore aspects of each focus area, through the lens of participant reports of data in early education. Access and Availability of Data To investigate access and availability of data in NC Pre-K, we ask: What data are collected in the NC Pre-K program? We find that a large amount of internal data is available in all counties, due to state requirements to develop these data sources. This finding suggests that the “data centralization” efforts highlighted by Petrides and Guiney (2002) has indeed extended down to early childhood education. However, counties do not report utilizing external data sources for program or instructional improvement. All county-level respondents reported utilizing developmental screeners and, to varying degrees, formative assessment data for instructional improvement. Officials from only one county employed formative assessment data for program improvement at the county level, even though the formative assessment software, used by all counties in the sample, has this functionality. In addition, administrative data are available in all counties, though their use is not for program or instructional improvement. While teachers in NC Pre-K classrooms had access to developmental screener results, it remains unclear how teachers might use these results to inform instruction. Capacity for Data Use and Action We ask two questions to better understand how available data are used in NC Pre-K classrooms: How are teachers, site administrators, and county administrators supposed to use the data they collect, and do teachers, site administrators, and county administrators use data as they are intended? Recall that, beyond the teacher survey results, findings related to teacher practices are derived via county-level respondent perceptions of their practices. North Carolina program requirements for NC Pre-K clearly stipulate how data are supposed to be used. Developmental screeners are intended to identify educational needs; formative assessments are intended to ascertain Data-Driven Decision Making in Early Educatio n 19 knowledge levels, social interactions, and problem-solving skills; and administrative data, though not intended for use by educators, facilitate program operation. In terms of the importance of data for decision-making, all county-level officials reported valuing the data they receive on children as tools for instructional improvement. All officials reported that formative assessment data are employed to some degree in the classroom for instructional improvement; although, how teachers reportedly value and use assessment data varies considerably. Some county-level participants revealed that some teachers—especially veteran teachers—do not interact with formative assessment data other than to comply with district or state requirements. Participants suggested that these teachers preferred using information drawn from previous data collection methods and rely on this data to inform their instruction. This finding is consistent with others who have found that teachers, and veteran teachers in particular, are resistant to data derived from standardized assessment systems (Ingram, Louis, & Schroeder, 2004). Further, this finding—that some teachers do not interact with formative assessment beyond compliance— seems at odds with the evidence from the teacher survey. In the teacher survey results, we reported that 92% of teachers use assessment data to individualize instruction and 80% use data to identify individual learning needs. It is possible that teachers report using data for these purposes at very high rates because it is the ostensible purpose of formative assessment data and they believe they are using the data effectively for these purposes. The county-level officials may not hold the same view and do not see teachers making the connection between data derived from formative assessments and modifications in professional practice. Indeed, other researchers have found that, while teachers often report using assessment data to individualize instruction, few teachers are able to articulate the specific ways in which they do so (Cohen-Vogel & Harrison, 2013). County-level respondents generally reported data-use practices that adhered to the practices outlined at the state level. However, we document cases in which counties adapt approved instruments for their own purposes. Developmental screeners are utilized in some counties to inform NC Pre-K placement and as a pre-and post- measure of growth. We also find that some counties use data more effectively than others, especially those with more buy-in concerning the assessment tool. Where most county respondents reported use of formative assessments by teachers for instructional improvement in the classroom, we found only one county utilizing the data for county-level program improvement. At the county level, there is a lack of utilization, rather than access to data, for program improvement among most of the counties sampled. The formative assessment data are only as good as a teacher’s ability to assess students and the information they enter as well as how well they act upon the data. Assessment data could be improved by providing all NC Pre-K teachers with professional development on administering assessments, considering that less than half of Pre-K teachers surveyed reported receiving such training in the past 12 months. Training has the potential to increase the accuracy of assessment data and possibly increase buy-in for use of assessment data. That said, reliable and valid scoring of formative assessment tools, such as GOLD, is difficult and such professional development should be rigorous and evidence-based. Last, our findings indicate that the data, to some degree, are as useful as teachers make them, with some teachers seeking guidance from assessment information for instructional improvement more than others. Our last research question begins to unravel the under-utilization of data in most counties: What are facilitators and inhibitors of effective DDDM in NC Pre-K? The answer straddles two elements of the DDDM framework: capacity for data use and action, as well as cultures of data use. Data from our survey suggest that less than half of teachers surveyed received training on using data from formative assessments to improve instruction in the past 12 months. Thus, a lack of training Education Policy Analysis Archives Vol. 27 No. 18 20 likely inhibits the capacity for more effective data use, especially in counties where teachers are resistant to full implementation of formative assessment data. County-level professional development on data use for program improvement also is necessary to more efficiently use the data available. A second inhibitor of data use includes barriers to vertical transference of data between Pre- K and kindergarten teachers. Our findings indicate three barriers to vertical data sharing: logistics, location, and aversion. Regarding logistics, in some counties students have a number of schooling options for kindergarten after leaving their Pre-K site and the attendance patterns can be unpredictable. Counties have not found a system to accommodate these challenging logistics to connect students with their data, prompting at least one county to discontinue attempts at vertical data sharing. Next, the location of NC Pre-K sites dictates, to some extent, the characteristics of vertical data sharing. Kindergarten teachers in schools with a Pre-K program located in the building are more likely to receive student assessment data from a Pre-K teacher and more likely to have a face to face meeting to discuss the data. Center-based NC Pre-K sites reportedly rely on parents to transfer data to the schools their children attend. The third barrier to vertical data sharing describes a cultural aversion on the part of kindergarten teachers to prior student assessments, a facet of the “cultures of data use” element of the DDDM framework. According to a few participants, some kindergarten teachers prefer to develop their own assessment of incoming students, believing that data from previous years may bias their initial impression of a child. Such preferences may be anchored in a lack of trust in the assessment tools being used, a distrust in the capacity of NC Pre-K teachers as assessors, or a combination of both. Whatever the underlying motivation, we find evidence that such aversions influence officials in those counties to place less value on vertical data sharing, prompting less focus on developing rich portfolios of student data. This phenomenon may explain some of the data sharing disparities among counties and deserves further investigation. Limitations Though our data involve three levels of governance in the early education system, including state officials, county officials, and teachers, we largely rely on participant reporting in our findings. Our research team did not make direct county or school observations. Therefore, we stress that our findings remain true to participant observation and thus are subject to participant bias. Nonetheless, these participant views are valuable, especially as our informants represent administrative roles, because policy actors’ views shape implementation at the street-level (Mintrop, 2012). Additionally, we focus on rural counties in North Carolina, which likely are not representative of more urban counties in the state. Finally, our data only come from a single state and are thus not representative of any other system. Looking Forward The significant investment in early childhood education demands efficient use of resources. Our findings indicate that DDDM is an area that should be targeted for improvement. We suggest two areas of focus: data utilization and vertical data sharing. Though county officials report that data are used for decision making by themselves and teachers, we detected that the data available could be utilized to a greater capacity. In order to realize this goal, LEA’s need to focus on professionally developing their workforce, including teachers and county-level officials, in data application or hire trained data analysts to counsel and coach teachers on data use. Further, school administrators likely play a critical role in developing cultures of data use in schools and in engaging teachers in structured meetings to utilize student data (Cohen-Vogel & Harrison, 2013). Data-Driven Decision Making in Early Educatio n 21 A second area of focus is on breaking down barriers to vertical data sharing. Counties need to develop formal vertical sharing policies, which would likely garner more resources and attach more cultural importance to the practice. More research needs to be conducted to document and describe how prevalent vertical data sharing is, what types of data kindergarten teachers receive, the types of students they receive data on, and how they utilize the data. Further, research should attend to the challenges of vertical data sharing between center-based NC Pre-K sites and elementary schools. North Carolina policymakers seem to be aware of this need and have responded by developing a state commission titled, “State Agency Collaboration on Early Childhood Education/Transition from Preschool to Kindergarten,” and one of the key priorities of the commission is developing stronger connections between disparate data systems (NC DHHS, 2017). References Anderson, S., Leithwood, K., & Strauss, T. (2010). Leading data use in schools: Organizational conditions and practices at the school and district levels. Leadership and Policy in Schools, 9(3), 292-327. https://doi.org/10.1080/15700761003731492 Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. London, UK: School of Education, King’s College Bryk, A. S., Sebring, P. B., Allensworth, E., Easton, J. Q., & Luppescu, S. (2010). Organizing schools for improvement: Lessons from Chicago. University of Chicago Press. Cohen, F. (2003). Mining data to improve teaching. Educational Leadership, 60(8), 53-56. Cohen-Vogel, L. (2011). “Staffing to the test”: Are today’s school personnel practices evidence based? Educational Evaluation and Policy Analysis, 33(4), 483-505. https://doi.org/10.3102/0162373711419845 Cohen-Vogel, L., & Harrison, C. (2013). Leading with data: Evidence from the National Center on Scaling Up Effective Schools. Leadership and Policy in Schools, 12(2), 122-145. https://doi.org/10.1080/15700763.2013.792934 Datnow, A., & Hubbard, L. (2016). Teacher capacity for and beliefs about data-driven decision making: A literature review of international research. Journal of Educational Change, 17(1), 7-28. https://doi.org/10.1007/s10833-015-9264-2 Data Quality Campaign. (2016). Roadmap for early childhood and K-12 data linkages: Key areas to ensure quality implementation. Retrieved from https://2pido73em67o3eytaq1cp8au- wpengine.netdna-ssl.com/wp-content/uploads/2016/03/Roadmap-for-Early-Childhood- and-K12-Data-Linkages.pdf Farrell, C. C., & Marsh, J. A. (2016). Metrics matter: How properties and perceptions of data shape teachers’ instructional responses. Educational Administration Quarterly, 52(3), 423-462. https://doi.org/10.1177/0013161X16638429 Firestone, W., & González, R. (2007). Culture and processes affecting data use in school districts. Yearbook of the National Society for the Study of Education, 106,132–154. https://doi.org/10.1111/j.1744-7984.2007.00100.x Gallagher, L., Means, B., Padilla, C. (2008). Teachers' Use of Student Data Systems to Improve Instruction: 2005 to 2007. Washington, DC: U.S. Department of Education. Guskey, T. R. (2003). How classroom assessments improve learning. Educational Leadership, 60(5), 6– 11. Halverson, R., Grigg, J., Prichett, R., & Thomas, C. (2007). The new instructional leadership: Creating data-driven instructional systems in school. Journal of School Leadership, 17(2), 159– 194. https://doi.org/10.1177/105268460701700202 https://2pido73em67o3eytaq1cp8au-wpengine.netdna-ssl.com/wp-content/uploads/2016/03/Roadmap-for-Early-Childhood-and-K12-Data-Linkages.pdf https://2pido73em67o3eytaq1cp8au-wpengine.netdna-ssl.com/wp-content/uploads/2016/03/Roadmap-for-Early-Childhood-and-K12-Data-Linkages.pdf https://2pido73em67o3eytaq1cp8au-wpengine.netdna-ssl.com/wp-content/uploads/2016/03/Roadmap-for-Early-Childhood-and-K12-Data-Linkages.pdf Education Policy Analysis Archives Vol. 27 No. 18 22 Horn, I. S., Kane, B. D., & Wilson, J. (2015). Making sense of student performance data: Data use logics and Mathematics teachers’ learning opportunities. American Educational Research Journal, 52(2), 208-242. https://doi.org/10.3102/0002831215573773 Ingram, D., Louis, K., & Schroeder, R. G. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106(6), 1258– 1287. https://doi.org/10.1111/j.1467-9620.2004.00379.x Kerr, K. A., Marsh, J. A., Ikemoto, G., Darilek, H., & Barney, H. (2006). Strategies to promote data use for instructional improvement: Actions, outcomes, and lessons from three urban districts. American Journal of Education, 112(4), 496-520. https://doi.org/10.1086/505057 Little, M. H., & Cohen-Vogel, L. (2017). Ready for school? Assessing America’s kindergarteners. Teachers College Record. Advance Online Publication. Little, M. H., Cohen-Vogel, L., & Curran, F. C. (2016). Facilitating the transition to kindergarten: What ECLS-K data tell us about school practices then and now. AERA Open, 2(3). https://doi.org/10.1177/2332858416655766 Louis, K., Leithwood, K., Wahlstrom, K., & Anderson, S. (2010). Learning from lead-ership: Investigating the links to improved student learning. New York, NY: Wallace Foundation. Marsh, J. A., & Farrell, C. C. (2015). How leaders can support teachers with data-driven decision making: A framework for understanding capacity building. Educational Management Administration & Leadership, 43(2), 269-289. https://doi.org/10.1177/1741143214537229 Means, B., Padilla, C., DeBarger, A., & Bakia, M. (2009). Implementing data-informed decision making in schools: Teacher access, supports and use. Washington, DC: U.S. Department of Education. Mintrop, H. (2012). Bridging accountability obligations, professional values and (perceived) student needs with integrity. Journal of Educational Administration, 50(5), 695-726. https://doi.org/10.1108/09578231211249871 Murnane, R. J., Sharkey, N. S., & Boudett, K. P. (2005). Using student-assessment results to improve instruction: Lessons from a workshop. Journal of Education for Students Placed at Risk (JESPAR), 10(3), 269-280. https://doi.org/10.1207/s15327671espr1003_3 National Institute for Early Education Research. (2018). North Carolina State of Preschool Report Card. Retrieved from http://nieer.org/wp-content/uploads/2018/05/North- Carolina_YB2017-1.pdf Peisner-Feinberg, E. S., Van Manen, K. W., & Mokrova, I. L. (2018). Variations in enrollment practices in the NC Pre-K program: 2016-2017 Statewide Evaluation. Chapel Hill, NC: The University of North Carolina, FPG Child Development Institute. Petrides, L., & Guiney, S. (2002). Knowledge management for school leaders: An ecological framework for thinking schools. Teachers College Record, 104(8), 1702–1717. https://doi.org/10.1111/1467-9620.00217 Piker, R. A., & Jewkes, A. M. (2014). Early childhood Eeucation in the context of educational reform: Possibilities and causes for concern. In The Obama Administration and Educational Reform (pp. 3-26). Emerald Group Publishing Limited. https://doi.org/10.1108/S1479- 358X20130000010001 Sharkey, N., & Murnane, R. (2003). Learning from student assessment results. Education Leadership, 61(3), 77-81. Stein, A., Freel, K., Hanson, A. T., Pacchiano, D., & Eiland-Williford, B. (2013). The Educare Chicago research-program partnership and follow-up study: Using data on program graduates to enhance quality improvement efforts. Early Education & Development, 24(1), 19- 41. https://doi.org/10.1080/10409289.2013.739542 https://doi.org/10.1177/2332858416655766 Data-Driven Decision Making in Early Educatio n 23 Sutherland, S. (2004). Creating a culture of data use for continuous improvement: A case study of an Edison project school. American Journal of Evaluation, 25(277), 277-293. https://doi.org/10.1177/109821400402500302 Wohlstetter, P., Datnow, A., & Park, V. (2008). Creating a system for data-driven decision-making: Applying the principal-agent framework. School Effectiveness and School Improvement, 19(3), 239- 259. https://doi.org/10.1080/09243450802246376 Yazejian, N., & Bryant, D. (2013). Embedded, collaborative, comprehensive: One model of data utilization. Early Education & Development, 24(1), 68-70. https://doi.org/10.1080/10409289.2013.736128 Zweig, J., Irwin, C. W., Kook, J. F., & Cox, J. (2015). Data collection and use in early childhood education programs: Evidence from the Northeast Region (REL 2015–084). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northeast & Islands. Retrieved from http://ies.ed.gov About the Authors Michael Little, Ph.D. American Institutes for Research mlittle@air.og Lora Cohen-Vogel University of North Carolina at Chapel Hill cohenvog@email.unc.edu James Sadler University of North Carolina at Chapel Hill sadlerja@email.unc.edu Becca Merrill University of North Carolina at Chapel Hill bmerrill@email.unc.edu mailto:mlittle@air.og mailto:cohenvog@email.unc.edu mailto:sadlerja@email.unc.edu mailto:bmerrill@email.unc.edu Education Policy Analysis Archives Vol. 27 No. 18 24 education policy analysis archives Volume 27 Number 18 March 4, 2019 ISSN 1068-2341 Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non- commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or EPAA. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizona State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A1 (Brazil), SCImago Journal Rank, SCOPUS, SOCOLAR (China). Please send errata notes to Audrey Amrein-Beardsley at audrey.beardsley@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. http://www.doaj.org/ mailto:audrey.beardsley@asu.edu https://www.facebook.com/EPAAAAPE Data-Driven Decision Making in Early Educatio n 25 education policy analysis archives editorial board Lead Editor: Audrey Amrein-Beardsley (Arizona State University) Editor Consultor: Gustavo E. Fischman (Arizona State University) Associate Editors: David Carlson, Lauren Harris, Eugene Judson, Mirka Koro-Ljungberg, Scott Marley, Molly Ott, Iveta Silova (Arizona State University) Cristina Alfaro San Diego State University Amy Garrett Dikkers University of North Carolina, Wilmington Susan L. Robertson Bristol University Gary Anderson New York University Gene V Glass Arizona State University Gloria M. Rodriguez University of California, Davis Michael W. Apple University of Wisconsin, Madison Ronald Glass University of California, Santa Cruz R. Anthony Rolle University of Houston Jeff Bale University of Toronto, Canada Jacob P. K. Gross University of Louisville A. G. Rud Washington State University Aaron Bevanot SUNY Albany Eric M. Haas WestEd Patricia Sánchez University of University of Texas, San Antonio David C. Berliner Arizona State University Julian Vasquez Heilig California State University, Sacramento Janelle Scott University of California, Berkeley Henry Braun Boston College Kimberly Kappler Hewitt University of North Carolina Greensboro Jack Schneider University of Massachusetts Lowell Casey Cobb University of Connecticut Aimee Howley Ohio University Noah Sobe Loyola University Arnold Danzig San Jose State University Steve Klees University of Maryland Jaekyung Lee SUNY Buffalo Nelly P. Stromquist University of Maryland Linda Darling-Hammond Stanford University Jessica Nina Lester Indiana University Benjamin Superfine University of Illinois, Chicago Elizabeth H. DeBray University of Georgia Amanda E. Lewis University of Illinois, Chicago Adai Tefera Virginia Commonwealth University David E. DeMatthews University of Texas at Austin Chad R. Lochmiller Indiana University A. Chris Torres Michigan State University Chad d'Entremont Rennie Center for Education Research & Policy Christopher Lubienski Indiana University Tina Trujillo University of California, Berkeley John Diamond University of Wisconsin, Madison Sarah Lubienski Indiana University Federico R. Waitoller University of Illinois, Chicago Matthew Di Carlo Albert Shanker Institute William J. Mathis University of Colorado, Boulder Larisa Warhol University of Connecticut Sherman Dorn Arizona State University Michele S. Moses University of Colorado, Boulder John Weathers University of Colorado, Colorado Springs Michael J. Dumas University of California, Berkeley Julianne Moss Deakin University, Australia Kevin Welner University of Colorado, Boulder Kathy Escamilla University ofColorado, Boulder Sharon Nichols University of Texas, San Antonio Terrence G. Wiley Center for Applied Linguistics Yariv Feniger Ben-Gurion University of the Negev Eric Parsons University of Missouri-Columbia John Willinsky Stanford University Melissa Lynn Freeman Adams State College Amanda U. Potterton University of Kentucky Jennifer R. Wolgemuth University of South Florida Rachael Gabriel University of Connecticut Kyo Yamashiro Claremont Graduate University Education Policy Analysis Archives Vol. 27 No. 18 26 archivos analíticos de políticas educativas consejo editorial Editor Consultor: Gustavo E. Fischman (Arizona State University) Editores Asociados: Armando Alcántara Santuario (Universidad Nacional Autónoma de México), Angelica Buendia, (Metropolitan Autonomous University), Alejandra Falabella (Universidad Alberto Hurtado, Chile), Antonio Luzon, (Universidad de Granada), José Luis Ramírez, (Universidad de Sonora), Paula Razquin (Universidad de San Andrés), Maria Alejandra Tejada-Gómez (Pontificia Universidad Javeriana, Colombia) Claudio Almonacid Universidad Metropolitana de Ciencias de la Educación, Chile Ana María García de Fanelli Centro de Estudios de Estado y Sociedad (CEDES) CONICET, Argentina Miriam Rodríguez Vargas Universidad Autónoma de Tamaulipas, México Miguel Ángel Arias Ortega Universidad Autónoma de la Ciudad de México Juan Carlos González Faraco Universidad de Huelva, España José Gregorio Rodríguez Universidad Nacional de Colombia, Colombia Xavier Besalú Costa Universitat de Girona, España María Clemente Linuesa Universidad de Salamanca, España Mario Rueda Beltrán Instituto de Investigaciones sobre la Universidad y la Educación, UNAM, México Xavier Bonal Sarro Universidad Autónoma de Barcelona, España Jaume Martínez Bonafé Universitat de València, España José Luis San Fabián Maroto Universidad de Oviedo, España Antonio Bolívar Boitia Universidad de Granada, España Alejandro Márquez Jiménez Instituto de Investigaciones sobre la Universidad y la Educación, UNAM, México Jurjo Torres Santomé, Universidad de la Coruña, España José Joaquín Brunner Universidad Diego Portales, Chile María Guadalupe Olivier Tellez, Universidad Pedagógica Nacional, México Yengny Marisol Silva Laya Universidad Iberoamericana, México Damián Canales Sánchez Instituto Nacional para la Evaluación de la Educación, México Miguel Pereyra Universidad de Granada, España Ernesto Treviño Ronzón Universidad Veracruzana, México Gabriela de la Cruz Flores Universidad Nacional Autónoma de México Mónica Pini Universidad Nacional de San Martín, Argentina Ernesto Treviño Villarreal Universidad Diego Portales Santiago, Chile Marco Antonio Delgado Fuentes Universidad Iberoamericana, México Omar Orlando Pulido Chaves Instituto para la Investigación Educativa y el Desarrollo Pedagógico (IDEP) Antoni Verger Planells Universidad Autónoma de Barcelona, España Inés Dussel, DIE-CINVESTAV, México José Ignacio Rivas Flores Universidad de Málaga, España Catalina Wainerman Universidad de San Andrés, Argentina Pedro Flores Crespo Universidad Iberoamericana, México Juan Carlos Yáñez Velazco Universidad de Colima, México javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/816') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/819') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/820') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/4276') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/1609') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/825') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/797') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/823') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/798') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/555') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/814') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/2703') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/801') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/826') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/802') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/3264') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/804') Data-Driven Decision Making in Early Educatio n 27 arquivos analíticos de políticas educativas conselho editorial Editor Consultor: Gustavo E. Fischman (Arizona State University) Editoras Associadas: Kaizo Iwakami Beltrao, (Brazilian School of Public and Private Management - EBAPE/FGV, Brazil), Geovana Mendonça Lunardi Mendes (Universidade do Estado de Santa Catarina), Gilberto José Miranda, (Universidade Federal de Uberlândia, Brazil), Marcia Pletsch, Sandra Regina Sales (Universidade Federal Rural do Rio de Janeiro) Almerindo Afonso Universidade do Minho Portugal Alexandre Fernandez Vaz Universidade Federal de Santa Catarina, Brasil José Augusto Pacheco Universidade do Minho, Portugal Rosanna Maria Barros Sá Universidade do Algarve Portugal Regina Célia Linhares Hostins Universidade do Vale do Itajaí, Brasil Jane Paiva Universidade do Estado do Rio de Janeiro, Brasil Maria Helena Bonilla Universidade Federal da Bahia Brasil Alfredo Macedo Gomes Universidade Federal de Pernambuco Brasil Paulo Alberto Santos Vieira Universidade do Estado de Mato Grosso, Brasil Rosa Maria Bueno Fischer Universidade Federal do Rio Grande do Sul, Brasil Jefferson Mainardes Universidade Estadual de Ponta Grossa, Brasil Fabiany de Cássia Tavares Silva Universidade Federal do Mato Grosso do Sul, Brasil Alice Casimiro Lopes Universidade do Estado do Rio de Janeiro, Brasil Jader Janer Moreira Lopes Universidade Federal Fluminense e Universidade Federal de Juiz de Fora, Brasil António Teodoro Universidade Lusófona Portugal Suzana Feldens Schwertner Centro Universitário Univates Brasil Debora Nunes Universidade Federal do Rio Grande do Norte, Brasil Lílian do Valle Universidade do Estado do Rio de Janeiro, Brasil Flávia Miller Naethe Motta Universidade Federal Rural do Rio de Janeiro, Brasil Alda Junqueira Marin Pontifícia Universidade Católica de São Paulo, Brasil Alfredo Veiga-Neto Universidade Federal do Rio Grande do Sul, Brasil Dalila Andrade Oliveira Universidade Federal de Minas Gerais, Brasil