TOWARDS AN OBJECTIVE EVALUATION OF TEACHER PERFORMANCE Journal website: http://epaa.asu.edu/ojs/ Manuscript received: 28/10/2016 Facebook: /EPAAA Revisions received: 18/1/2017 Twitter: @epaa_aape Accepted: 18/1/2017 education policy analysis archives A peer-reviewed, independent, open access, multilingual journal Arizona State University Volume 25 Number 24 March 20, 2017 ISSN 1068-2341 Investigating Student Exposure to Competency-Based Education Sarah Ryan & Joshua D. Cox Education Development Center United States Citation: Ryan, S., & Cox, J. D. (2017). Investigating student exposure to competency-based education. Education Policy Analysis Archives, 25(24). http://dx.doi.org/10.14507/epaa.25.2792 Abstract: In recent years, most U.S. states have revised policy by providing schools at least some flexibility to move away from the Carnegie unit system, with its focus on credits and “seat time,” toward competency-based policies that link student advancement to mastery of content. Yet, there is little systematically collected information about how competency-based education is implemented, making it difficult to evaluate the impact on student outcomes. Using data from 600 students in grades 9-12 and confirmatory factor analytic techniques, we report initial reliability and validity results from the pilot administration of a survey designed to capture student exposure to elements that have been described as essential to a competency-based, student-centered model for learning and instruction. These elements include mastery-based progression, personalization, flexible assessment, and the development of specific skills and dispositions. Results suggest that the survey offers a way to reliably measure and study variation in the implementation of competency-based education. Importantly, the survey provides a way to capture implementation from the student perspective, leveraging the fact that student reports about their classroom experiences may be a particularly reliable source of information about instructional practice. Keywords: competency-based education; secondary education; structural equation modeling; surveys http://epaa.asu.edu/ojs/ http://dx.doi.org/10.14507/epaa.25.2792 Education Policy Analysis Archives Vol. 25 No. 24 2 La exposición de los estudiantes a la educación basada en competencias Resumen: Recientemente, la mayoría de los estados en los Estados Unidos han revisado la póliza educativa con métodos que han proveído flexibilidad para alejarse del sistema de unidades de Carnegie, que se centra en los créditos y el "tiempo en el asiento o escritorio", hacia políticas basadas en competencias que conectan el progreso del estudiante con la materia del contenido. Sin embargo, hay poca información recopilada sistemáticamente sobre cómo se implementa la educación basada en competencias, lo que dificulta evaluar el impacto en los resultados de los estudiantes. Utilizando datos de 600 estudiantes de los grados 9-12 con técnicas analíticas con factores confirmatorios, informamos los resultados iniciales de confiabilidad y validez de la administración inicial de una encuesta diseñada para capturar la exposición de los estudiantes a elementos que se han descritos como esenciales para un estudiante basado en competencias centradas para el aprendizaje y la instrucción. Estos elementos incluyen la progresión basada en el la maestría, la personalización, la evaluación flexible y el desarrollo de habilidades y disposiciones específicas. Los resultados sugieren que la encuesta ofrece una manera de medir y estudiar con fiabilidad la variación en la implementación de la educación basada en competencias. Es importante destacar que la encuesta proporciona una forma de captar la implementación desde la perspectiva del estudiante, aprovechando el hecho de que los informes de los estudiantes sobre sus experiencias en el salón pueden ser una fuente particularmente confiable de información sobre la práctica de instrucción. Palabras-clave: educación basada en competencias; educación secundaria; modelos de ecuaciones estructurales; encuestas A exposição os estudantes à educação baseada em competências Resumo: Recentemente, a maioria dos estados dos Estados Unidos ter revisto a política educacional com métodos que proporcionaram flexibilidade para se afastar das unidades do sistema Carnegie, que incide sobre o crédito e "tempo no banco ou mesa" para políticas com base em competências que ligam o progresso do estudante com teor de matéria. No entanto, pouca informação é sistematicamente coletadas sobre como a educação baseada em competências é implementado, o que torna difícil avaliar o impacto sobre os resultados dos alunos. Usando dados de 600 alunos 9.12 com técnicas analíticas com fator de confirmação, relatamos os resultados iniciais de confiabilidade e validade da administração inicial de uma pesquisa projetado para capturar a exposição aluno elementos que fo ram descritos como essencial para uma aprendizagem baseada em competências centrado no aluno e instrução. Esses elementos incluem progressão com base em perícia, personalização, avaliação flexível e desenvolvimento de habilidades e disposições específicas. Os resultados sugerem que a pesquisa fornece uma maneira de medir e estudar a variação na implementação da educação baseada em competências de forma confiável. É importante ressaltar que a pesquisa fornece uma maneira de capturar a implementação da perspectiva do estudante, aproveitando o facto dos relatórios de estudantes sobre as suas experiências em sala de aula pode ser uma fonte particularmente confiável de informações sobre a prática de ensino. Palavras-chave: educação baseada em competências; educação secundária; modelagem equação estrutural; pesquisas Investigating Student Exposure to Competency -Based Education 3 Introduction Many states are beginning to move away from policies under which student advancement is based on the traditional Carnegie unit system, with its focus on credits and “seat time,” and toward policies that provide schools with the flexibility to link advancement to a student’s mastery of content (Marion & Leather, 2015; Scheopner-Torres, Brett, & Cox, 2015). The movement toward competency-based approaches to learning and instruction addresses a recommendation from the National Center on Time & Learning (2011) that learning programs should factor in the different amounts of time students need to achieve proficiency and should consider basing student advancement on proficiency, rather than on the amount of time students spend sitting in a classroom. This shift also reflects the position that students are more engaged in their learning when they are granted greater flexibility in how they earn credit (United States Department of Education, 2011). Stakeholders representing a variety of perspectives are increasingly embracing the notion that student-centered (Jobs For the Future, 2014) reforms like competency-based education will yield increased achievement and better preparation for college and the workforce. Similar claims have been made about other so-called student-centered approaches, including both personalized learning (Bill & Melinda Gates Foundation, 2015) and deeper learning (Alliance for Excellent Education, 2012). Both educators and researchers report that implementation of competency-based education (also referred to as proficiency-based education, or simply competency education) varies widely (Scheopner-Torres et al., 2015; Steele et al., 2014). Implementation challenges include carrying out necessary yet substantial shifts in instruction, assessment, and grading in the classroom; addressing student misconceptions about such shifts in school practices and policies; and supporting student development of the skills and dispositions necessary for student success under a competency-based model (Le, Wolfe, & Steinberg, 2014; Scheopner-Torres et al., 2015). Further, there is a dearth of systematically collected information about how and to what extent instructional practices reflecting elements described as essential to competency-based education are being implemented, making it difficult to investigate how competency-based reforms are or are not leading to changes in student outcomes (Freeland, 2014; Haynes et al., 2016; Le et al., 2014). Moreover, and perhaps surprisingly given the emphasis on student voice in the literature on competency-based education (Le et al., 2014), relatively little is known about how students perceive and experience changes to school policies and instructional practices. In what follows, we report results from the pilot administration of a survey designed to capture student exposure to elements currently described as essential to competency-based education. These elements, which are multi-dimensional, include mastery-based progression, personalization, flexible assessment, and the development of specific skills and dispositions (Sturgis, 2016). Valid and reliable measures of student exposure to elements of competency-based education are needed if researchers are to more carefully investigate the relationship between implementation and student outcomes. Given that student reports about their classroom experiences appear to be a reliable source of information about instructional practice (Kane & Staiger, 2012), measures based on student-reported exposure to competency-based education hold promise for facilitating research of this nature. The results indicate that this survey reliably measures various dimensions underlying each of the hypothesized elements of competency-based education, providing support for the future use of the survey by both researchers and practitioners to examine implementation and its association with student outcomes. Yet, the results also raise questions about whether competency- based education is a distinct, bounded model for secondary education comprising all four elements Education Policy Analysis Archives Vol. 25 No. 24 4 described as essential to the approach. We conclude by discussing the implications of these findings for practice, for research, and for policy. Renewed Interest in Competency-Based Education In recent years, competency-based education has garnered renewed attention as a reform to increase graduation rates and to ensure that students are prepared for postsecondary success (Scheopner-Torres et al., 2015). The theory is that a larger share of students will ultimately reach proficiency in a given content area if they are given the freedom to advance at their own pace and if their learning experiences are tailored to their needs and interests (Lewis et al., 2014; Sturgis & Patrick, 2010). Several generations of competency-based education have been documented in the research literature, tracing back to at least the 1960s (Brown, 1994). Current interest in this reform continues a long tradition of challenging the idea that learning happens at a particular pace and place for all students (Brown, 1994; Dewey, 1938; Le et al., 2014; Spady & Mitchell, 1977). Until recently, however, competency-based education was primarily considered a niche approach targeting vocational education and the adult learning segment of the higher education system (Ford, 2014). Recent advances in online learning, learning analytics, and adaptive learning technology, combined with growing demand for demonstrable college and career readiness outcomes among high school graduates, have prompted renewed and expanded interest in the development of major competency- based initiatives (Ford, 2014). Marking a shift from earlier iterations of competency-based education, policymakers have been more responsive to the present movement (Steele et al., 2014). As of 2014, 42 states had authorized at least some flexibility for local education agencies to link the completion of academic credit to demonstration of proficiency rather than only to seat time (Carnegie Foundation for the Advancement of Teaching, 2014). Over time, competency-based education has been described using a variety of frameworks, all of which share some degree of overlap. Several decades ago, Spady (1977) described competency- based education as consisting of six critical elements including outcomes, time, instruction, measurement, certification, and program adaptability. More recently, based on interviews with educators and a literature review of recent developments in the competency-based education movement, Steele and colleagues (2014) articulated three core principles, including flexible pacing, opportunities to personalize learning, and the requirement that students demonstrate proficiency (and earn credit) by applying knowledge and skills. Most recently, Sturgis (2016) outlined the following four elements as essential to the approach: student advancement through demonstration of mastery based on explicit and measurable learning objectives reflecting important competencies; personalized instruction that provides a student with customized supports and opportunities to engage in anytime/anywhere learning; multiple modes of, and multiple opportunities for, assessment that allow a student to apply his or her learning; and an emphasis on the development of specific skills and dispositions critical for success in a learning environment where students are expected to play a more prominent role in directing their own learning. The research described below, and the student survey used in the research, focus on the four essential elements described by Sturgis (2016). These elements share substantial overlap with the components of competency-based education articulated both in other recent literature (for example, see Competency Works, 2014; Patrick & Sturgis, 2013; Scheopner-Torres et al., 2015; Steele et al., 2014) and in legislation and policy in states that have adopted competency-based education. Alongside renewed interest in the promise of competency-based education has been a “wild proliferation” (Weise, 2014) of interest in student-centered reforms broadly, perhaps especially in Investigating Student Exposure to Competency -Based Education 5 the realm of education philanthropy. For example, the past several years have also witnessed growing enthusiasm for other student-centered models, including personalized learning (Bill & Melinda Gates Foundation, 2015) and deeper learning (Alliance for Excellent Education, 2012; William & Flora Hewlett Foundation, 2014), as a path to better student college and workforce preparation. While the terms used to name these overlapping approaches vary, they share in common some or all of the elements used to describe competency-based education. This suggests growing consensus about the core of student-centered educational reform broadly, and the centrality of these elements to student-centered reforms. Nonetheless, the variation in terminology is noteworthy, particularly to the extent that it both reflects and contributes to imprecision, if not confusion, in how the elements of competency-based education and other student-centered approaches are defined—an issue that is not new, but was recognized some three decades ago by Spady and colleagues (Spady, 1977; Spady & Mitchell, 1977). Echoing this lack of precision, both in language and meaning, is wide variability within and across schools and districts in how competency- based policies and practices are conceived and implemented. Ultimately, these issues pose significant challenges for efforts to understand the relationship between the implementation of competency- based education and student outcomes. Implementation of Competency-Based Education and Student Outcomes While many are enthusiastic about the promise of competency-based education, the bridge between a promising idea and impact on student outcomes is implementation—and innovations are seldom implemented as intended (Berman & McLaughlin, 1976). Although there is general agreement among proponents of competency-based education about its essential elements, implementation varies substantially across states and districts (Steele et al., 2014; Scheopner-Torres et al., 2015; Stump & Silvernail, 2015). Competency-based policies range from those that simply allow flexibility in awarding credit, to policies that completely transform the education system “from a time-based system to a learning-based system” (Sturgis & Patrick, 2010, p. 4). Some states require implementation of a competency-based diploma system in all districts; in other states districts can choose to implement elements of competency-based education, but state policies neither require nor restrict implementation of the reform. States also vary in how they implement competency-based grading policies and graduation requirements (Sturgis, 2014). In some contexts, districts are required to make competency-based credits available, but in other places districts are allowed to decide whether these will be offered. Some states specify how students will demonstrate competency (e.g., scores on statewide or national tests, portfolios), while others leave it to the discretion of the district to determine how competencies are measured. Certain states allow credits to be awarded for demonstration of competency in any course, but others only offer this in certain subjects. Beyond variation in the nature of state competency-based policies, how these policies are implemented at the local level also differs. For instance, one study (Steele et al., 2014) of implementation in several districts throughout the United States documented variability in which competency-based education elements districts chose to implement. Some districts focused primarily on the assessment of proficiency on competencies, but others focused on personalization of learning, including student choice and flexible pacing. This study also found variability across sites in a number of other areas including: the extent to which learning experiences that occurred outside of school hours and off of school grounds were counted toward the fulfillment of course requirements, the criteria used to determine student proficiency on a specific standard or competency, and the use of technology to facilitate online learning opportunities and operate learning management systems. Education Policy Analysis Archives Vol. 25 No. 24 6 In order to develop a body of empirical evidence about how competency-based education is related to student outcomes, it will be necessary to continue building a better understanding of variability in implementation, including the development of valid and reliable ways to measure implementation fidelity. In particular, practitioners and policymakers would benefit from a better understanding of how students’ experiences with competency-based education are consistent with current descriptions of the model and its component elements. As research about the use of student perception surveys to gather information about students’ classroom experiences has demonstrated, student surveys may provide more consistent results than classroom observations about what is happening in the classroom (Balch, 2012; Ferguson, 2012; Kane & Staiger, 2012), offering an important source of information about implementation. Researchers describe five aspects of implementation fidelity (Dane & Schneider, 1998; Durlak & DuPre, 2008; Dusenbury, Brannigan, Falco, & Hansen, 2003). Adherence references whether specific features of a program are implemented as prescribed by the program model. Exposure reflects the amount of the program delivered and/or the extent to which those served by the program were exposed to each element of the model. Quality refers to how well the program is implemented, including the caliber of the delivered model features. Program differentiation indicates the degree to which the essential elements of a program are distinguishable from each other and from other programs. Finally, responsiveness reflects the manner in which those exposed react to or engage in the program. At the student level, the survey used in this research captures information about student exposure to each of the hypothesized elements of competency-based education; this includes information about whether the student is exposed to instructional practices associated with each element and information about the frequency with which the student is exposed to such practices (the survey does not, however, capture duration, or the amount of time a student is exposed to these practices). At the school level, results can provide information about adherence, or the extent to which specific elements prescribed in the current literature about competency-based education are implemented. Results from this survey can also be used to explore program differentiation, or the extent to which hypothesized essential elements of competency-based education are distinguishable from each other and whether, as a whole, these elements constitute a bounded, coherent model that is distinct from other programs—an issue of particular emphasis in this research. Conceptual Framework This research offers one of the first empirically based attempts to operationalize and measure what have been proposed as essential elements of competency-based education. Information about students’ exposure to competency-based education can support practitioners and policymakers in making more informed decisions about implementation, and can facilitate research addressing the relationship between implementation and student outcomes. The limited empirical research base on the implementation and outcomes of competency- based education, along with the absence of conceptual clarity about what key elements of competency-based education should look like in practice, pose challenges for measurement. Our approach in the current study is informed by the position that validating new constructs involves an iterative, ongoing process of refining both theory and measurement (Westen & Rosenthal, 2003). We begin this process by drawing from the existing literature on competency-based education to inform the constructs we study in the current research—constructs intended to capture the hypothesized key elements of competency-based education as measured by their underlying dimensions. Investigating Student Exposure to Competency -Based Education 7 At the core of competency-based education is an approach to teaching and learning that allows students to advance in school based on demonstrated mastery of important content, typically as defined through measurable learning objectives reflecting competencies required for success in college, career, and life (Le et al., 2014). One dimension of progression through demonstration of mastery is competency-based pacing (CB PACE; see Figure 1), which involves student exposure to instructional practices that allow them to work at their own level and pace, advancing to the next level of study based on demonstrated mastery of skills rather than on their age or number of hours they’ve spent in class (Le et al., 2014). A second dimension is competency-specific feedback (CB FEED), which is intended to ensure that students understand what is expected of them, including how their efforts will be assessed and what mastery (or proficiency—the terms are often used interchangeably) looks like. See Appendix Table A1 for a description of each dimension and the indicators of each. One of the more elusive elements of competency-based education—and other student- centered reforms—is student personalization. The notion of personalization in K–12 education broadly is both increasingly popular and variably defined (for example, see Le et al., 2014). In the competency-based education literature, dimensions of personalization typically include personalized options for practicing and applying skills (PERS APP; this is sometimes referred to as “anytime, anywhere” learning), along with exposure to personalized instructional practices (PERS INSTR) that respond to the needs of individual students. With respect to student assessment, the literature on competency-based education also emphasizes flexible assessment (FLEX). Flexible assessment reflects the idea that students should have multiple and varied opportunities to demonstrate mastery of important skills as part of a summative assessment system (i.e., not simply quizzes or exams). An emphasis on flexible assessment also reflects a belief that students may require multiple attempts to demonstrate mastery, such that falling short of mastery on the first attempt is interpreted not as a failure but as indicating a need for additional practice and support before subsequent attempts. Finally, current literature suggests the need to support student development of specific skills and dispositions hypothesized as necessary for student success within a competency-based framework for learning and instruction. Although we use the phrase skills and dispositions here, we note that the literature uses a range of overlapping terms, including, for example, work-study habits, non-cognitive skills, and socio-emotional skills. Amidst an era of great interest in students’ social and emotional well-being, a range of skills and dispositions have been proposed in the literature on competency-based education as important for student success under this model. We focus on five dimensions reflecting the development of student skills and dispositions. The first is exposure to instructional practices that encourage respect for others (RSPCT; Kallick & Costa, 2009). The second is exposure to instructional practices that encourage students to persevere (PRSVR; Farrington et al., 2012) even when faced with a challenge. We also focus on exposure to instructional practices that encourage students to take responsibility for their learning (RSPNS; Lewis et al., 2014), as well as student demonstration of ownership over their learning (OWN; Le et al., 2014). Finally, we focus on student exposure to instructional practices that encourage peer collaboration (COLL; Kallick & Costa, 2009). In this research, we model latent factors representing student exposure to the dimensions underlying hypothesized elements of competency-based education. Using data collected via a survey designed to capture student feedback about the implementation of competency-based education in their courses, we address three research questions: (1) Can dimensions underlying hypothesized elements of competency-based education be reliably measured?; (2) To the extent that dimensions of competency-based education elements can be reliably measured, how are they associated with one another?; and (3) To what extent do the results provide evidence of construct validity? Education Policy Analysis Archives Vol. 25 No. 24 8 We investigate these questions using confirmatory factor analysis (CFA), which can show whether the observed variables hypothesized to measure an unobserved construct demonstrate a level of shared variance high enough to suggest that those items represent a common underlying factor (i.e., the construct of interest). Per the conceptual framework depicted in Figure 1, we consider three potential ways of representing student exposure to dimensions underlying the hypothesized elements of competency-based education. We begin with a correlated traits model (Model A) in which various dimensions of instructional practice underlying the four key elements of competency-based education are represented by the 10 latent factors depicted in Figure 1. In the correlated traits model, the 10 latent factors are considered to be related to varying degrees but no attempt is made to measure one or more higher-order factors. Figure 1. Conceptual framework for the study. Note: The conceptual framework hypothesizes three potential manifestations of competency-based education implementation, as represented by three latent factor models. In Model A, dimensions underlying hypothesized elements of competency-based education are modeled as a set of interrelated latent factors, although no underlying explanatory structure is imposed on the interrelationships In Model B, it is hypothesized that interrelationships among these dimensions are explained by a set of second-order factors representing the hypothesized elements of competency-based education. Model C imposes a structure by which interrelationships among the 10 individual dimensions are explained by an overall competency-based education second-order factor. In the figure, shaded ellipses depict first-order latent factors, and unshaded ellipses depict second-order latent factors. Each first-order latent factor is measured via observed indicators (see Table 1 and Appendix Table A1), which are not depicted. Investigating Student Exposure to Competency -Based Education 9 To explore whether dimensions underlying the hypothesized essential elements of competency-based education can be represented in a manner consistent with the current description of competency-based education as a distinct program model, we also consider 2 second-order factor models. In Model B (see Figure 1), which is most consistent with current descriptions, the dimensions were grouped as indicators of second-order factors representing student exposure to the hypothesized key elements of competency-based education. This model assumes that the second- order factors (elements of competency-based education) explain the correlations among the first- order factors (dimensions of each element). We also explore an alternative model (Model C; see Figure 1) that assumes the correlations among the 10 first-order factors are explained by one overall competency-based education second-order factor rather than by a set of component elements. Method Participants After securing Institutional Review Board approval, the data for this study were collected in the spring of 2016 at two high schools in the Northeast, including a small rural high school and a midsize suburban high school. Approximately 80% of students in grades 9–12 (n=599) at the two schools completed the survey; missing responses primarily reflected student absence on the date of survey administration (n=83), including a field trip for approximately half of students in grade 12 at one high school. Students in grade 9 made up 33% of the sample, and students in grade 10 made up 24%, followed by students in grades 11 (29%) and 12 (14%). Just over half (52%) of the sample was female. A majority of the sample expected to complete a bachelor’s degree (39%) or above (42%). Survey Development We developed this survey in the context of a research-practice partnership that included teachers, school and district administrators, state department of education staff, and researchers. The work of the partnership, including the development of the survey, was funded by the Institute of Education Sciences through the Regional Educational Laboratory (REL) program and was facilitated by researchers at REL Northeast and Islands, one of 10 RELs across the country. The overall goal of the REL partnerships is to support practitioners’ and policymakers’ use of empirical evidence to inform policy and practice and to improve student outcomes. Partnership research is guided by the needs of these stakeholders who, in the case of this survey, identified the need to capture student feedback about how competency-based education is actually being implemented. Resources developed through the partnership are freely available and no organization or individual profits from their use. The survey used in this research, the Competency-Based Learning Survey for Students (Ryan & Cox, 2016), can found at https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=REL2016165. We used seminal survey development resources, including Fowler (2008), Rea and Parker (2005), and Wright and Marsden (2010), to guide the development of the survey instrument. Survey development relied on close collaboration between the researchers and a six-member advisory committee made up of practitioners from schools implementing competency-based education along with researchers having expertise in competency-based approaches to learning and instruction at the secondary level. Ongoing collaboration with the advisory committee was critical to ensure that the survey items would elicit useful, valid, and reliable information about student exposure to competency-based education policies and practices. Practitioner involvement was especially valuable for ensuring that the item language would be meaningful for students across a variety of state contexts, particularly given variability in terminology surrounding competency-based education. Several steps were taken to establish face (i.e., Does the instrument measure what it is intended to measure?) and content (i.e., Does the instrument tap into the various dimensions of the https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=REL2016165 Education Policy Analysis Archives Vol. 25 No. 24 10 specific constructs of interest?) validity. First, in collaboration with the advisory committee, we developed an initial pool of survey items aimed at providing complete coverage of the hypothesized constructs. The initial item pool included more items than were necessary to ensure complete coverage. The advisors reviewed all draft items, providing feedback on each item in response to a number of specific prompts. This process was used to evaluate the items’ face validity and content validity. We refined the item pool in response to this feedback, and we cycled through this iterative process of review and revision until the item pool covered the constructs completely and constituted a survey that could be administered without overburdening the survey respondents or the school staff that would administer the survey. The refined item pool formed the first draft of the survey. We then administered the initial survey draft to seven students and conducted cognitive interviews with these students. Cognitive interviewing is a method for identifying and correcting problems with surveys that involves administering a draft survey to a respondent while concurrently interviewing him or her to determine if the survey items are eliciting the information that the researcher intends (Beatty & Willis, 2007). During cognitive interviews respondents were asked to think aloud as they completed the survey, thereby making explicit their interpretations of instructions, items, and response choices, as well as their decision points. Information gathered during the cognitive interviews was analyzed and discussed with the advisory committee to improve the clarity, relevance, length, and coverage of survey items. Measures We describe below each of the 10 latent factors we modeled to represent different dimensions of the hypothesized elements of competency-based education. Descriptive statistics for the hypothesized indicators of each latent factor are presented in Table A1. Mastery-based progression. Two latent factors reflect dimensions of mastery-based progression: competency-based pacing (CB PACE; five items), or exposure to instructional practices associated with the opportunity to progress through demonstration of mastery, and competency-specific feedback (CB FEED; three items), or exposure to guidance and feedback from teachers about how student mastery of important content will be determined. Personalization. Dimensions of personalization are reflected through two latent factors. The first latent factor reflects exposure to personalized options for practicing and applying skills (PERS APP; five items), and the second reflects exposure to personalized instructional practices based on the needs of individual students (PERS INSTR; six items). Flexible assessment. The flexible assessment dimension (FLEX; seven items) reflects exposure to multiple and varied opportunities to demonstrate learning. Skills and dispositions. Five underlying dimensions reflect the skills and dispositions hypothesized as important for student success under competency-based education. These dimensions include: exposure to instructional practices that encourage respect for others (RSPCT; three items); exposure to instructional practices that encourage students to persevere even when faced with a challenge (PRSVR; three items); exposure to instructional practices that encourage students to take responsibility for their learning (RSPNS; three items); student demonstration of ownership over learning (OWN; three items); and exposure to instructional practices that encourage peer collaboration (COLL; three items). Analysis We used confirmatory factor analysis (CFA) to examine the fit between each hypothesized model (see Figure 1) and the data. Analyses were conducted using Mplus statistical software, Version 7.0 (Muthén & Muthén, 2016). An advantage of CFA is that the analyses produce a number of fit Investigating Student Exposure to Competency -Based Education 11 statistics, making it possible to evaluate how well the hypothesized model fits the observed data. We examined fit indices including the chi-squared statistic, the Comparative Fit Index (CFI) and the Root Mean Square Error of Approximation (RMSEA). While small misspecifications in a model often lead to inflation of the chi-squared statistic even when the model is correctly specified (Curran, West, & Finch, 1996), the CFI accounts for sample size, with a value above 0.90 indicating a good fit between the model and the data (Hu & Bentler, 1999). The RMSEA indicates how well the model would fit the population covariance matrix given unknown but optimally chosen parameters; the suggested upper-limit cutoff is 0.06 (Hu & Bentler, 1999). The robust weighted least squares (WLSMV) estimator was used to estimate the models described here. The WLSMV estimator is most appropriate in this research given that almost all observed variables are categorical in nature. This estimator uses the probit link and provides standard errors and a chi-squared test statistic that are robust to non-normality. Results Dimensional Structure The initial correlated traits CFA model (Model A [original]; see Figure 1) included 10 latent factors, as described above, each as measured by its originally hypothesized indicators, per Table 1. Results (Table 1) suggested an adequate fit between this model and the data (χ2(734)=2181.50; CFI=.92; RMSEA=.06). However, a review of factor loadings revealed three (CB PACE4, PERS INSTR5, and FLEX2) that fell at or below the recommended .40 threshold (Stevens, 2009), suggesting that these indicators should be removed from the model. We also reviewed model modification indices, which indicate how the addition or removal of specific parameters would improve model fit. Because modification indices are purely data-based and not informed by theory, over-reliance on modification indices can quickly become an exploratory journey, increasing the risk of Type I error. Thus, no more than a few modifications should be made to an initial confirmatory model, provided that a clear theoretical justification exists (McDonald & Ho, 2002). The modification indices for the original model included many potential changes. However, we identified just one change as consistent with the literature on competency-based education. The change involved modeling an original indicator (CB PACE1, “I know what I need to do to show my teachers that I am making progress on each competency”) of the competency-based pacing latent factor as an indicator of the student demonstration of ownership over learning factor instead. This change is consistent with the suggestion that students must be prepared to guide their own learning under competency-based education (Le et al., 2014; Lewis et al., 2014). In total, four changes were made to the original model: three indicators were dropped from the model, and CB PACE1 became a fourth indicator of student demonstration of ownership over learning (OWN4; see Table 1). This revision to the original model (Table 1, Model A [revised]), which reflects the changes described above (changes in bold font in Table 1), fit the data well (χ2(618)=1429.61; CFI=.95; RMSEA=.05). All factor loadings in the revised model were statistically significant at p≤.001. Correlations among the 10 latent factors are provided in Table 2. Our next step was to estimate a second-order factor model, a model in which the first-order latent factors in turn become indicators of a second-order factor. The initial second-order factor model (Model B; see Figure 1) is most consistent with current descriptions of competency-based education as comprising four essential elements. This model included three second-order factors, representing three of the four key elements of competency-based education: mastery-based progression, as indicated by the first-order factors CP PACE and CB FEED; personalization, as indicated by the first-order factors PERS APP and PERS INSTR; and skills and dispositions, as indicated by the first-order factors RSPCT, PRSVR, RSPNS, OWN, and COLL. We did not model the fourth key element of competency-based Education Policy Analysis Archives Vol. 25 No. 24 12 education, flexible assessment, as a second-order factor given that this element was reflected through just one dimension (FLEX). Table 1 Standardized Loadings on Latent Factors Reflecting the Constructs of Interest and Model Fit Statistics Model A (original) β Model A (revised) β Model B β Model C β CB PACE by CB PACE1 .57 OWN4 .57 .57 .57 CB PACE2 .67 .66 .66 .66 CB PACE3 .63 .62 .62 .62 CB PACE4 -.22 omit -- -- -- CB PACE5 .72 .71 .71 .71 CB FEED by CB FEED1 .81 .81 .82 .82 CB FEED2 .79 .79 .78 .79 CB FEED3 .73 .72 .73 .72 PERS APP by PERS APP1 .56 .56 .55 .54 PERS APP2 .97 .95 .96 .95 PERS APP3 .76 .62 .62 .62 PERS APP4 .72 .59 .58 .58 PERS APP5 .94 .98 .99 .99 PERS INSTR by PERS INSTR1 .63 .63 .63 .63 PERS INSTR2 .69 .69 .69 .69 PERS INSTR3 .74 .74 .74 .74 PERS INSTR4 .71 .70 .70 .70 PERS INSTR5 .40 omit -- -- -- PERS INSTR6 .60 .59 .59 .59 FLEX by FLEX1 .63 .61 .61 .61 FLEX2 -.29 omit -- -- -- FLEX3 .63 .62 .62 .62 FLEX4 .49 .42 .42 .42 FLEX5 .55 .48 .47 .47 FLEX6 .61 .59 .60 .60 FLEX7 .64 .63 .63 .63 RSPCT by RSPCT1 .82 .83 .82 .82 RSPCT2 .92 .92 .93 .93 RSPCT3 .89 .89 .89 .89 Investigating Student Exposure to Competency -Based Education 13 Table 1 (Cont’d.) Standardized Loadings on Latent Factors Reflecting the Constructs of Interest and Model Fit Statistics Model A (original) β Model A (revised) β Model B β Model C β PRSVR by PRSVR1 .88 .89 .89 .89 PRSVR2 .87 .87 .87 .87 PRSVR3 .82 .82 .82 .82 RSPNS by RSPNS1 .85 .85 .84 .84 RSPNS2 .86 .86 .86 .85 RSPNS3 .49 .49 .49 .49 OWN by OWN1 .76 .75 .75 .75 OWN2 .67 .66 .65 .66 OWN3 .76 .75 .75 .75 COLL by COLL1 .92 .92 .92 .92 COLL2 .79 .79 .79 .79 COLL3 .80 .80 .80 .80 Second-order factors MB PROG by CB EDUC by CB PACE .88 CB PACE .82 CB FEED .71 CB FEED .67 PERS APP .53 PERSON by PERS INSTR .92 PERS APP .55 FLEX .76 PERS INSTR .97 RSPCT .92 PRSVR .81 SKILLS by RSPNS .91 RSPCT .93 OWN .73 PRSVR .83 COLL .93 RSPNS .92 OWN .75 COLL .94 Model fit Χ2(df) 2181.5(734) 1429.61(618) 1668.91(648) 1770.28(653) RMSEA (90% CI) .060 (.057–.063) .049 (.046–.052) .054 (.051–.057) .056 (.051–.057) CFI .92 .95 .94 .94 Χ2diff 227.34(30), p=.00 290.21(35), p=.00 Note: Source is the Competency-Based Learning Survey for Students, n=599. Bolded font indicates a modification made to Model A (original), resulting in Model A (revised). Education Policy Analysis Archives Vol. 25 No. 24 14 Model B also fit the data well (χ2(648)=1668.91; CFI=.94; RMSEA=.05). Given that Model B is nested within Model A, we used a chi-squared difference test to determine whether Model B fit the data as well as Model A. The result was significant (χ2diff=227.34, p=.00; see Table 1), suggesting that although both models fit the data well, Model A is preferred over Model B. We also estimated an alternative second-order factor model (Model C; see Figure 1) in which all 10 first-order latent factors were modeled as indicators of one underlying second-order factor, which we might think of as representing competency-based education. Again, this model fit the data well (χ2(653)=1770.28; CFI=.94; RMSEA=.06). Because Model C is also nested within Model A, we again used a chi-squared difference test to determine whether Model C fit the data as well as Model A. Here too, the result was significant (χ2diff=290.21, p=.00; see Table 1), indicating that Model A is also preferred over Model C. Table 2 Latent Factor Correlations and Reliabilities for Model A (Revised) 1 ρ=.75 2 ρ=.75 3 ρ=.83 4 ρ=.83 5 ρ=.85 6 ρ=.75 7 ρ=.75 8 ρ=.74 9 ρ=.81 10 ρ=.75 1 CB PACE 2 CB FEED .63 3 PERS APP .56 .31 4 PERS INSTR .84 .63 .53 5 FLEX .74 .50 .63 .80 6 RSPCT .45 .47 .20 .58 .36 7 PRSVR .70 .54 .43 .85 .66 .80 8 RSPNS .69 .63 .48 .84 .70 .64 .81 9 OWN .79 .63 .34 .73 .54 .59 .75 .79 10 COLL .64 .60 .43 .81 .68 .75 .85 .88 .69 Note: Source is the Competency-Based Learning Survey for Students, n=599. All correlations significant at p≤.001. In sum, the CFA results suggest that the correlated traits model (Model A), which included 10 latent factors reflecting different dimensions underlying each of the four elements described as essential to competency-based education, fit the data well. Further, all 10 latent factors were positively and significantly correlated. The second-order factor models (Models B and C), both of which are more consistent with a conceptualization of competency-based education as a bounded, distinct model for secondary schooling as opposed to a more arbitrary mix of various dimensions of instructional practice, also fit the data well. However, neither second-order factor model fit the data as well as the correlated traits model (Model A). We return to the implications of these results in the discussion. Construct Reliability To assess the internal consistency of each of the latent factors, we used the reliability coefficient (rho [ρ]) developed by Raykov (2001) and recommended (Brown, 2006; Raykov, 2001) for use in CFA analyses, particularly when factor indicators are categorical. Construct reliabilities above .70 are considered to reflect high internal consistency (Kline, 2000). All construct reliabilities in the revised Model A ranged from .74–.85 (see Table 2). Investigating Student Exposure to Competency -Based Education 15 Convergent and Divergent Construct Validity Within a construct validation framework, convergent validity reflects the notion that measures we would expect to be related are, in fact, related. In contrast, divergent validity reflects the idea that measures we would not expect to be related share only a weak or nonexistent relationship. Perhaps the most robust evidence of convergent validity is obtained by examining how strongly the measures under study are related to previously established measures of the constructs of interest. Given that there are no existing reliable and valid measures of student exposure to competency-based education, we were not able to examine this kind of evidence. Although we were not able to examine correlations among measures of the same constructs, we did consider the direction and magnitude of the correlations among the latent factors we modeled as another way to explore convergent and divergent validity. Almost all correlations among the 10 latent factors were positive, moderate or strong, and statistically significant (see Table 2). To explore divergent validity, we examined the correlations between each of the 10 latent factors and two additional survey items (see Table 3): “Students in my courses all work on the same assignment at the same time” and “My teachers spend most of class time giving a lecture or presentation to the whole class.” These items were intended to capture the extent to which students reported exposure to practices associated with what is often described as a traditional approach to instruction, or one that is teacher-directed and assumes that students will all move through the same content at about the same pace (Wolfe, Steinberg, & Hoffman, 2013). We anticipated that as student exposure to the various dimensions of competency-based education increased, their exposure to more traditional instructional practices would decrease. Generally consistent with this expectation, all but one of the latent factors shared a negative or nonexistent relationship with both of these survey items. In one exception, the latent factor RSPCT shared a weak positive relationship with the item “Students in my courses all work on the same assignment at the same time.” Table 3 Correlations Between Latent Factors from Model A (Revised) and Two Survey Items Representing a Traditional Approach to Instruction CB PACE CB FEED PERS APP PERS INSTR FLEX RSPCT PRSVR RSPNS OWN COLL Item 1 -.18*** -.01 -.11*** -.05 -.14*** .10* -.09* -.12** .04 -.06 Item 2 -.15*** -.11** -.06 -.11*** -.12*** -.06 -.19*** -.08 -.06 -.10* Note: Source is the Competency-Based Learning Survey for Students, n=599. Item 1 wording is “Students in my courses all work on the same assignment at the same time” and Item 2 wording is “My teachers spend most of class time giving a lecture or presentation to the whole class” (response options for both are 0=Never, 1=Seldom, 2=Sometimes, 3=Often, 4=Always). p≤.05*, p≤.01**, p≤.001*** The positive and moderate to strong correlations among most of the latent factors are potentially indicative of convergent validity, or the idea that measures that should be related are actually related. The negative correlations these factors shared with items representing a more traditional approach to teaching and learning are suggestive of divergent validity, or the idea that measures that should not be related are indeed not related. However, we note that these results are exploratory and somewhat difficult to interpret without additional research. Education Policy Analysis Archives Vol. 25 No. 24 16 Construct Generalizability Another aspect of construct validity is generalizability (Messick, 1995). Multigroup CFA can be used to test whether the constructs of interest are generalizable across populations (Sass, 2011) or, in other words, whether the measures under study are invariant across populations. Although this step is often overlooked, it is important to establish measurement invariance (i.e., the same constructs can be measured in the same way across populations) before attempting to draw comparisons across different populations—for example, different schools—on the constructs of interest (Sass, 2011). This research used data from two different high schools, one that was just beginning to implement competency-based learning and one that had been competency-based for several years, providing an opportunity to test whether our measures were invariant across these two groups using multigroup CFA. Muthén and Asparouhov (2002) assert that in order to make meaningful comparisons of factor distributions across groups, a majority of the variables serving as latent factor indicators should have both threshold and loading invariance. The criteria suggested by Muthén and Asparouhov were satisfied (see Appendix for details), indicating that the measures in Model A (revised) were invariant across the two schools. Discussion Interest in competency-based and other student-centered reforms at the secondary level continues to grow, but research lags behind (Freeland, 2014; Haynes et al., 2016). As one consequence, changes in educational policy and practice associated with the shift to competency- based education often lack empirical evidence. One obstacle for both researchers and practitioners interested in better understanding student experiences and outcomes under competency-based education has been the lack of tools that could be used to measure and study variation in implementation. Through the testing of a survey that captures student-reported exposure to hypothesized essential elements of competency-based education, this study begins to address the need for measurement instruments that will allow researchers to investigate how this ambitious, if somewhat nebulous, reform is actually being implemented, and with what consequence for student outcomes. Overall, results from the confirmatory factor analyses indicated that the correlated traits model (Model A, revised) provided the best fit with the data. In this model, various dimensions of the four hypothesized elements of competency-based education were represented by 10 latent factors, all of which demonstrated high internal consistency. The model allowed for correlations among these 10 latent factors, but we did not group them into one or more higher-order factors. We also examined two second-order factor models. The first (Model B) modeled the first-order latent factors as indicators of second-order factors representing elements of competency-based education; the other (Model C) modeled all 10 first-order latent factors as indicators of a single competency- based education second-order factor. The second-order models reflected two ways of representing competency-based education as a bounded, distinct model for secondary schooling, as opposed to a more arbitrary mix of various dimensions of instructional practice (Model A). Although both second-order factors models fit the data well, results from chi-squared difference tests indicated that neither model provided a better fit to the data than Model A. Conceptually, these results indicate that the survey reliably captures distinct dimensions of instructional practice associated with competency-based education. But the results also indicate that these dimensions may not cluster together in a manner consistent with the description of competency-based education as a distinct model that can be distinguished from other approaches to secondary education. We return to this point below in discussing directions for both future research and policy. Investigating Student Exposure to Competency -Based Education 17 With respect to construct validity, we worked closely with a group of six advisors that included practitioners and researchers with expertise in competency-based education to establish both face and content validity of the survey instrument. We also considered the extent to which our results provided preliminary evidence supporting convergent and divergent validity of the constructs operationalized in this research. Without existing reliable and valid measures of student exposure to elements of competency-based education, we were not able to examine how our measures correlated with other established measures, which would have provided the most robust evidence about convergent validity. We did, however, examine the direction and magnitude of correlations among the latent factors we modeled, and almost all of the correlations among the 10 latent factors were positive, moderate or strong, and statistically significant. Nonetheless, we emphasize that these results are exploratory. It is difficult to draw firm conclusions about construct validity, particularly given the limited empirical basis upon which to base expectations about the distinctive elements of competency-based education—including the component dimensions of those elements—and the relationships among them. As we discuss below, future research will be necessary to more comprehensively evaluate convergent and divergent validity and other aspects of construct validity. For both practitioners and researchers, results from this research suggest that the survey offers a way to reliably measure and study variation in the implementation of practices commonly associated with competency-based education. Importantly, the survey provides a way to capture implementation from the student perspective. In the remaining sections, we outline implications for practice, for future research that can provide needed empirical evidence about a school reform model that is increasingly making its way into state and district policy, and, finally, we discuss implications for policy itself. Implications for Practice Given that student reports about their classroom experiences appear to be a particularly reliable source of information about instructional practice (Kane & Staiger, 2012), student-reported exposure to competency-based education may be especially useful as schools seek to understand and refine instruction. Results from student perception surveys such as this one can also provide feedback for the improvement of learning and instruction (Ferguson, 2010), and can be used to inform data-based decision making (Lai & Schildkamp, 2013). For example, where students report low exposure to particular dimensions of instructional practice associated with competency-based education, this could be useful feedback for teachers and schools as they work toward more consistent implementation of related practices. It may also be valuable for practitioners and policymakers to consider differences among particular student subgroups in their reported exposure to dimensions of competency-based education. Although competency-based education has been described as uniquely capable of addressing the academic and social-emotional needs of students who struggle under traditional systems of schooling, whether this is the case remains an open question (Lewis et al., 2014). Further, either collectively or in collaboration with researchers, practitioners might also triangulate other sources of evidence about instructional practice, for example from classroom observations or staff surveys, with student-reported experiences. This type of analysis would provide better information about adherence, or the extent to which hypothesized elements of competency- based education are implemented as prescribed in the literature at both the classroom and school levels. Such an analysis would also provide information about the extent to which students’ perceptions of instructional practices under competency-based education parallel the perceptions of their teachers. Instances of substantial misalignment between student and teacher perceptions about what is happening in the classroom would provide useful opportunities for school staff to reflect Education Policy Analysis Archives Vol. 25 No. 24 18 together on the sources of this disconnect. Potential sources might include, for example, student and/or teacher over- or underreporting of competency-based instructional practices or nuances in the wording of student survey items. Implications for Research With respect to future research, a next step should be to administer this survey across more schools both to assess reliability within schools and to measure variation in how students experience competency-based education among schools and districts. As noted above, subsequent survey administration could be carried out in conjunction with qualitative research, including both observations and interviews in schools, to document the existence of certain practices and compare these data with student feedback. Combining student survey administration with qualitative data collection could also prove valuable for evaluating how much variation in student responses derives from measurement issues as opposed to real variation in their classroom and school experiences. There is also a need for continued research investigating the program differentiation aspect of implementation fidelity, or the degree to which the essential elements of competency-based education are distinguishable from each other and from other programs. We considered program differentiation in this research by comparing how well three potential ways of modeling competency-based education implementation fit the observed data. The initial second-order model conceptualized the individual dimensions as reflecting underlying elements of competency-based education, and the other second-order model conceptualized these dimensions as reflecting a singular underlying “competency-based education” construct. While both second-order models fit the data well, neither fit the data as well as the initial model in which the dimensions of each element were modeled as correlated but separate. On the one hand, it is possible that we have not fully captured the elements of competency-based education through the dimensions we measure and, were we able to do so, we would find that the separate dimensions (i.e., first-order factors) actually are better modeled as representing one or more of the hypothesized elements (i.e., second-order factors). On the other hand, the results may instead suggest that these dimensions do not cluster together in a manner that supports the conceptualization of competency-based education as a distinct model with four constituent elements. In other words, competency-based education may actually unfold as a more arbitrary mix of instructional practices that, at the level of implementation, are combined in ways that can’t necessarily be distinguished from other models for secondary education—a possibility consistent with research indicating that whole-school models often lack the specificity required to achieve, and to measurably detect, high implementation fidelity (Desimone, 2002; O’Donnell, 2008). Based on these findings, future research on program differentiation might use this survey with students from a larger sample of schools to study whether elements described as essential to competency-based education actually constitute a coherent and distinct model for secondary education. This could include studies that administer the survey both with students in schools that have adopted a competency-based model and with students in schools that have not adopted this approach. Survey administration with both types of students would provide evidence about the extent to which the survey can be used not only to distinguish among schools demonstrating different levels of implementation fidelity, but also to distinguish between schools that have and have not adopted a competency-based approach. Student survey results could also be compared with information collected through other modes (e.g., teacher surveys, classroom observations) in order to examine whether different sources of information about program differentiation, and about other aspects of implementation fidelity, provide similar or different results. Investigating Student Exposure to Competency -Based Education 19 Future research should also continue to explore the validity of constructs reflecting student exposure to hypothesized elements of competency-based education, including the validity of constructs that attempt to combine these elements into one overall measure of competency-based education implementation. In the absence of previously established valid and reliable measures of student exposure to competency-based education, our efforts to assess construct validity were limited to examining the direction and magnitude of correlations among the first-order latent factors as a way to consider convergent validity. Almost all correlations among the 10 latent factors, which reflect different dimensions of the hypothesized competency-based education elements, were positive, moderate or strong, and statistically significant. Yet, we emphasize that these results are exploratory and difficult to interpret without additional research. A correlation between two measures provides a somewhat ambiguous signal about convergent and divergent validity given the potential influences of both trait and method variance (Furr & Bacharach, 2013). A strong correlation may indicate that two measures share trait variance, meaning that the constructs that they are intended to measure share something in common. However, a strong correlation could also indicate that the two measures share method variance, indicating that they are correlated primarily because they use the same method of measurement. Future research using multi-trait, multi-method techniques (Widaman, 1985) could investigate whether factors defined by different measures (e.g., online survey, paper/pencil survey) of the same trait (e.g., dimensions of competency-based education) support the validity of the measures reported here, and also whether factors defined by different traits but measured with the same method argue for method effects. Ultimately, construct validation involves a perpetual process of refining both theory and measurement (Westen & Rosenthal, 2003). A more comprehensive analysis of construct validity with respect to competency-based education will require continued research, including iterative refinement of both theory and measurement. This should include, for example, collecting longitudinal data to facilitate the investigation of predictive validity, which will also require specificity about the outcomes we should expect to be especially impacted by exposure to competency-based education. At the secondary level, competency-based education is typically proposed as a reform to improve students’ college and career readiness—but what does this mean, specifically? For example, is competency-based education intended to improve academic achievement as measured by standardized assessments, performance on college entrance exams, or other measures of college and career readiness, including social and emotional outcomes such as engagement and intrinsic motivation? Or is it intended to impact all of the above? Finally, this survey, which reliably captures student exposure to various dimensions of instructional practice associated with competency-based education, might be used to more carefully investigate the relationship between implementation and student outcomes. A handful of studies in recent years have attempted to estimate the impact of various student-centered approaches, including competency-based, personalized, student-centered, and “deeper” learning approaches (Haynes et al., 2016; Steele et al., 2014; Zeiser, Taylor, Rickles, Garet, & Segeritz, 2014). Yet these studies have yielded limited knowledge about the effects of competency-based education due to considerable variation in its implementation. For example, in one recent study (Haynes et al., 2016), designed to compare outcomes among students in schools identified a priori as competency-based or non-competency-based, the researchers found this to be a problematic distinction. Instructional practices commonly associated with competency-based education in the literature varied substantially within competency-based schools; moreover, these same practices were also evident, and similarly variable, in schools identified as non-competency-based. The researchers concluded that competency-based education cannot be characterized as a present/non-present phenomenon. Rather than treating the implementation of competency-based education as a dichotomous Education Policy Analysis Archives Vol. 25 No. 24 20 designation, future research might use the survey described in this study to investigate how the nature and extent of student exposure to competency-based education impacts student outcomes. Implications for Policy Although policy prescriptions are best informed by a cumulative body of evidence, rather than by any single study (McDonnell, 2000), the results from this research, while preliminary, suggest several directions for policy. First, we see a need for policy at national, state, and local levels to establish greater precision in the language used to define competency-based education and its key elements. As Hess (2016) notes, the school reform landscape is littered with words that have been stripped of meaning, which is true, to varying degrees, about many of the terms used to define elements of competency-based education, and to define student-centered reforms more generally. Ultimately, the lack of clarity in what such placeholders mean avoids complicated questions (Hess, 2016), including questions about how to coherently define and successfully implement changes in policy and practice, and about how to measure and evaluate these changes and their implications for students. As noted earlier, imprecision in the language used to describe and define competency-based education and related approaches is accompanied by wide variability within and across schools and districts in how competency-based policies and practices are conceived and implemented. Thus, it is perhaps not surprising that our results did not necessarily support the conceptualization of competency-based education as a distinct model with four constituent elements, at least as these elements have currently been defined. Second, the results, which raise numerous questions for future research, highlight the fact that greater empirical evidence about the nature and effectiveness of competency-based education is needed to inform both policy and advocacy at both the national and state levels. Most U.S. states have now authorized at least some flexibility for local education agencies to link the completion of academic credit to demonstration of proficiency rather than simply to seat time (Carnegie Foundation for the Advancement of Teaching, 2014). Yet, there remains little available empirical evidence about the effect of competency-based education on student outcomes (Freeland, 2014; Haynes et al, 2016; Le et al., 2014). Moving forward, policies at all levels of the education system related to competency-based education would benefit from a stronger grounding in evidence from a variety of sources and using a range of methods. Related, proponents of competency-based and other student-centered educational reforms can best support sound policy by engaging in evidence- driven advocacy—even, or perhaps especially, when the evidence challenges prevailing assumptions about how well these reforms work (McCarthy, 2016). Third, the results draw attention to the fact that input from students is an important source of information as state and especially district policymakers seek to understand how competency- based education is being implemented in the classroom, and student voices can provide guidance about where changes in policy might be necessary. Recent research about the use of student perception surveys to gather data about students’ classroom experiences has demonstrated that student surveys may provide particularly consistent feedback about what is happening in the classroom (Balch, 2012; Ferguson, 2012; Kane & Staiger, 2012), offering an important source of information about implementation. While much of this previous research has been situated in the literature on teacher evaluation, we believe the survey studied in this research is best used not as an accountability tool, but rather as a source of information about where changes in policy may be needed to clarify the defining features of competency-based education and to support implementation. For example, during our discussions with each of the participating districts about their survey results, leaders and teachers identified specific patterns that, from their perspective, Investigating Student Exposure to Competency -Based Education 21 indicated a need for both changes in policy and changes in how existing policies were communicated to teachers and students. Finally, we remind the reader that policy changes, while necessary, are almost certainly insufficient to achieve successful implementation of competency-based education. As Spady and Mitchell (1977) wisely advised several decades ago, doing so will also require attending to the social order of the school, for example by developing a sense of ownership for the proposed implementation among students, staff, and parents, by providing rewards to teachers and students for risking new approaches to their respective roles, and by encouraging greater authenticity in the relationships and goals of students and teachers. Conclusion Recent evidence from a nationally representative sample of high school graduates indicates that almost half of U.S. high school students complete neither a college- or career-ready course of study (Bromberg & Theokas, 2016). Policymakers, employers, and the public voice concerns that America’s youth are graduating without the skills and knowledge needed for postsecondary success. In response, many states and districts are turning to competency-based education policies and other so-called student-centered reforms as a strategy to ensure student readiness for college-level work and the workforce. As competency-based and other student-centered reforms increasingly make their way into policy, empirical research lags behind. Research must keep pace with changes in policy and practice by providing evidence about both the implementation and the effects of competency-based reforms. This should include research that builds upon the current study to investigate the link between student-reported exposure to competency-based education and student outcomes. References Alliance for Excellent Education. (2012). What about deeper learning? Washington, DC: Author. Balch, R. T. (2012). The validation of a student survey on teacher practice. Nashville, TN: Vanderbilt University. Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287-311. https://doi.org/10.1093/poq/nfm006 Berman, P., & McLaughlin, M. W. (1976). Implementation of educational innovation. Educational Forum, 40(3), 344-370. https://doi.org/10.1080/00131727609336469 Bill & Melinda Gates Foundation. (2015). Personalized learning: What is it? Seattle, WA: Author. Bromberg, M., & Theokas, C. (2016) Meandering toward graduation: Transcript outcomes of high school graduates. Washington, DC: Education Trust. Brown, M. (1994). An introduction to the discourse on competency-based training (CBT). In F. Patrick, D. Tate, & S. Wright (Eds.), A collection of readings related to competency-based training (pp. 1-17). Geelong, Victoria: Deakin University. Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: Guilford. Carnegie Foundation for the Advancement of Teaching. (2014). 50-state scan of course credit policies [working draft]. Stanford, CA: Author. Competency Works. (2014). What is competency education? Vienna, VA: International Association for K-12 Online Learning. https://doi.org/10.1093/poq/nfm006 https://doi.org/10.1080/00131727609336469 Education Policy Analysis Archives Vol. 25 No. 24 22 Curran, P. J., West, S. G., & Finch, J. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological methods, 1(1), 16. https://doi.org/10.1037/1082-989X.1.1.16 Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23-45. https://doi.org/10.1016/S0272-7358(97)00043-3 Desimone, L. M. (2002). How can comprehensive school reform models be successfully implemented? Review of Educational Research, 72(3), 433-479. https://doi.org/10.3102/00346543072003433 Dewey, J. (1938). Education and experience. Eastford, CT: Martino Fine Books. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3-4), 327-350. https://doi.org/10.1007/s10464-008-9165-0 Dusenbury, L., Branningan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237-256. https://doi.org/10.1093/her/18.2.237 Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance—A critical literature review. Chicago, IL: Consortium on Chicago School Research. Ferguson, R. F. (2010). Student perceptions of teaching effectiveness. Cambridge, MA: National Center for Teacher Effectiveness and the Achievement Gap at Harvard University. Ferguson, R. F. (2012). Can student surveys measure teaching quality? Phi Delta Kappan, 94(3), 24-28. ttps://doi.org/10.1177/003172171209400306 Ford, K. (2014). Competency-based education: History, opportunities, and challenges. Adelphi, MD: Center for Innovation in Learning and Student Success at University of Maryland. Fowler, F. J. (2008). Survey research methods (4th ed.). Thousand Oaks, CA: Sage. Freeland, J. F. (2014, August 28). Why is competency-based education so hard to study? [Web log.] Retrieved from http://educationnext.org/competency-based-education-hard-study/ Furr, R. M., & Bacharach, V. R. (2013) Psychometrics: An introduction. Los Angeles, CA: SAGE Publications. Haynes, E., Zeiser, K., Surr, W., Hauser, A., Clymer, L., Walston, J., Bitter, C., & Yang, R. (2016). Looking under the hood of competency-based education: The relationship between competency-based education practices and students’ learning skills, behaviors, and dispositions. Quincy, MA: Nellie Mae Education Foundation. Hess, F. (2016, December 29). Seven reflections on school reform in 2017. [Web log.] Retrieved from http://blogs.edweek.org/edweek/rick_hess_straight_up/2016/12/seven_reflections_on_sc hool_reform_in_2017.html Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: A multidisciplinary journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118 Jobs For the Future. (2014). Putting students at the center: A reference guide. Boston, MA: Author. Kallick, B. O., & Costa, A. L. (2009). Habits of mind across the curriculum: Practical and creative strategies for teachers. Alexandria, VA: Association for Supervision and Curriculum Development. https://doi.org/10.1037/1082-989X.1.1.16 https://doi.org/10.1016/S0272-7358(97)00043-3 https://doi.org/10.3102/00346543072003433 https://doi.org/10.1007/s10464-008-9165-0 https://doi.org/10.1093/her/18.2.237 https://doi.org/10.1177/003172171209400306 http://educationnext.org/competency-based-education-hard-study/ http://blogs.edweek.org/edweek/rick_hess_straight_up/2016/12/seven_reflections_on_school_reform_in_2017.html http://blogs.edweek.org/edweek/rick_hess_straight_up/2016/12/seven_reflections_on_school_reform_in_2017.html https://doi.org/10.1080/10705519909540118 Investigating Student Exposure to Competency -Based Education 23 Kane, T. J., & Staiger, D. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Research Paper. MET Project. Seattle, WA: Bill & Melinda Gates Foundation. Kline, P. (2000). The handbook of psychological testing (2nd ed.). London: Routledge. Lai, M. K., & Schildkamp, K. (2013). Data-based decision making: An overview. In K. Schildkamp, M. K. Lai & L. Earl (Eds.), Data-based decision making in education (pp. 9-21). Dordrecht, Netherlands: Springer. https://doi.org/10.1007/978-94-007-4816-3_2 Le, C., Wolfe, R. E., & Steinberg, A. (2014). The past and the promise: Today's competency education movement. Boston, MA: Jobs For the Future. Lewis, M. W., Eden, R., Garber, C., Rudnick, M., Santibañez, L., & Tsai, T. (2014). Equity in competency education: Realizing the potential, overcoming the obstacles. Boston, MA: Jobs for the Future. Marion, S., & Leather, P. (2015). Assessment and accountability to support meaningful learning. Education Policy Analysis Archives, 23(9), 1-19. http://dx.doi.org/10.14507/epaa.v23.1984 McCarthy, P. T. (2016, January 6). Evidence at the crossroads pt. 7: Using evidence to do the most good, even when it reveals an “inconvenient truth.” [Web log.] Retrieved from http://wtgrantfoundation.org/evidence-at-the-crossroads-pt-7-using-evidence-to-do-the- most-good-even-when-it-reveals-an-inconvenient-truth McDonald, R. P., & Ho, M. H. R. (2002). Principles and practice in reporting structural equation analyses. Psychological methods, 7(1), 64-82. https://doi.org/10.1037/1082-989X.7.1.64 McDonnell, L. M. (2000). From research findings to policy advice: A guide for AERA members. Washington, DC: American Educational Research Association. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741 Muthén, B. O., & Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in mplus. (Web Note No. 4). Los Angeles: Mplus. Muthén, L. K., & Muthén, B. O. (2016). Mplus user’s guide. Los Angeles, CA: Muthén & Muthén. National Center on Time & Learning. (2011). Learning time in America: Trends to reform the American school calendar. Boston, MA: Author. O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in k-12 curriculum intervention research. Review of Educational Research, 78(1), 33-84. https://doi.org/10.3102/0034654307313793 Patrick, S., & Sturgis, C. (2013). Necessary for success: Building mastery of world class skills. A state policymaker’s guide to competency education. Vienna, VA: International Association for K-12 Online Learning. Raykov, T. (2001). Estimation of congeneric scale reliability via covariance structure analysis with nonlinear constraints. British Journal of Mathematical and Statistical Psychology, 54, 315-323. https://doi.org/10.1348/000711001159582 Rea, L. M., & Parker, R. A. (2005). Designing and conducting survey research: A comprehensive guide. San Francisco: Jossey-Bass. Ryan, S., & Cox, J. D. (2016). Guide to the competency-based learning survey for students (REL 2016–165). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northeast & Islands. https://doi.org/10.1007/978-94-007-4816-3_2 http://dx.doi.org/10.14507/epaa.v23.1984 http://wtgrantfoundation.org/evidence-at-the-crossroads-pt-7-using-evidence-to-do-the-most-good-even-when-it-reveals-an-inconvenient-truth http://wtgrantfoundation.org/evidence-at-the-crossroads-pt-7-using-evidence-to-do-the-most-good-even-when-it-reveals-an-inconvenient-truth https://doi.org/10.1037/1082-989X.7.1.64 https://doi.org/10.1037/0003-066X.50.9.741 https://doi.org/10.3102/0034654307313793 https://doi.org/10.1348/000711001159582 Education Policy Analysis Archives Vol. 25 No. 24 24 Sass, D. A. (2011). Testing measurement invariance and comparing latent factor means within a confirmatory factor analysis framework. Journal of Psychoeducational Assessment, 29(4), 347-363. https://doi.org/10.1177/0734282911406661 Scheopner-Torres, A. S., Brett, J., & Cox, J. D. (2015). Competency-based learning: Definitions, policies, and implementation. Waltham, MA: Regional Educational Laboratory Northeast & Islands at Education Development Center. Spady, W. G. (1977). Competency-based education: A bandwagon in search of a definition. Educational Researcher, 6(1), 9-14. https://doi.org/10.3102/0013189X006001009 Spady, W. G., & Mitchell, D. E. (1977). Competency-based education: Organizational issues and implications. Educational Researcher, 6(2), 9-15. https://doi.org/10.3102/0013189X006002009 Steele, J. L., Lewis, M. W., Santibañez, L., Faxon-Mills, S., Rudnick, M., Stecher, B. M., & Hamilton, L. S. (2014). Competency-based education in three pilot programs: Examining implementation and outcomes. Santa Monica, CA: RAND Corporation. Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). New York: Routledge. Stump, E., & Silvernail, D. (2015). Proficiency-based diploma systems in Maine: Implementing district-level high school graduation policies. Portland, ME: University of Southern Maine, Center for Education Policy, Applied Research and Evaluation. Sturgis, C. (2014). Progress and proficiency: Redesigning grading for competency education. Vienna, VA: International Association for K-12 Online Learning. Sturgis, C. (2016). Reaching the tipping point: Insights on advancing competency education in New England. Vienna, VA: International Association for K-12 Online Learning. Sturgis, C., & Patrick, S. (2010). When success is the only option: Designing competency-based pathways for next generation learning. Vienna, VA: International Association for K-12 Online Learning. United States Department of Education. (2011, May). Increasing educational productivity: Innovative approaches and best practices. [Web log.] Retrieved from http://blog.ed.gov/2011/05/increasing-educational-productivity-innovative-approaches- best-practices/ Weise, M. (2014, November). Is there any difference between competency-based education and mastery-based learning? [Web log post.] Retrieved from http://www.competencyworks.org/higher- education-2/is-there-any-difference-between-competency-based-education-and-mastery- based-learning/ Westen, D., & Rosenthal, R. (2003). Quantifying construct validity: Two simple measures. Journal of personality and social psychology, 84(3), 608. https://doi.org/10.1037/0022-3514.84.3.608 Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait- multimethod data. Applied Psychological Measurement, 9(1), 1-26. https://doi.org/10.1177/014662168500900101 William & Flora Hewlett Foundation. (2014). What is deeper learning? Menlo Park, CA: Author. Wolfe, R. E., Steinberg, A., & Hoffman, N. (2013). Anytime, anywhere: Student-centered learning for schools and teachers. Cambridge, MA: Harvard Education Press. Wright, P. V., & Marsden, J. D. (Eds.) (2010). Handbook of survey research. Bingley, UK: Emerald Group Publishing. Zeiser, K., Taylor, J., Rickles, J., Garet, M. S., & Segeritz, M. (2014). Evidence of deeper learning outcomes: Findings from the study of deeper learning. Menlo Park, CA: Hewlett Foundation. https://doi.org/10.1177/0734282911406661 https://doi.org/10.3102/0013189X006001009 https://doi.org/10.3102/0013189X006002009 http://blog.ed.gov/2011/05/increasing-educational-productivity-innovative-approaches-best-practices/ http://blog.ed.gov/2011/05/increasing-educational-productivity-innovative-approaches-best-practices/ http://www.competencyworks.org/higher-education-2/is-there-any-difference-between-competency-based-education-and-mastery-based-learning/ http://www.competencyworks.org/higher-education-2/is-there-any-difference-between-competency-based-education-and-mastery-based-learning/ http://www.competencyworks.org/higher-education-2/is-there-any-difference-between-competency-based-education-and-mastery-based-learning/ https://doi.org/10.1037/0022-3514.84.3.608 https://doi.org/10.1177/014662168500900101 Investigating Student Exposure to Competency -Based Education 25 Appendix A Table A1. Sample Descriptive Statistics on Survey Items, Organized by Original Constructs (Constructs Measured by Latent Factors) Item Construct and Item description M(SD) Mastery-based progression (CB PACE), as indicated by how often each of the following occurs in the student’s current coursesa: CB PACE1 I know what I need to do to show my teachers that I am making progress on each competency 2.64(0.81) CB PACE2 I must show my teachers that I have mastered each competency before I can move on to the next one 1.86(1.26) CB PACE3 I am able to move on to the next competency when I am ready, even if other students in the course are not ready 1.41(1.24) CB PACE4 Students in my courses work on the same competency at the same time 3.12(0.93) CB PACE5 I understand how the competencies in my courses will help me in the future 2.08(1.05) Competency-specific feedback (CB FEED), as indicated by how often each of the following occurs in the student’s current coursesa: CB FEED1 My teachers share examples of excellent work on each competency 2.08(0.91) CB FEED2 My teachers let me know how my work will be assessed or graded for each competency 2.54(0.95) CB FEED3 My teachers give me a rubric so that I know how I am progressing on each competency 2.70(0.95) Personalization: Credit (PERS APP), as indicated by the share of the student’s current courses in which each of the following is trueb: PERS APP1 I am able to complete some or all course requirements online 0.36(1.02) PERS APP2 If I complete a project that wasn't assigned at school but is related to a course I am taking, I can earn credit for the project in that course 0.46(0.65) PERS APP3 I can earn credit for taking courses at another high school 0.86(0.70) PERS APP4 I can earn credit for taking courses at a college 1.00(0.68) PERS APP5 I can earn credit for completing an internship or job shadowing 0.73(0.73) Education Policy Analysis Archives Vol. 25 No. 24 26 Table A1 (Cont’d.) Sample Descriptive Statistics on Survey Items, Organized by Original Constructs (Constructs Measured by Latent Factors) Item Construct/Item description M(SD) Personalization: Individual (PERS INSTR), as indicated by how often each of the following occurs in the student’s current coursesa: PERS INSTR1 My teachers work with students in small groups or individually 1.86(0.90) PERS INSTR2 My teachers notice if I need extra help 2.00(1.03) PERS INSTR3 My teachers teach the material in several different ways in order to help students learn 1.80(1.04) PERS INSTR4 My teachers and I discussed how I am doing on each competency 0.97(0.90) PERS INSTR5 My teachers gave me written feedback on my work 1.90(0.88) PERS INSTR6 I have had opportunities to choose how to show my teachers what I have learned 1.37(1.02) Flexible assessment (FLEX), as indicated by the number of times each of the following occurred in the students current coursesc: FLEX1 I have created drawings or models to show what I have learned 0.90(0.85) FLEX2 I have taken tests or quizzes to show what I have learned 2.63(0.68) FLEX3 I have given a performance to show what I have learned 0.83(0.90) FLEX4 I have given a presentation to show what I have learned 1.38(0.86) FLEX5 I have completed a project at school to show what I have learned 1.52(0.87) FLEX6 If I do poorly on an assessment on the first try, I can try again 2.07(1.03) FLEX7 To show that I have mastered a course competency, I must demonstrate my learning in more than one way 1.86(1.07) Skills and dispositions: Respect (RSPCT), as indicated by how often each of the following occurs in the student’s current coursesa: RSPCT1 Teachers encourage students to respect the feelings of others 2.62(1.13) RSPCT2 Teachers show students how to treat each other with respect 2.38(1.14) RSPCT3 Teachers explain to students how they can disagree with each other in a respectful way 2.22(1.12) Investigating Student Exposure to Competency -Based Education 27 Table A1 (Cont’d.) Sample Descriptive Statistics on Survey Items, Organized by Original Constructs (Constructs Measured by Latent Factors) Item Construct/Item description M(SD) Skills and dispositions: Perseverance (PRSVR), as indicated by how often each of the following occurs in the student’s current coursesa: PRSVR1 When I have trouble learning something new, my teachers give me advice and strategies that help me to stick with it 2.10(1.04) PRSVR2 My teachers notice when I take extra time and effort on something that is difficult for me 1.95(1.17) PRSVR3 If I get a low score on an assessment, my teachers help me figure out how I can still do well in the class 2.08(1.17) Skills and dispositions: Responsibility (RSPNS), as indicated by how often each of the following occurs in the student’s current coursesa: RSPNS1 Teachers show students how to keep track of their progress on each of the competencies 1.84(1.09) RSPNS2 Teachers show students strategies for making sure all assignments and assessments are completed on time 1.90(1.01) RSPNS3 Teachers encourage students to take responsibility for their work 2.83(0.90) Skills and dispositions: Ownership (OWN), as indicated by how often each of the following occurs in the student’s current coursesa: OWN1 I know which steps to take during high school in order to prepare for what I want to do after I graduate 1.86(1.03) OWN2 If I need information that I don't have in order to complete an assignment, I know where to get it 1.89(1.02) OWN3 If it is difficult for me to get an assignment done on my own, I know strategies I can use to get the work finished 1.77(1.09) Skills and dispositions: Collaboration (COLL), as indicated by how often each of the following occurs in the student’s current coursesa: COLL1 Teachers show students how to help each other learn 2.23(1.10) COLL2 Teachers explain to students how to work together in groups 2.48(0.96) COLL3 Teachers encourage students to help each other outside class 2.35(1.04) Note: Source is the Competency-Based Learning Survey for Students, n=599. a Response options included 0=Never, 1=Seldom, 2=Sometimes, 3=Often, 4=Always. b Response options included 0=None, 1=Some, 3=All or most. c Response options included 0=Not at all, 1=1–2 times, 2=3–4 times, 3=5 or more times. Education Policy Analysis Archives Vol. 25 No. 24 28 Appendix B Measurement Invariance Testing Procedures Per the recommended procedure for establishing measurement invariance (Muthén & Muthén, 2016), we began with a model in which item thresholds and factor loadings were freely estimated across the two groups, with scale factors fixed at one in both groups and factor means fixed at zero. This model fit the data well (χ2(1236)=1838.40; CFI=.96; RMSEA=.04). We then estimated a model in which thresholds and factor loadings were constrained to be equal across the two groups. Although this model fit the data well (χ2(1351)=2042.23; CFI=.96; RMSEA=.04), the chi-square difference test value was significant (χ2 diff(115)=280.75, p=.00), meaning that constraining all of these parameters significantly worsened the fit of the model. Modification indices suggested allowing the thresholds for two items (CB PACE5 and FLEX6) to be freely estimated in both groups. After making this change, this model also fit the data well (χ2(1339)=1895.69; CFI=.96; RMSEA=.04), and the chi-square difference test value was non-significant (χ2 diff(103)=109.33, p=.31). Investigating Student Exposure to Competency -Based Education 29 About the Authors Sarah Ryan Education Development Center sryan@edc.org Sarah Ryan is a Senior Research Associate at Education Development Center. Her research focuses on youth preparation for and trajectories through college and the workforce, and spans disciplines including educational policy, the sociology of education, and higher education. Joshua D. Cox Education Development Center jcox@edc.org Joshua D. Cox is a Research Associate at Education Development Center. His research focuses on educational policies aimed at increasing student readiness for college and/or career, including competency-based learning reforms. education policy analysis archives Volume 25 Number 24 March 20, 2017 ISSN 1068-2341 Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non- commercial purposes only, and no alteration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or EPAA. EPAA is published by the Mary Lou Fulton Institute and Graduate School of Education at Arizona State University Articles are indexed in CIRC (Clasificación Integrada de Revistas Científicas, Spain), DIALNET (Spain), Directory of Open Access Journals, EBSCO Education Research Complete, ERIC, Education Full Text (H.W. Wilson), QUALIS A2 (Brazil), SCImago Journal Rank; SCOPUS, SOCOLAR (China). Please send errata notes to Audrey Amrein-Beardsley at audrey.beardsley@asu.edu Join EPAA’s Facebook community at https://www.facebook.com/EPAAAAPE and Twitter feed @epaa_aape. mailto:sryan@edc.org mailto:jcox@edc.org http://www.doaj.org/ mailto:audrey.beardsley@asu.edu https://www.facebook.com/EPAAAAPE Education Policy Analysis Archives Vol. 25 No. 24 30 education policy analysis archives editorial board Lead Editor: Audrey Amrein-Beardsley (Arizona State University) Editor Consultor: Gustavo E. Fischman (Arizona State University) Associate Editors: David Carlson, Margarita Jimenez-Silva, Eugene Judson, Mirka Koro-Ljungberg, Scott Marley, Jeanne M. Powers, Iveta Silova, Maria Teresa Tatto (Arizona State University) Cristina Alfaro San Diego State University Ronald Glass University of California, Santa Cruz R. Anthony Rolle University of Houston Gary Anderson New York University Jacob P. K. Gross University of Louisville A. G. Rud Washington State University Michael W. Apple University of Wisconsin, Madison Eric M. Haas WestEd Patricia Sánchez University of University of Texas, San Antonio Jeff Bale OISE, University of Toronto, Canada Julian Vasquez Heilig California State University, Sacramento Janelle Scott University of California, Berkeley Aaron Bevanot SUNY Albany Kimberly Kappler Hewitt University of North Carolina Greensboro Jack Schneider College of the Holy Cross David C. Berliner Arizona State University Aimee Howley Ohio University Noah Sobe Loyola University Henry Braun Boston College Steve Klees University of Maryland Nelly P. Stromquist University of Maryland Casey Cobb University of Connecticut Jaekyung Lee SUNY Buffalo Benjamin Superfine University of Illinois, Chicago Arnold Danzig San Jose State University Jessica Nina Lester Indiana University Adai Tefera Virginia Commonwealth University Linda Darling-Hammond Stanford University Amanda E. Lewis University of Illinois, Chicago Tina Trujillo University of California, Berkeley Elizabeth H. DeBray University of Georgia Chad R. Lochmiller Indiana University Federico R. Waitoller University of Illinois, Chicago Chad d'Entremont Rennie Center for Education Research & Policy Christopher Lubienski University of Illinois, Urbana-Champaign Larisa Warhol University of Connecticut John Diamond University of Wisconsin, Madison Sarah Lubienski University of Illinois, Urbana-Champaign John Weathers University of Colorado, Colorado Springs Matthew Di Carlo Albert Shanker Institute William J. Mathis University of Colorado, Boulder Kevin Welner University of Colorado, Boulder Michael J. Dumas University of California, Berkeley Michele S. Moses University of Colorado, Boulder Terrence G. Wiley Center for Applied Linguistics Kathy Escamilla University of Colorado, Boulder Julianne Moss Deakin University, Australia John Willinsky Stanford University Melissa Lynn Freeman Adams State College Sharon Nichols University of Texas, San Antonio Jennifer R. Wolgemuth University of South Florida Rachael Gabriel University of Connecticut Eric Parsons University of Missouri-Columbia Kyo Yamashiro Claremont Graduate University Amy Garrett Dikkers University of North Carolina, Wilmington Susan L. Robertson Bristol University, UK Gene V Glass Arizona State University Gloria M. Rodriguez University of California, Davis Investigating Student Exposure to Competency -Based Education 31 archivos analíticos de políticas educativas consejo editorial Editor Consultor: Gustavo E. Fischman (Arizona State University) Editores Asociados: Armando Alcántara Santuario (Universidad Nacional Autónoma de México), Jason Beech, (Universidad de San Andrés), Angelica Buendia, (Metropolitan Autonomous University), Ezequiel Gomez Caride, (Pontificia Universidad Católica Argentina), Antonio Luzon, (Universidad de Granada), José Luis Ramírez, Universidad de Sonora) Claudio Almonacid Universidad Metropolitana de Ciencias de la Educación, Chile Juan Carlos González Faraco Universidad de Huelva, España Miriam Rodríguez Vargas Universidad Autónoma de Tamaulipas, México Miguel Ángel Arias Ortega Universidad Autónoma de la Ciudad de México María Clemente Linuesa Universidad de Salamanca, España José Gregorio Rodríguez Universidad Nacional de Colombia, Colombia Xavier Besalú Costa Universitat de Girona, España Jaume Martínez Bonafé Universitat de València, España Mario Rueda Beltrán Instituto de Investigaciones sobre la Universidad y la Educación, UNAM, México Xavier Bonal Sarro Universidad Autónoma de Barcelona, España Alejandro Márquez Jiménez Instituto de Investigaciones sobre la Universidad y la Educación, UNAM, México José Luis San Fabián Maroto Universidad de Oviedo, España Antonio Bolívar Boitia Universidad de Granada, España María Guadalupe Olivier Tellez, Universidad Pedagógica Nacional, México Jurjo Torres Santomé, Universidad de la Coruña, España José Joaquín Brunner Universidad Diego Portales, Chile Miguel Pereyra Universidad de Granada, España Yengny Marisol Silva Laya Universidad Iberoamericana, México Damián Canales Sánchez Instituto Nacional para la Evaluación de la Educación, México Mónica Pini Universidad Nacional de San Martín, Argentina Juan Carlos Tedesco Universidad Nacional de San Martín, Argentina Gabriela de la Cruz Flores Universidad Nacional Autónoma de México Omar Orlando Pulido Chaves Instituto para la Investigación Educativa y el Desarrollo Pedagógico (IDEP) Ernesto Treviño Ronzón Universidad Veracruzana, México Marco Antonio Delgado Fuentes Universidad Iberoamericana, México Ernesto Treviño Villarreal Universidad Diego Portales Santiago, Chile Inés Dussel, DIE-CINVESTAV, México Paula Razquin Universidad de San Andrés, Argentina Antoni Verger Planells Universidad Autónoma de Barcelona, España Pedro Flores Crespo Universidad Iberoamericana, México José Ignacio Rivas Flores Universidad de Málaga, España Catalina Wainerman Universidad de San Andrés, Argentina Ana María García de Fanelli Centro de Estudios de Estado y Sociedad (CEDES) CONICET, Argentina Juan Carlos Yáñez Velazco Universidad de Colima, México javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/819') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/820') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/4276') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/1609') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/825') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/797') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/555') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/823') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/798') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/814') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/801') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/2703') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/802') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/816') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/826') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/804') javascript:openRTWindow('http://epaa.asu.edu/ojs/about/editorialTeamBio/3264') Education Policy Analysis Archives Vol. 25 No. 24 32 arquivos analíticos de políticas educativas conselho editorial Editor Consultor: Gustavo E. Fischman (Arizona State University) Editoras Associadas: Geovana Mendonça Lunardi Mendes (Universidade do Estado de Santa Catarina), Marcia Pletsch, Sandra Regina Sales (Universidade Federal Rural do Rio de Janeiro) Almerindo Afonso Universidade do Minho Portugal Alexandre Fernandez Vaz Universidade Federal de Santa Catarina, Brasil José Augusto Pacheco Universidade do Minho, Portugal Rosanna Maria Barros Sá Universidade do Algarve Portugal Regina Célia Linhares Hostins Universidade do Vale do Itajaí, Brasil Jane Paiva Universidade do Estado do Rio de Janeiro, Brasil Maria Helena Bonilla Universidade Federal da Bahia Brasil Alfredo Macedo Gomes Universidade Federal de Pernambuco Brasil Paulo Alberto Santos Vieira Universidade do Estado de Mato Grosso, Brasil Rosa Maria Bueno Fischer Universidade Federal do Rio Grande do Sul, Brasil Jefferson Mainardes Universidade Estadual de Ponta Grossa, Brasil Fabiany de Cássia Tavares Silva Universidade Federal do Mato Grosso do Sul, Brasil Alice Casimiro Lopes Universidade do Estado do Rio de Janeiro, Brasil Jader Janer Moreira Lopes Universidade Federal Fluminense e Universidade Federal de Juiz de Fora, Brasil António Teodoro Universidade Lusófona Portugal Suzana Feldens Schwertner Centro Universitário Univates Brasil Debora Nunes Universidade Federal do Rio Grande do Norte, Brasil Lílian do Valle Universidade do Estado do Rio de Janeiro, Brasil Flávia Miller Naethe Motta Universidade Federal Rural do Rio de Janeiro, Brasil Alda Junqueira Marin Pontifícia Universidade Católica de São Paulo, Brasil Alfredo Veiga-Neto Universidade Federal do Rio Grande do Sul, Brasil Dalila Andrade Oliveira Universidade Federal de Minas Gerais, Brasil