We need meaningful, systematic evaluation 18 Global Education Review is a publication of The School of Education at Mercy College, New York. This is an Open Access article distributed under the terms of the Creative Commons Attribution-Noncommercial 3.0 Unported License, permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. Citation: Urban, Mathias (2017). We need meaningful, systematic evaluation, not a preschool PISA. Global Education Review, 4 (2). 18-24 We Need Meaningful, Systemic Evaluation, Not a Preschool PISA Mathias Urban University of Roehampton A Preschool PISA If you thought an international organisation committed to promoting economic growth was an unlikely candidate to intervene in your local preschool, think again. The Organisation for Economic Co-operation and Development (OECD) has an established history of showing an interest in education systems of its member states. This interest has manifested itself most prominently in a series of international standardised test for 15 year olds – the Programme for International Student Assessment (PISA) and the global angst its rankings and league tables has created. How does your country’s school system fare in relation to Finland or South Korea (in all due respect to Finnish and South Korean readers of this piece)? The OECD has now announced the launch of an international standardised assessment programme for five-year-old children, the International Early Learning and Child Well- being Study (IELS). According to the recently set up website (http://www.oecd.org/edu/school/international -early-learning-and-child-well-being-study.htm) the programme will assess four “early learning domains” (emerging literacy, emerging numeracy, self-regulation, empathy and trust). Assessing each domain, we learn, will take “approximately 15 minutes” using a “tablet- based” test. Further “indirect assessment of children’s skills will be obtained from parents and staff through written and online questionnaires.” Additional information will be provided by “the study administrators (sic) observations.” An international consortium has been contracted to administer the study, consisting of the Australian Council for Educational Research (ACER)1, the International Association for the Evaluation of Educational Achievement (IEA)2, and cApStAn3, and a timeline has been announced: the study will go ahead in the “Northern Hemisphere” in 2018, the “Southern Hemisphere” in 2019, followed by “quality control and analysis” and “report” in 2019-2020. The process of publishing more detailed information on the OECD website also saw changes to the project: earlier announcements of a pilot (which could have been evaluated) were removed and the title of the study was changed from the original International Early Learning to International Early Learning and Child Well- being Study. However, the acronym remains the same (IELS) and there is only fleeting reference to well-being on the website and accompanying documents. If this initiative has escaped your attention, you are in good company. Despite having consulted with government ______________________________ Corresponding Author: Mathias Urban, University of Roehampton, Froebel College, London SW15 5PJ United Kingdom Email: mathias.urban@roehampton.ac.uk http://www.oecd.org/edu/school/international-early-learning-and-child-well-being-study.htm http://www.oecd.org/edu/school/international-early-learning-and-child-well-being-study.htm mailto:mathias.urban@roehampton.ac.uk We need meaningful, systematic evaluation 19 representatives of 16 countries in a “scoping group” since 2012, little to no information about the initiative has been shared with the international early childhood community. The lack of information and absence of any meaningful consultation with early childhood professionals and scholars has been pointed out repeatedly. An article published in Contemporary Issues in Early Childhood by Peter Moss and co-authors in August 2016 was first to ask, “Did you know about this?” (Moss et al., 2016). Until today, based on my own and colleagues’ experiences from talking with practitioners in many countries, the answer is a resounding “No.” Several publications have expressed concerns about IELS, its underlying assumptions, the process, and the implication – for young children as well as for the early childhood profession. Shortly after Moss’ article in CIEC, Beth Blue Swadener (Arizona State University) and I published a paper titled Democratic accountability and contextualised systemic evaluation. The piece was published in International Critical Childhood Policy Studies (Urban & Swadener, 2016) and on the website of the Reconceptualising Early Childhood Education (RECE) network (receinternational.org) and signed in support by nearly 200 academics, professionals and activists from over 20 countries. Other critical publications in various national and international contexts followed, e.g., Alan Pence’s Baby PISA (Pence, 2017) and Margaret Carr, Linda Mitchell and Lesley Rameka’s piece on IELS and Te Whāriki (Carr, Mitchell and Rameka, 2016; Mackey, Hill and de Vocht, 2016). There is an update on recent developments in the current issue of CIEC (Moss & Urban, 2017). I summarise the key arguments below. Garbage in – garbage out? Young children and standardised assessment don’t go well together One of the key methodological concerns about IELS is its apparent disregard for any evidence that suggests caution is appropriate when using standardised testing of young children for international comparative purposes. In the US, a country with an established history of high- stakes testing, studies consistently show the low reliability and validity of standardised tests of children, especially in contexts of large-scale comparison (Meisels, 2004, 2006; Meisels & Atkins-Burnett, 2006; Madaus & Clarke, 2001; Raudenbush, 2005). Referring to these arguments we have argued that the findings from IELS will be “largely meaningless due to their disconnect with and disrespect for diverse, locally embedded approaches to early childhood education and care” (Urban & Swadener, 2016, pp 7, 8). While the collection of child-based data on a global scale, in order to produce PISA style country rankings and league tables raises serious ethical questions (see There can only be one below) it also points to other critical aspects of IELS. Not least that it is a waste of resources and a missed opportunity as it will draw attention away from meaningful local and international initiatives to create in-depth understandings of complex early childhood systems, develop meaningful systemic evaluation and support much-needed improvement of experiences and outcomes for all children. The OECD must be aware of the existing meta-analyses of standardised test results of young children. That it keeps pushing ahead regardless lead us, in our 2016 article written on behalf of the Reconceptualising Early Childhood Education network, to ask whose interests are served by rolling out IELS: Promoting and rolling out standardised assessment and comparison approaches regardless of overwhelming evidence that 20 Global Education Review 4(2) they cannot achieve their stated goals raises the question whether political and corporate profit interests are being privileged over valid research, children’s rights and meaningful evaluation. (Urban & Swadener, 2016, p. 7) There can only be one (way of teaching children)? IELS’s disregard for diversity and children’s rights IELS in itself is not the problem. Or, more specifically, it is only a small aspect of a much bigger global problem. As we have pointed out repeatedly, IELS is another step towards drawing early childhood into a global standardised assessment framework that is unable (unwilling!) to see children’s experiences in the education system through any other lens than the one provided by PISA. The OECD is open about the connection. The IELS “Call for Tenders” states that information gathered from children at preschool age will eventually provide information on the trajectory between early learning outcomes and those at age 15, as measured by PISA. In this way, countries can have an earlier and more specific indication of how to lift the skills and other capabilities of its young people. (OECD, 2015, p. 103) What is stated here as an intention for the future has immediate consequences today. This is evident, for example, in a recent e-mail exchange with a colleague in a country that has become of interest for the OECD. Both the country and the colleague shall not be named in this piece. What can be said is that the country in question has recently adopted a highly ambitious integrated policy framework for early childhood, based on a holistic and rights-based understanding of public responsibility for all young children. A meeting was called by the country’s Ministry of Education, to discuss the direction of education policies with a delegation from the OECD. At that meeting it was made clear that the country’s commitment to holistic child development should be abandoned, and resources focused on improving the country’s PISA score instead: Dear Mathias [XXXX] is trying to be accepted by OECD. They did a study about education in [XXXX]. They presented as results […] the big gap of [XXXX]ian children related to other countries. They insisted a lot that the study demonstrated that children are not learning what they need because their performance was very low. […] The key issue is the discussion between Human development vs. scholarly objectives related to meeting international standards. In the background all is about PISA´s test and [XXXX]ian results in order to be accepted in OECD […] they argue that children are wasting time with play, arts and literature. (Personal communication, 2017, my emphasis) The OECD’s commitment to ensuring that children in participating countries no longer engage in wasteful activities like play is only one, albeit striking, example of its disregard for the diversity of possible approaches to culturally embedded educational and child rearing practices. The United Nations Declaration of the rights of Indigenous Peoples (DRIPS) explicitly recognises the right of Indigenous Peoples to diversity and to education “in a manner appropriate to their cultural methods of teaching We need meaningful, systematic evaluation 21 and learning” (Article 14), and to “dignity and diversity of their cultures, traditions, histories and aspirations which shall be appropriately reflected in education and public information” (Article 15). The OECD’s tunnel vision continues at an operational level. According to the IELS website, children’s perspectives will only be sought after the tests have been completed. Children will be asked if they liked the assessment activity, its content and different aspects. These debriefing sessions will be used to ensure children’s well-being during the assessment but also to provide valuable feedback about the assessment material and procedures. In addition, children will be asked about their favourite learning activities in different settings. (http://www.oecd.org/edu/school/interna tional-early-learning-and-child-well- being-study.htm) Apart from this post-fact assessment, there seems to be no intention to engage with children before the test. There is no indication that children’s (or practitioners’) consent to participate in IELS will be sought. To base a research project on assumed (instead of informed) consent would be met with astonishment – and rejection – by any university ethics committee, as any research student will know. The OECD seems to have exempted themselves from such standards. The curious incident of the evidence in the night-time As Moss et al (2016) and others have pointed out, the OECD has chosen to take a highly selective approach to evidence that informs the field of early childhood at international level. The Organisation adopts a particular paradigmatic position which might be described as hyper-positivistic… the OECD is free to choose its position. However, it should be aware that it has made a choice and taken a particular perspective. It should also be aware that there are other choices and other perspectives. Yet on both counts it shows a total lack of self-awareness (Moss et al., 2016, p. 346) This undeclared paradigmatic position persistently denies that other positions exist, and have indeed existed for many years. Over the past 25 years reconceptualist scholars have contributed to a rapidly growing body of research and knowledge that offer alternative – postcolonial, critical, feminist, indigenous, transdisciplinary – understandings of what it means to educate and care for young children: “Such research and knowledge is rendered invisible by OECD, its existence not even acknowledged” (Moss & Urban, 2017). More specifically, as Moss et al., (2016) remind us, the OECD chooses not to engage with any scholarship critical of PISA. Critical points raised by Morris (2016), Alexander (2010, 2012) and others are similarly relevant to testing 5 year olds for international comparison. “National education systems”, Robin Alexander (2012) reminds us, “are embedded in national culture.” Which explains why “no educational policy or practice can be properly understood except by reference to the web of inherited ideas and values, habits and customs, institutions and world views, that make one country distinct from another” (p. 5). Similar arguments have been made by the OECD itself in the first two Starting Strong reports (OECD, 2001, 2006): ECEC policy and the quality of services are deeply influenced by underlying assumptions about childhood and education: what does childhood mean in this society? How should young children be reared and educated? What are the purposes of education and care, of early http://www.oecd.org/edu/school/international-early-learning-and-child-well-being-study.htm http://www.oecd.org/edu/school/international-early-learning-and-child-well-being-study.htm http://www.oecd.org/edu/school/international-early-learning-and-child-well-being-study.htm 22 Global Education Review 4(2) childhood institutions? What are the functions of early childhood staff? (OECD, 2001, p. 63). How great expectations in Washington are dashed in Oakland (Wellington, Berlin and Dublin) If governance theory has shown one thing it is this: no one does as they are told. Ever. That top- down implementation of policies doesn’t work has been at the centre of research into the governance of complex systems (like education systems) for many years – hence the title of this section, in reference to a classic paper by Pressman and Wildavsky (1984). However, the entire OECD operation seems firmly grounded in a firm belief that it is possible to transfer policies from one context (country, culture) to another, and to implement them without distortion. The naivety of this “implied model of enlightened policymakers objectively and rationally applying lessons from other countries” (Moss et al., 2016) has been pointed out by Paul Morris (2016). But even a model that doesn’t work in the first place can be (ab)used for other purposes. As Morris notes there is a wholly unsurprising tendency for policymakers to view such comparative data on pupil performance as an expedient resource, which serves a primarily symbolic role in the theatre of politics and provides a massive source of evidence, from which they can hunt for correlations to legitimize their own ideological preferences. (Morris, 2016, p. 11) The great expectations nurtured by the OECD in relation to IELS are being dashed already in many countries – as they refuse to take part in the initiative. Critical statements are being published in New Zealand, Germany, Ireland, Belgium, the UK, to name just a few, drawing on our arguments (Moss et al., 2016; Urban & Swadener, 2016; Moss & Urban, 2017) and building resistance among scholars, professionals and activists. An international critical coalition is beginning to take shape. TINA, you’re not our friend. There are alternatives to the ‘dictatorship of no alternatives’ At a recent meeting at the OECD headquarters in Paris to discuss IELS we were asked if we were opposed to quantitative methods. This, of course, is a) not the case and b) not the point. What we are opposed to, when it comes to evaluating the workings of complex, diverse, and culturally embedded support systems for young children and their families, is to be told that there is no alternative to standardised assessment and decontextualised measurement. We are strongly supportive of evaluation – not least as a way of holding governments and ourselves to account. We are also convinced of the importance of learning with and from others in international contexts. The OECD itself has shown that such approaches are possible. The landmark Starting Strong I+II studies (2001, 2006) are examples of a carefully designed and conducted exploration of early childhood systems in 20 countries, based on respect for diversity. My own international work (as that of many others) draws on the leadership provided by John Bennett and his co-authors in Starting Strong I+II. What we have come to understand is that early childhood care, education and development services and practices are at their best when they are developed and supported as part of a Competent System (Urban, 2012; Urban, Vandenbroeck, Van Laere, Lazzari, & Peeters, 2012; Vandenbroeck, Urban, & Peeters, 2016). There are alternatives to IELS in its current form and I have no doubt the We need meaningful, systematic evaluation 23 international early childhood community would be supportive of a meaningful, contextualised leaning initiative, conducted in respectful and participatory ways. Notes 1. ACER is an “independent, not-for-profit research organization”; its mission is “to create and promote research-based knowledge, products and services that can be used to improve learning across the lifespan” (www.acer.org). ACER led the management of PISA in 2006 and 2009. 2. IEA is a Netherlands-based “international cooperative of national research institutions, government research agencies, scholars and analysts working to evaluate, understand and improve education worldwide” (www.iea.nl). IEA has a long history of providing international comparative assessments in education, including the Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS). 3. cApStAn is a Belgium-based company that provides “linguistic quality control” for multilingual projects (www.capstan.be). The company has been involved in PISA since 1998. References Alexander RJ (2010) ‘World class schools’ – noble aspirations or globalized hokum? Compare 40(6): 801–817. Alexander RJ (2012) Moral panic, miracle cures and education policy: What can we really learn from international comparisons? Scottish Education Review 44(1): 4–21. Carr, M., Mitchell, L. and Rameka, L. (2016) Some thoughts about the value of an OECD international assessment framework for early childhood services in Aotearoa New Zealand. Contemporary Issues in Early Childhood 17(4): 450–454. Madaus, G. F., & Clarke, M. (2001). The adverse impact of high-stakes testing on minority students: Evidence from one hundred years of test data. In G. Orfield & M. L. Kornhaber (Eds.). Raising standards or raising barriers? Inequality and high-stakes testing in public education (pp. 85–106). New York: The Century Foundation Press. Meisels, S. J., & Atkins-Burnett, S. (2006). Evaluating early childhood assessments: A differential analysis. In K. McCartney & D. Phillips (Eds.), Handbook of Early Childhood Development (pp. 533–549). Oxford: Blackwell Publishing. Meisels, S.J. (2004). Should We Test Four -Year Olds? [Commentary]. Pediatrics, 113(5), 1401-1402. Meisels, S.J. (2006). Accountability in Early Childhood: no easy answers. Occasional Paper 6, March 2006. Chicago: Erikson Institute. Morris P. (2016) Education Policy, Cross-National Tests of Pupil Achievement, and the Pursuit of World-Class Schooling. London: UCL Institute of Education Press. Moss, P., & Urban, M. (2017). The OECD’s International Early Learning Study: what happened next. Contemporary Issues in Early Childhood, 18(2): 250-258. doi:10.1177/1463949117714086. Moss, P., Dahlberg, G., Grieshaber, S., Mantovani, S., May, H., Pence, A., . . . Vandenbroeck, M. (2016). The Organisation for Economic Co-operation and Development’s International Early Learning Study: Opening for debate and contestation. Contemporary Issues in Early Childhood, 17(3), 343-351. doi:10.1177/1463949116661126 OECD. (2001). Starting Strong. Early Childhood Education and Care. Paris: OECD. OECD. (2006). Starting Strong II. Early Childhood Education and Care. Paris: OECD. OECD. (2015). Call for tenders: International Early Learning Study. Retrieved from http://www.oecd.org/callsfortenders/CfT 100001420 International Early Learning Study.pdf Pence, A. (2017). Baby PISA: Dangers that can Arise when Foundations Shift. Journal of Childhood Studies, 41(3), 54-58. Pressman, J. L., & Wildavsky, A. B. (1984). Implementation: How Great Expectations in Washington are Dashed in Oakland : Or, why It's Amazing that Federal Programs Work at All, this Being a Saga of the Economic Development Administration as Told by Two Sympathetic Observers who Seek to Build Morals on a Foundation of Ruined Hopes. Berkely, CA: Colegio Nacional de Ciencias Políticas y Administración Pública: Fondo de cultura económica. Raudenbush, S. (2005). Newsmaker Interview: How NCLB testing can leave some schools behind. Preschool matters, 3(2), Rutgers University: National Institute of Early Education Research. http://www.acer.org/ http://www.iea.nl/ http://www.capstan.be/ 24 Global Education Review 4(2) Urban, M. (2012). Researching Early Childhood Policy and Practice. A Critical Ecology. European Journal of Education, 47(4), 494-507. doi:10.1111/ejed.12012 Urban, M., & Swadener, B. B. (2016). Democratic accountability and contextualised systemic evaluation. A comment on the OECD initiative to launch an International Early Learning Study (IELS). International Critical Childhood Policy Studies, 5(1), 6-18. Urban, M., Vandenbroeck, M., Van Laere, K., Lazzari, A., & Peeters, J. (2012). Towards Competent Systems in Early Childhood Education and Care. Implications for Policy and Practice. European Journal of Education, 47(4), 508-526. doi:10.1111/ejed.12010 Vandenbroeck, M., Urban, M., & Peeters, J. (Eds.). (2016). Pathways to Professionalism in Early Childhood Education and Care. London and New York: Routledge. About the Author Mathias Urban, PhD, is Professor of Early Childhood Studies and Director of the Early Childhood Research Centre at the University of Roehampton, London. His research interests unfold around questions of diversity and equality, social justice, evaluation and professionalism in working with young children, families and communities in diverse socio-cultural contexts. He is an International Research Fellow with the Velma E. Schmidt Critical Childhood Public Policy Research Collaborative, a member of the PILIS research group (Primera Infancia, Lenguaje e Inclusión Social), Chair of the DECET (Diversity in Early Childhood Education and Training) network, President of the International Froebel Society and a member of the AERA special interest group Critical Perspectives on Early Childhood.