Smith, W. C. (2017, December 13). Review of The global education race: Taking the measure of PISA and international testing, by S. Sellar, G. Thompson, & D. Rutkowksi. Education Review, 24. December 13, 2017 ISSN 1094-5296 Sellar, S., Thompson, G. & Rutkowski, D. (2017). The global education race: Taking the measure of PISA and international testing. Edmonton, Canada: Brush Education. Pp. 106 ISBN: 978-1-55059-711-0 Reviewed by William C. Smith UNESCO - Global Education Monitoring Report France In a globalized world, countries look beyond national boundaries for ‘best practices’ and engage in policy borrowing. The massive increase in international large- scale assessments (Heyneman & Lee, 2014), has provided comparisons used to identify shining stars such as Finland and Shanghai. As more and more counties participate, government ministries, the media, and the public want to know where their education system fits into the hierarchy. In Sam Sellar, Greg Thompson, and David Rutkowksi’s The Global Education Race: Taking the Measure of PISA and International Testing, they make it clear that the OECD’s Programme for International Student Assessment (PISA) is the dominant racetrack on which countries and schools systems position themselves against one another. Nobody wants to be left out or left behind. This scenario was clear to me on a trip to Australia. On August 30, 2017, the front page of The Australian rang out with Education Review /Reseñas Educativas 2 the headline, “Prove Teaching Skills or Fail Course”. According to this article, results showing continually declining scores on the national assessment were compounded by “last year’s dire international test scores in reading, maths and science in the Program for International Student Assessment.” One of the main factors for inhibiting progress? Poorly prepared teachers. Therefore, according to this report, in order to change course, additional, more rigorous assessments are being put in place to evaluate the classroom readiness of preservice teachers. This news story was the last of five articles that month that mentioned PISA as part of a general call for change. The Global Education Race is not the first book to focus on the impacts of PISA. Since Bieber and Martens (2011) claimed that research on the effects of the “PISA study on national reforms in education policy-making is scarce” (p. 102) a mushrooming of literature exploring the validity and effects of PISA (see Meyer & Benavot, 2013) has emerged. The early chapters of The Global Education Race follow other research in examining media perspectives on PISA, with particular emphases on national rankings or league tables. The release of PISA results can spur education crisis moments for countries, creating space to push through reform or reaffirm policy decisions. As the authors recognize, in essence “PISA produces catalyst data” (Sellar et al., 2017, p. 19), a point that is well documented through research on PISA shock in various countries including Germany, Japan, Denmark, and Switzerland (Beiber and Martens, 2011; Heyneman & Lee, 2014). In Breakspear’s (2012) analysis of 37 OECD countries and economies, he found that over 80% reported some influence of PISA on their education policies. Highlighting the mounting pressure felt in Australia, The Global Education Race illustrates how PISA results for some countries have become the goal instead of a tool to understand processes and barriers to education quality in the country. For example, in their 2013 Education Act, Australia targeted a top 5 ranking in PISA by 2025 as one of its goals. Chapters on stories and rankings bring the role of media to the forefront. Although the OECD claims less than 1% of their results focus on league tables, national ranking is often the most emphasized, and sometimes only, result included in the media. This situation is due in part to the needs of the media, which relies on producing simple, newsworthy material that is easily understood by the reader (Yasukawa et al., 2017). This book addresses the tension between the OECD communication of massive amounts of seemingly complex results and the requirements of the media, and commends the OECD for making substantial efforts to inform journalists of results, provide journalists with training to correctly interpret results, and engage with the media across multiple channels – coordinated pre-launch events, interviews, and social media campaigns. Ultimately, however, national rankings garner the most attention. Stories are constructed around national rankings, and, after the initial media rush, these stories “persist, or can even gain momentum, as memes that become detached from their basis in PISA findings” (Sellar et al., 2017, p. 22). Finland is put forward as an example memes. In evaluating Finland different countries and stakeholder groups project their own reality on what makes the Finnish education system high quality, with explanations often aligning with their own political interests. Once PISA results are released, national leaders use changes in scores to scandalize prior administrations or shift the blame to others, including teachers and schools. Policymaker reaction to rankings is often aligned with their desired narrative in an attempt to “support pre- Review of The Global Education Race by W. Smith 3 existing arguments for educational change” (Sellar et al., 2017, p. 63). Setting itself apart, the unique addition of The Global Education Race is the target audience. Written for educators, parents, and other stakeholders unfamiliar with PISA and perhaps only exposed to PISA through media messages, this book aims to “help people understand what PISA is, what it does, and what it cannot do” (Seller et al., 2017, p. 7). So why should people care about PISA especially when “many classroom teachers and parents have either not heard or know very little about it” (Seller et al., 2017, p. 7) and only a small percentage of 15 year old students are included in the sample? Individuals still indirectly experience the effects of PISA through changes in policy and rhetoric that shape educational values. PISA also increasingly influences national assessments, as Addey (2017) points out the alignment of national assessments with PISA is one of the six long-term objectives the OECD has identified in the PISA Longer-Term Strategy. Additionally, with the normative pressure to join and the addition of PISA for Development and PISA for Schools, if your country is absent from the PISA participation list, the absence is likely short- lived. Having set the context on global education competition, chapters on testing and comparison provide the non-academic target audience with the tools to critically examine the portrayal of PISA in the media. When reading a news story, such as the one I was reading in Australia, what questions should spring to mind? The Global Education Race helps media consumers evaluate the reality of PISA, providing the necessary insights to address key questions that should be asked when any set of results are presented. Question 1: Who is being tested? It is important to recognize that PISA covers only 15 year olds in school. As the book outlines for 16 participating countries this means at least one in five 15 year olds are not included in the sample. For Costa Rica this was closer to one in two. Question 2: What is being assessed? PISA evaluates the application of knowledge to solve real life problems, it does not evaluate how well students have learned a curriculum. Since it is not curriculum based it is an inappropriate measure to use for evaluating the success of an education system in implementing a curriculum. Question 3: Do the results represent the same thing in all countries or economies? Not all countries release all results from PISA. One of the most striking examples provided was from the 2009 PISA, which was conducted in 12 Chinese regions, but only results from Shanghai, the highest performing region in reading with a score of 556 was released. If all 12 were included, China would have scored a 486, below the OECD average. Question 4: What do the national rankings mean? Actually very little. There is a reason the OECD tries to downplay rankings. Minor changes can impact rankings and rankings themselves do not identify whether there are real significant differences in country performances. The Global Education Race claims countries are best placed in high, average, and low-performing tiers. When recognized in this manner, there are minimal changes in where countries are placed between PISA rounds. Question 5: What valid conclusions can be drawn from the results? The media commonly promote the link between PISA scores and national economic prosperity. While this position is supported by some (Hanushek & Woessman, 2015), including the OECD (Addey & Sellar, 2018). This book and other recent research (Komatsu & Rappleye, 2017) makes it clear that there is no association Education Review /Reseñas Educativas 4 between scores on an international assessment and economic growth. This set of questions is an impressive and useful start, which should help everyone realize that “even under the very best circumstances, PISA is, by definition, an assessment in select content areas (intentionally divorced from curriculum) that is administered on a single day to a sample of 15-year-olds that are enrolled in school. In an ideal setting, the PISA design and sample limits inferences to a narrowly defined population regarding their performance in a narrowly defined set of topics” (Sellar et al., 2017, p. 53). While the book only touches upon them in passing, two other lines of inquiry should inform the critical consumer. First, it is important to recognize the increasing number of countries participating in PISA when comparing national rankings over time. In The Global Education Race, Denmark is an example of a country declining in the league tables in reading from a ranking of 17th in 2000 to a ranking of 24th in 2009. Although the authors point out that this decline in ranking did not match the minimal decline in score (from 497 to 495) what was not mentioned was the greater than 50% increase in participants in PISA over that same period (Smith, 2014). In their analysis of Australia’s rankings over time, Baroutsis and Lingard (2017) demonstrate the importance of accounting for increased participation. When limiting the national rankings to the 32 countries that participated in PISA between 2000 and 2012, Baroutsis and Lingard assert that the declines for Australia were not nearly as drastic as those emphasized in the media: instead of falling from 6th to 19th Australia would rank 12th in mathematics, and instead of falling from 4 th to 13th Australia would rank 9th in reading. Second, questions should arise on the appropriateness of a reference group. With the OECD encouraging all countries to look to the highest performing systems for best practices (Addey, 2017), a “nation’s referential position is no longer conditioned and legitimated by similarities with a society and a schooling system, but on the basis of their placement in the global rankings of PISA” (Baroutsis & Lingard, 2017, p. 445). As cultural, social, and historical factors play such an important part in the design and goals of an education system, it is important to ask whether comparing Romania to Shanghai or Uruguay to Finland makes sense and to what extent the extrapolation of lessons is really possible. The use of simplified student assessment scores by the media is not limited to PISA. The lessons from The Global Education Race on questioning and carefully evaluating the validity of media messages can, and should, be used when reviewing other results, such as school report cards or league tables from national assessments. PISA is part of a larger global testing culture that shapes society and valorizes quantified indicators of quality (Smith, 2016). The movement of PISA from a low stakes to high stakes exam mirrors the transformation toward testing for accountability and away from formative purposes (Smith, 2014, 2016). As the authors (2017) point out “PISA has become high stakes for government, and these stakes invariably become policy drivers that have an impact on schools, teachers, and students” (p. 70). Similar to other attempts that link test scores to high stakes accountability, the use of PISA has led to undesirable gaming-the- system behavior. PISA-participating countries have seen an increase in test preparation and a narrowing of the curriculum, where “subjects that are not assessed by PISA, such as arts and music, can suffer” (Sellar et al., 2017, p. 92). The influence of PISA on national assessments make it unlikely that the inclusion of subjects in national assessments will change; 99% of national assessments currently Review of The Global Education Race by W. Smith 5 include mathematics and reading but only 37% include art and culture (Anderson & Winthrop, 2016). The Global Education Race expands the audience of PISA research in encouraging critical consumers to make their democratic voice heard. The book is not anti-PISA or anti-testing but instead calls for a more careful interpretation of PISA results by both the media and public. The simplified representation of results and pressing sense of urgency felt by policymakers contributes to a race that pits diverse countries against each other as they attempt to reach the pinnacle of the international league table. The risk inherent in the global education race is that “school systems may find themselves running in the wrong direction in pursuit of reforms that will not get them where they need to go” (Sellar et al., 2017, p. 3). The Global Education Race acts as an informative guidebook that not only provides the essential context for understanding what PISA is, but also equips individuals with the necessary tools to question often-exaggerated media claims. Thoughtful consideration of PISA and other testing results can help shape policy responses, if an informed public pressures leaders to avoid hastily conceived or pre- packaged short-term solutions to education challenges. References Addey, C. (2017). Golden relics & historical standards: How the OECD is expanding global education governance through PISA for Development. Critical Studies in Education. doi.org/10.1080/17508487.2017.1352006 Addey, C., & Sellar, S. (2018, Forthcoming). Why do countries participate in PISA? Understanding the role of international large-scale assessments in global education policy. In A. Verger, H. K. Altinyelken, & M. Novelli (Eds.), Global Education Policy and International Development: New Agendas, Issues and Policies. New York: Bloomsbury Publishing. Anderson, K., & Winthrop, R. (2016). Building global consensus on measuring learning. In S. McGrath & Q. Gu (Eds.), Routledge Handbook of International Education and Development (pp. 93-107). New York: Routledge. Baroutsis, A., & Lingard, B. (2017). Counting and comparing school performance: An analysis of media coverage of PISA in Australia, 2000-2014. Journal of Education Policy, 32(4), 432-449. Bieber, T., & Mertens, K. (2011). The OECD PISA study as a soft power in education? Lessons from Switzerland and the US. European Journal of Education, 46(1), 101-116. Breakspear, S. (2012). The policy impact of PISA: An exploration of the normative effects of international benchmarking in school system performance. OECD Education Working Paper, No. 71. Paris: OECD. Hanushek, E., & Woessman, L. (2015). The Knowledge Capital of Nations: Education and the Economics of Growth. Cambridge, MA: MIT Press. Heyneman, S. P., & Lee, B. (2014). The impact of international studies of academic achievement on policy and research. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of International Large-scale Assessment: Background, Technical Issues, and Methods of Data Analysis (pp. 37-72). London: CRC Press. Komatsu, H., & Rappleye, J. (2017). A new global policy regime founded on invalid statistics? Hanushek, Woessman, PISA and economic growth. Comparative Education, 53(2), 166-191. Education Review /Reseñas Educativas 6 Meyer, H.-D., & Benavot, A. (2013). PISA, Power and Policy: The Emergence of Global Educational Governance. Oxford: Symposium Books. Sellar, S., Thompson, G., & Rutkowski, D. (2017). The Global Education Race: Taking the Measure of PISA and International Testing. Edmonton, Canada: Brush Education. Smith, W. C. (2014). The global transformation toward testing for accountability. Education Policy Analysis Archives, 22(116). http://dx.doi.org/10.14507/epaa.v22.1571 Smith, W. C. (Ed.) (2016). The Global Testing Culture: Shaping Educational Policy, Perceptions, and Practice. Oxford: Symposium Books. The Australian. (2017). Prove teaching skills or fail course. The Australian (August 30, 2017). Yasukawa, K., Hamilton, M. & Evans, J. (2017). A comparative study of national media responses to the OECD Survey of Adult Skills: Policy making from the global to the local? Compare, 47(2), 271-285. About the Reviewer William C. Smith, Ph.D., is a Senior Policy Analyst with UNESCO’s Global Education Monitoring (GEM) Report and a Research Affiliate at Penn State University’s Population Research Institute. His larger research is situated around the role social policy at the national and international level plays in education equity and outcomes. This article continues his current work on the power of student test scores to shape policy, influence student outcomes, and warp the education process. His recent publications in this line of work include the edited book, The Global Testing Culture: Shaping Education Policy, Perceptions, and Practice. Review of The Global Education Race by W. Smith 7 Education Review/Reseñas Educativas/Resenhas Educativas is supported by the edXchange initiative’s Scholarly Communications Group at the Mary Lou Fulton Teachers College, Arizona State University. Copyright is retained by the first or sole author, who grants right of first publication to the Education Review. Readers are free to copy, display, and distribute this article, as long as the work is attributed to the author(s) and Education Review, it is distributed for non-commercial purposes only, and no alteration or transformation is made in the work. More detail s of this Creative Commons license are available at http://creativecommons.org/licenses/by-nc-sa/3.0/. All other uses must be approved by the author(s) or Education Review. Education Review is published by the Scholarly Communications Group of the Mary Lou Fulton Teachers College, Arizona State University. Disclaimer: The views or opinions presented in book reviews are solely those of the author(s) and do not necessarily represent those of Education Review. Connect with Education Review on Facebook (https://www.facebook.com/pages/Education- Review/178358222192644) and on Twitter @EducReview https://www.facebook.com/pages/Education-Review/178358222192644 https://www.facebook.com/pages/Education-Review/178358222192644