Literature Review Journal of Learning Development in Higher Education ISSN: 1759-667X Issue 12: November 2017 E for Enigma: The Teaching Excellence Framework (TEF) and the challenges of measuring excellence A book review of French, A. and O’Leary, M. (2017) Teaching Excellence in Higher Education: Challenges, Changes and the Teaching Excellence Framework. Bingley, UK: Emerald Publishing Limited. Dr Alexandra J. F. Patel University of Leicester, UK Introduction The Teaching Excellence Framework (TEF) aims to recognise and reward excellence in teaching and learning, and help inform prospective students' choices for higher education (Higher Education Funding Council for England, 2017). Developed by the Department of Education in England and first trialled in 2016, the TEF claims to be a voluntary scheme that recognises excellent teaching and helps students choose where to study through awarding institutions a gold, silver or bronze status. From 2018-19, universities with a TEF award will be allowed to increase their tuition fees in line with inflation. The TEF has, however, been met with some concern. This debate is critical in shaping how we think of teaching and learning, the ways in which we seek to develop educational practice and, as learning developers, should be something we engage with, as Amanda French points out in Chapter 2 of this book: The TEF, one can argue therefore, should be regarded as much more than a tool for measuring teaching excellence; it has the potential to become the key vehicle for changing the face of the HE sector in Britain (pp.11-12). The book entitled ‘Teaching Excellence in Higher Education: Challenges, Changes and the Teaching Excellence Framework’ helps us to understand the TEF by placing it within the social, political, and historical contexts in which it developed. It discusses some of the main factors driving the development of TEF, such as the increased marketisation, Patel E for Enigma: The Teaching Excellence Framework (TEF) and the challenges of measuring excellence Journal of Learning Development in Higher Education, Issue 12: November 2017 2 globalisation and accountability of HE, that have led to the contentious Higher Education and Research Bill and the introduction of the Teaching Excellence Framework (TEF). This book, instead, seeks to develop the concept that teaching is multifaceted and complex, and measurement of excellence must be contextualised by many factors such as discipline, institution and socio-political climate. The book considers how we can capture excellence in the wide-ranging contexts of Higher Education (HE) teaching and what TEF means for those involved in learning and teaching.I In doing so, it hopes to inspire debate. This review will provide a brief background to the authors and describe the structure of the book. It will then discuss three main questions that the authors address – what are the socio-political drivers behind TEF? Can TEF measure teaching excellence? What approaches might be more meaningful in developing teaching? Author background Teaching Excellence in Higher Education is edited by Dr Amanda French and Dr Matt O’Leary, with contributions from Professor Sue Robson and Associate Professor Phil Wood. French and Leary have over 25 years’ experience each of teaching across colleges, schools and universities, and both are currently based in the School of Education in Birmingham. Robson is Professor of Education at Newcastle University and is actively involved in research around the internationalisation of Higher Education. Wood is an Associate Professor in the School of Education at the University of Leicester and involved in teaching and research around pedagogic innovation and organisational change, using the lens of complex theory and process philosophy. Book structure The book is concise, well written and thought provoking, with six chapters, including an introduction and conclusion. Chapter 2 Contextualising Excellence in Higher Education Teaching: Understanding the Policy Landscape by French puts the Higher Education and Research Bill and consequent Patel E for Enigma: The Teaching Excellence Framework (TEF) and the challenges of measuring excellence Journal of Learning Development in Higher Education, Issue 12: November 2017 3 Teaching Excellence Framework in a socio-political context – a valuable contribution to understanding the discourse and approach of TEF through its drivers and origins. French also considers how well teaching excellence can be measured by the TEF metrics which use: the Higher Education Funding Council for England (HEFCE) data on students’ entry and retention; the National Student Survey (NSS); the Destination of Leavers from Higher Education (DLHE) survey and a contextualised information report. Chapter 3 From Teaching Excellence to Emergent Pedagogies: A Complex Process Alternative to Understanding the Role of Teaching in Higher Education by Wood gives a thorough review of the literature around what excellence might be, whether the TEF can hope to measure it and what other approaches might be more meaningful to developing learning and teaching. He puts forward the idea of ‘emergent pedagogy’ to help the reader appreciate the complex interactions and relationships that we simplify under the term ‘teaching’. Chapter 4 Monitoring and Measuring Teaching Excellence in Higher Education: From Contrived Contemplation to Collective Collaboration by O’Leary considers what teaching excellence means to teachers themselves, explores the diversity of learning and teaching environments, activities and participants, and then helpfully evaluates the types of evidence used to measure teaching quality in terms of: student voice; student outcomes; HEA fellowship and observation of teaching. Chapter 5 Developing and Supporting Teaching Excellence in Higher Education by Robson explores the need for recognition of diverse teaching excellence and approaches that promote development of teachers such as collaborative enquiry, reflection and dialogue. Having examined some of the fundamental problems and limitations around TEF, in the conclusion, French and O’Leary finish on a positive note: the TEF is encouraging debate around teaching excellence and shifting the power dynamic between research and teaching to give teaching more importantance. Patel E for Enigma: The Teaching Excellence Framework (TEF) and the challenges of measuring excellence Journal of Learning Development in Higher Education, Issue 12: November 2017 4 What are the socio-political drivers behind TEF? The authors put forward a consensus that some of the main drivers behind TEF are several beliefs held by the government. First, there appears to be a prevailing idea that higher education would be better if conceptualised and managed as a business, that knowledge is a tradable commodity and teachers ought to be held accountable for student satisfaction and performance. As such the TEF aims to determine whether teaching offers ‘value for money’. Second, that Higher Education needs to be ‘opened up’ to competition between existing and new institutions with reduced regulation and that students, as consumers, would benefit from a simple ranking system for teaching. French argues that far from opening up education, the TEF will perpetuate the imbalance that exists around institutional prestige and funding, and may actually reduce widening participation. Third, that teaching is of poor quality, lacks research or evidenced-based approaches, and that measuring and ranking institutions, and linking this to status and tuition fees, will help develop teaching practices. O’Leary points out that although successive Ministers of State for Universities and Science have claimed that teaching is weak, this is based on assumption, not evidence, and that overall NSS scores are increasing. Fourth, that a ‘New Public Management’ approach to running institutions, using big data analysis and performance management processes, will improve teaching standards. French suggests that it is hard to draw meaningful conclusions about teaching from big data sets that generalise across the institutions and, as with Research Excellence Framework (REF), it may be used for accountability, to compare institutional status, to influence funding and HEI teaching culture, and therefore it may lead to institutions gaming the system. This book is therefore invaluable in helping situate current government policy around HE and the development of the TEF in its political context and encourages us to consider the approaches, motives and discourse in a critical way. Can TEF measure teaching excellence? Again the authors are in agreement that, whilst TEF may help balance the undue focus on the REF and research, it falls short of measuring teaching excellence for many reasons and certainly does not help develop teaching. First, it is widely acknowledged in the literature that defining teaching excellence is difficult, if not impossible, as different individuals, disciplines and institutions view it in different ways. O’Leary illustrates this well Patel E for Enigma: The Teaching Excellence Framework (TEF) and the challenges of measuring excellence Journal of Learning Development in Higher Education, Issue 12: November 2017 5 by examining the range of learning environments, activities and agents that may be involved in pure or vocational subjects. Wood further argues that criteria for measuring teaching needs to be contextualised, alluding to differing disciplinary practices by quoting Readings (1996) that ‘an excellent boat is not excellent by the same criteria as an excellent plane’. Second, French and O’Leary both question whether the TEF matrices can be used as proxies for teaching. The use of the NSS to measure teaching is flawed as satisfaction is not learning - in fact the best learning experiences may be unsettling or challenging. Students may make unreliable judgements about teaching quality, having little experience of teaching in other institutions, or tending to poorly rate teachers or learning experiences that do not fit their expectations. O’Leary further points out that even experienced classroom observers with clear criteria find it challenging to consistently rate teaching, and questions how we can use student satisfaction when little is known about how they evaluate their educational experience. French raises important questions about using the DLHE as a proxy of teaching. It captures student destinations and wages six months after graduation but these outcomes are also dependent on institutional reputation, student socio-economic status and the likelihood of a degree leading to either immediate employment, as in teaching or nursing, or higher wages, as in accountancy or law degrees. Furthermore, measures such as student recruitment, retention, progression and attainment collected by the Higher Education Statistics Agency (HESA) are also weak proxies for teaching due to the multitude of other factors at work, not least of all a student’s own role in their learning. Additionally, these do not capture the value-added improvement that occurs during study in many widening participation cases, where a learning and teaching experience can be truly transformational. This book makes a thorough consideration of what these measures can and cannot tell us about teaching excellence. A further area of consideration that is not addressed in this work, but would be vital to this debate, is the validity of the grouping of universities within the benchmark system. An average of the metrics for the institutions within a group create the benchmark for that group, which all group members are then measured against. Those that fall far from the average receive a positive or negative flag, which ultimately determines the gold or bronze status of an institution (Bagshaw and Morris, 2016). However, an institution that is above the average of one group, may be below the average of a different group. In other words, an ‘average’ University compared to ‘inadequate’ universities could receive a gold award, but the same ‘average’ University compared to ‘excellent’ Universities could receive a bronze award. Therefore, we should ask to what extent would moving an institution from Patel E for Enigma: The Teaching Excellence Framework (TEF) and the challenges of measuring excellence Journal of Learning Development in Higher Education, Issue 12: November 2017 6 one group to another change its flags and therefore its rating? Furthermore, as this simple allocation to a group is critical in determining an institution’s flags - how are these groups decided? What approaches might be more meaningful in developing teaching excellence? Wood’s consideration of the language around TEF is insightful, suggesting that instead of being developmental, the discourse that TEF uses is about measurement, accountability and human capital. Thankfully, the authors discuss a range of evidence-based approaches that support the development of teaching and learning in a meaningful way but tend not to quantify it. Wood argues that teaching excellence emerges from an infinite number of complex and nuanced factors and therefore cannot be broken down into simplistic measures, likert scales or tick box criteria. Wood draws on complexity theory and process philosophy to develop the idea of ‘emergent pedagogies’. Teaching can be seen as a complex adaptive system, having many elements and interactions, such as learning, curriculum, assessment, students and teachers. The interactions are complex, disproportionate, there are both positive and negative feedback loops, boundaries are hard to define and the system is ever changing. As such we cannot hope to reduce it and maintain its meaning. Instead, Wood, drawing on Richardson, Cilliers and Lissack (2007) suggests that: …to understand them, at least in part, we need to use different perspectives to build ever richer, if incomplete, models of the system we are interested in. (P.58) Wood discusses five lenses or foci that can help the development of teaching– values, personal and collaborative growth, organisational and societal contexts. Robson also emphasises the value of collaboration in developing teaching. Collaborative networks for professional enquiry can support critical reflectivity, encourage engagement with literature, provide an opportunity for the development and reinterpretation of ideas around pedagogy, and help to develop a quality culture within institutions. This helps colleagues to appreciate the complexities around teaching, such as how it is situated within structures, policies and environments, and how agendas at local, institutional and national levels shape practice and identity. Leadership and recognition are also highlighted by O’Leary and Robson. Patel E for Enigma: The Teaching Excellence Framework (TEF) and the challenges of measuring excellence Journal of Learning Development in Higher Education, Issue 12: November 2017 7 O’Leary expresses concern over the Higher Education Academy (HEA) promoting leadership at more senior levels of the fellowship scheme. For Robson, however, leadership in teaching is not through managerial roles but the ability to inspire; to change practice and policy through a more democratic culture, encouraging a sense of agency and developing identity. Lastly, O’Leary suggests that whilst observation can be used as a performance management tool which tends not to develop teaching and can discourage innovation, it can be most effective as a low-stakes collaboration between peers and students, in supporting reflection and giving students agency in shaping their education. This book frames an excellent discussion of some of the key literature around teaching excellence and importantly suggests meaningful ways in which teaching can be developed. As learning developers, we not only have a stake in the way teaching approaches at our institutions are measured and ranked, and a need to have our role in supporting learning recognised, we also have valuable expertise in understanding how people, even teachers, learn. It is therefore important that we actively contribute to this ongoing debate. References Bagshaw, A. and Morris, D. (2016) Fun with flags: how metrics will work in TEF outcomes. Available at: http://wonkhe.com/blogs/policy-watch-fun-with-flags-the-role-of- metrics-in-tef-outcomes/ (Accessed: 21 October 2017). Higher Education Funding Council for England (HEFCE) (2017) About the TEF. Available at: http://www.hefce.ac.uk/lt/tef/whatistef/ (Accessed: 21 October 2017). Author details Dr Alex Patel is an Educational Developer at Leicester Learning Institute, at the University of Leicester. Her main interests are around developing student staff partnerships, and developmental approaches to academic and digital literacies. Due to her cross-disciplinary http://wonkhe.com/blogs/policy-watch-fun-with-flags-the-role-of-metrics-in-tef-outcomes/ http://wonkhe.com/blogs/policy-watch-fun-with-flags-the-role-of-metrics-in-tef-outcomes/ http://www.hefce.ac.uk/lt/tef/whatistef/ Patel E for Enigma: The Teaching Excellence Framework (TEF) and the challenges of measuring excellence Journal of Learning Development in Higher Education, Issue 12: November 2017 8 background, she is also interested in how we attempt to represent the world using different research methodologies. A book review of French, A. and O’Leary, M. (2017) Teaching Excellence in Higher Education: Challenges, Changes and the Teaching Excellence Framework. Bingley, UK: Emerald Publishing Limited. Introduction Author background Book structure What are the socio-political drivers behind TEF? Can TEF measure teaching excellence? What approaches might be more meaningful in developing teaching excellence? References Author details