TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017 TEF: why and how? Ideological and operational imperatives driving policy Ian McNay This short piece attempts to identify the origins of the Teaching Excellence Framework (TEF), to locate it within the wider framework of policy for higher education (HE) in the UK - more specifically England - to identify characteristics that will endure whatever tinkering at the edges happens as a result of the trial and error approach adopted towards many issues in contemporary politics, including the REF (McNay, 2016), which can be seen as a reference point for what we can anticipate. Government acknowledges that the approach will be applied to TEF: ‘we will continue to trial and pilot changes to ensure that the framework continues to improve’ (Department for Education, 2016, paragraph 7). The basic principle is that HE is seen as a business, operating in a competitive market, with universities described as ‘service delivery agents’ by one senior civil servant, and students, as customers, put ‘at the heart of the system’. Since government no longer funds teaching, except to top up costs of expensive essentials in STEM subjects, it is no longer a near monopoly client controlling through resource allocation policy, but has re-shaped itself as a students’ champion, a blend of the Consumers’ Association, sponsors of the Which? Guide to universities, and the Competition and Markets Authority, monitoring probity in provision and publicity. Its agent for this will be the new Office for Students, which at least is within the education ministry, not business, where research remains. The secondary principles underpin traditional Conservative attitudes since the collapse of Butskellism: - value for money – as in economy, efficiency, effectiveness, in that order; - a belief, if applied to higher education, demonstrated to be mistaken by Gareth Williams as long ago as 1992, that competition enhances quality and reduces costs (Williams, 1992); that is also false for other sectors as currently evidenced by prisons, forensic services, energy companies and transport provision. So, Jo Johnson (DBIS, 2015) thinks that new providers should be able to award their own degrees as soon as they open their doors – the level playing field syndrome - because such [unproven] ‘high quality challenger institutions… will add a positive competitive dynamic’ to the sector (Havergal, 2016a). That was the argument behind the polytechnics, and later the TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017 Open University, but they had a much long probationary period – the OU had an academic advisory committee for 6 years before operating with full autonomy; - a suspicion of professionals as autonomous ‘experts’, particularly those in public service, whose first loyalty should be to the state as employer, and compliance with its views, recently seen in attitudes to the judiciary supporting elective democracy within the Brexit process, and, ironically, ‘service providers’ in the central civil service and embassies world-wide. Value for money promoting efficiency, and accountability, curbing autonomy, were the initial drivers behind what has become the REF, to monitor whether academic professionals in the universities of that era were doing what they were paid to do - research. Many were not. The RQA/RAE/REF then conditioned the award of funds for research and distorted strategic and resource support away from teaching, still the second main expenditure across the HE sector, after administration. Before fees were re-introduced (they existed when I was a student in the mid-1960s) only four HEIs out of nearly 150 got more money from government for research than for teaching. The dominant discourse later became that teaching in HE was of poor quality because of government emphasis on research, though that causal link was denied. The White Paper (BIS, 2015) expresses a ‘concern that too often the incentive at an institutional and individual level skews activity away from teaching’ – with no acceptance of government responsibility for setting those incentives. We are to blame for a negative policy impact I identified for HEFCE 20 years ago (McNay, 1997), and which it acknowledged then. Bahram Bekhradnia, formerly the boss of HEFCE - which funded HE and had a duty to ensure quality - was more generally critical in saying that universities are ‘not very good at teaching’. He drew on various comparative projects across Europe and the wider world (Havergal, 2015). That may have been true in the Russell Group universities, as evidenced by a trial TEF run by Times Higher Education (THE), which put none in the top ten (Havergal, 2016b) and had several well into the bottom half of the league table - Bristol, King’s College, London (KCL), London School of Economics and Political Science (LSE), Goldsmiths, St. Andrews, Edinburgh were all ranked below Greenwich. Senior managers attributed this to the impact of REF – the Research Excellence Framework, a government policy, but with academics being blamed – and the solution seen as replicating that approach for assessing teaching excellence, when many academics believe that the REF and its antecedents have had significant negative impact (McNay, 1997, 2007, 2016) . The 2016 HEPI academic experience survey showed that student satisfaction is in decline and student assessment of value for money fell steeply, by 20 TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017 percentage points in England since fees were last trebled – another government policy (Neves and Hillman, 2016). It is worth noting that the THE survey of academic staff (Grove, 2017) showed that 39 per cent agreed that teaching was the most important function of a scholar, with only 24 per cent disagreeing. The same survey showed that 55 per cent of academic respondents believed that research is valued more highly than teaching in their institution. So, leaders and managers are also culpable, but not the teachers now being frameworked. The motivations driving entry to the REF were described by one of my favourite ex-vice- chancellors as ‘fame and fortune’ – esteem and funding, thereby combining intrinsic and extrinsic elements. This will also apply to TEF which will have a ranking and a reward. But, with a difference – politicians do learn, but slowly. The ranking will be initially at corporate level; and the reward will be paid by the students in even higher fees, or by teachers because managers will be allowed to increase student numbers. Recruitment of international students will also be conditional on a good grade. Those factors mean that 134 HEIs, mainly in England, where the policy operates, are taking part in year 2 of the exercise, despite staff attitudes, as Stuart Croft, VC of Warwick acknowledged in a letter to THE on 2 February, 2017, where his argument was simple: ‘the government has us over a barrel’. Officially, in England, there is no cap on numbers, but that may not last long when the size of the 18+age cohort expands rapidly soon after the start of the next decade. However, that will be balanced by leaving the EU and losing the obligation of give loans to people from 27 other countries, delighting the Thatcherites. Conditions of student financial support have become harsher and data on the ‘graduate premium’ suggest it is declining, also acknowledged by government (DBIS, 2015) so the calculation about whether to enter HE may change with the limit to numbers emerging from decisions by potential students. The other value for money lesson learned from REF is about the cost of the evaluation exercise itself. There has long been pressure to reduce peer assessment in REF and use metrics as the dominant evidence base for quality. For TEF, the search was for existing metrics, because there is not a tradition of peer review within politicians’ living memories. Some of us may be nostalgic for the Council for National Academic Awards, which validated polytechnic degrees, with its formative, developmental approach. Even Teaching Quality Assessment, the last experiment in this field, involved visits, observation and discussion as well as rooms full of paperwork, but it cost a lot to find very, very little to criticise: not the government expectation, so it was abandoned. TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017 Government acknowledges that there are no good metrics, but they are prepared to use bad ones, or proxies, to get something done, with, initially, little flexibility from a ‘one-size-fits-all’ core (Department for Education, 2016) beyond socio-economic background of students, and an institutional submission to stake a claim for excellence against institutional benchmarks. The chair of the TEF panel acknowledges that all data are flawed in some respect, especially those from the National Student Survey which are corrupted by student self-interest in rating highly the quality of their university and where 25 student unions are committed to a boycott because of the impact on TEF ratings (Grove, 2017a). There are some positives for universities that value teaching – greater parity of status between teaching and research careers, with explicit career paths and rewards for teaching, which may redress the distorting imbalance that has developed. The three main metrics concern teaching quality, learning environment, student outcomes and learning gains (DBIS, 2015, chapter 3). Teaching quality will use student satisfaction statistics, which risks popularity displacing quality, with negative effects. Learning gain is notoriously difficult to measure; some pilot, unpublished, work on this I did with John Pratt showed Oxbridge students had a learning loss, given that their high entry qualifications were not matched by high degree classification. Teaching excellence may be related to contact hours and student time spent studying as well as, possibly, the proportion of staff on permanent contracts. Study time depends on self-reporting, not a reliable process. Outcomes will use leaver destination statistics to measure high level skills development to promote social mobility and enhance productivity. The consultation showed only about 40 per cent of respondents supported several government proposals on this, but they will be retained despite that (Department of Education, 2016, paragraph 38). However, the quantifiable metric is salary, and recent HESA stats have shown that salaries in law are related to family background of students on entry. They will also vary by the differences between working in family law and corporate law, so skewing the advice given to students. HESA stats also show that the job market is institutionally racist and sexist, something often hidden by treating employment and further study as a single metric, when the second may be a fall-back after prejudiced decisions on the former. HEIs do, though, need to examine why black students on average gain lower final degree classifications than others with the same entry qualifications. The report on responses to the consultation raises issues about ideological drivers for proposals to support new market entrants. There were responses from 132 ‘state’ HEIs, of which 18 are quoted in the text; private alternative providers, including for profit organisations with foreign owners provided 21 responses with 8 quoted. In percentage terms, that is a quotation rate of 13 TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017 per cent and 38 per cent, three times as many for new entrants encouraged by government as for established HEIs. For student unions, representing those ‘at the heart of the system’, the rate is even worse: three quoted from 35 responses: 8.6 per cent. (Department of Education, 2016). In the end, only six alternative providers will take part in full this year (HEFCE, 2017), despite government sponsorship of them as high quality entrants. The RAE/REF has shown that any metrics adopted will be ‘gamed’. My work (McNay, 2016) has shown that research approaches become less innovative, more conformist to fit with perceived assessment panel prejudices. Hardly, then, an approach to encourage developments and diversity in teaching and learning: another unexpected negative consequence, moving authority to managers and away from academic professionals. The professionals do not think the TEF will achieve its aims. In the THE survey only four per cent of academics thought that the proposed framework will accurately assess teaching quality, with 75 per cent saying it will not. Similarly, 12 per cent believe it will improve quality, with 64 per cent saying it will not. Administrators were equally sceptical (Grove, 2017b). The record of institutional strategic leaders is not good: when high fees for taught courses came in, the extra funding was taken from teaching departments to spend on central marketing, and iconic buildings, not invested in the teaching process. Yet students’ top priority for savings, if needed, is…buildings (HEA/HEPI, 2016; Jones et al, 2016). TEF may see a transfer of funds from teaching departments to internal employment agencies, given a racist employment market. I know of one HEI that reduced its intake of BME students, as part of a tactic of raising its UCAS entry tariff, so as to improve its league table position. TEF plans are to reward widening access to counteract that, as part of a wider aim to double numbers of disadvantaged entrants, but that is hardly an indicator of teaching excellence at that stage, and is not affected by the quality of teaching staff activity. So, there is confusion, lack of clarity, but, as with research, emergent greater control. There is discontinuity built in: such uncertainty risks loss of innovation and creativity essential to quality improvement. As with the impact criterion for research quality, an aim of TEF is to change behaviour (DBIS, 2015). That change may be towards compliance, conformity and convergence to an isomorphic range of provision. At least we have been warned. I have tried to indicate what to expect, but, expect the unexpected as well. As one head of HEFCE said about the RAE: ‘You never know how it will all turn out’ (McNay, 1998). TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017 Reference list DBIS (2015) Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice, Cm 9141, London, Department of Business, Innovations and Skills. Department for Education (2016), Teaching Excellence Framework: year two and beyond. Government technical consultation response. London, Department for Education. Grove, J. (2017a) ’TEF to limit use of ‘flawed’ NSS scores’, Times Higher Education, 2 February, p8. Grove, J. (2017b) ‘We want to be happy in our work, but…’, Times Higher Education, 16 February, pp32-43. Havergal, C. (2015) ‘English universities ‘not very good at teaching’, says Hepi president’, www.timeshighereducation.com/news-teaching-excellence-framework (Accessed 15 December, 2016). Havergal, C. (2016a) ‘Absolute beginners: degree powers from day one’, Times Higher Education, 19 May, p7. Havergal, C. (2016b) ‘A new perspective’, Times Higher Education, 23 June, pp 36-49. Higher Education Academy/Higher Education Policy Institute (2016) ‘The 2016 Student Academic Experience Survey’, York/Oxford, HEA/HEPI. Higher Education Funding Council for England (2017), ‘299 universities and colleges take part in year 2 of the TEF’, www.hefce.ac.uk/news/newsarchivw/2017/ (Accessed 21 February 2017). Jones, S., Sutcliffe, M.J., Bragg, J., Harris, D. (2016) ‘To what extent is capital expenditure in UK higher education meeting the pedagogical needs of staff and students?’, Journal of Higher Education Policy and Management, 38(4): 477-489. McNay, I. (1997) The impact of the 1992 RAE on institutional and individual behaviour in English higher education: the evidence from a research project, Bristol, HEFCE. McNay, I. (1998) ‘The RAE and after: ‘You never know how it will all turn out’’, perspectives, 2(1). TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017 McNay, I. (2007) ‘Values, principles and integrity: academic and professional standards in UK higher education’, Higher Education Management and Policy, 19(3): 43-66. McNay, I. (2016) ‘Imbalancing the academy: the impact of research quality assessment’, Sociologia Italiana, 8: 119-150. Neves, J and Hillman, N. (2016), The 2016 Student Academic Experience Survey, Oxford, HEPI. Temple, P. (2013), ‘Aspects of UK private higher education’, in Callender, C. and Scott, P. (eds.) Browne and Beyond: Modernizing English higher education, London, IoE Press. Williams, G. (1992) Changing Patterns of Finance in Higher Education, Buckingham, SRHE/OpenUP. TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017 The Teaching Excellence Framework (TEF): yet more competition – and on the wrong things! Phil Race When the UK Green Paper which heralded the TEF was released in 2015, it claimed that the aim was to introduce something much less burdensome than the Research Excellence Framework (REF). Working in higher education institutions (HEIs) as a consultant, I see that the ‘less burdensome’ intention is already clearly failing, with the energy of staff at all levels being sapped by numerous planning and strategy meetings as institutions gear themselves up for the inevitable competition and league-table fallout which will accompany the TEF. Participating as a National Teaching Fellow in the various consultations preceding the present Higher Education bill, it was clear that there were problems with the three words involved in TEF. What are the best metrics to try to quantify ‘teaching’? In particular, what on earth might ‘excellence’ in teaching be, and how could this possibly be measured in a valid or reliable way? And even the word ‘framework’ implies a concept so complex that it could hardly be expected to be fair or productive. So how can we enhance the student experience of higher education? Higher education institutions and practices tend to change very slowly. Einstein is reputed to have said ‘it is sheer madness to keep doing the same thing, and to expect different results’. Therefore, if we want different and better results, we need to be doing different things, not just continuing to put the same old ways of doing things under spotlights, focusing on a few selected dimensions. Overburdened staff in HEIs are already burning themselves out, using traditional methods of assessment and feedback introduced long ago when student numbers were much smaller. Indeed, assessment and feedback take up far more time and energy than ‘teaching’ per se and additional TEF burdens may thus be a step too far. There are numerous ways to try to teach well. Any attempt to highlight particular aspects of good teaching will exclude other equally worthy ways of giving students a good higher education experience. But perhaps the biggest problem is with the word ‘excellence’. Rewarding excellence breeds competition and the losers, who will always be far more numerous than the winners, may well be disappointed and discouraged – and therefore less likely to try again to TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017 excel. Most of the benefits of collaboration may be lost when individuals, departments and institutions are made to compete. It would be much wiser to redirect the energy currently being expended on TEF to reviewing the processes of assessment and feedback. Successful educational outcomes, including the now oft-cited ‘longitudinal’ ones, depend very strongly on the quality of the assessment and of the consequent feedback that students experience, (e.g. HEA, 2012). The higher education sector still over-uses some processes, including the much-criticised unseen written exams (which often continue to measure what is remembered rather than what is learned), and coursework essays (despite all the research which shows that we are poor as a sector at grading them fairly or reliably, and despite ever-increasing concerns about who actually wrote them). Essays remain a good way to get formative feedback to students on their thinking and writing, but assessing them fairly is fraught with difficulty. Even though the National Student Survey, which feeds in to the TEF, has been adjusted for 2017 onwards and includes issues relating to the student experience of assessment and feedback, known as an aspect which students themselves find the least satisfactory, I would nevertheless argue that the TEF would do better to focus more on assessment rather than teaching. So, in short, the TEF seems set to measure the wrong things and to generate greater competition in a world which instead needs much more emphasis on collaboration. Rather than vain attempts to continue to do the same old things better, processes are needed to recognise and reward outstanding assessment and feedback practice, which itself needs to be better monitored, developed and changed to keep up with the present and future needs of our students, and of the world beyond HEIs. Reference list Higher Education Academy (2012) A Marked Improvement: transforming assessment in higher education. York: Higher Education Academy. Available at: https://www.heacademy.ac.uk/resource/marked-improvement-transforming-assessment-higher- education-assessment-review-tool (Accessed: 03 April 2017). TEF Special Edition Compass: Journal of Learning and Teaching, Vol 10, No 2, 2017