Iberica 13 ibérica 44 (2022): 345-368 iSSN: 1139-7241 / e-iSSN: 2340-2784 https://doi.org/10.17398/2340-2784.44.345 Abstract This study aims to explore the attitudes and practices of Spanish university lecturers towards their use of emI, and more specifically, their implicit and explicit assessment of their students’ english production. The research is based on an initial survey which was completed by teaching staff from a medium-sized university in madrid, as well as the findings of two subsequent focus groups consisting of a one-hour moderated discussion on the topic. The results were then processed through NVivo software. The participants in both the survey and the focus groups were drawn from a variety of academic disciplines and had different levels of experience in delivering their classes and their assessment in english. The most significant findings of the research were centred on the distinction between emI lecturers’ implicit and explicit assessment of their students’ use of english in their assignments and exams. Key words: emI, assessment, higher education, internationalisation, teacher and student perceptions. Resumen “Realmente no estoy evaluando el lenguaje” – Las percepciones del profesorado de materias impartidas en inglés acerca de su evaluación implícita y explícita de la producción lingüística del alumnado este estudio pretende explorar las actitudes y prácticas del profesorado universitario español hacia su uso del inglés como medio de instrucción (emI – English-medium instruction) y, en particular, su evaluación implícita y explícita de la “I’m really not assessing the language” - emI lecturers’ perceptions of their implicit and explicit assessment of students’ language production María Dolores Rodríguez Melchor & Andrew Samuel Walsh Universidad Pontificia de Comillas (Spain) drm@comillas.edu & aswalsh@comillas.edu 345 Ibérica 44 (2022): 00-00 mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH producción en inglés del alumnado. La investigación se basa en una encuesta inicial realizada entre el personal docente de una universidad de tamaño medio en madrid y en los resultados de dos grupos focales a posteriori que consistieron en una discusión estructurada de una hora sobre el tema. Los resultados se procesaron con el software NVivo. Los participantes, tanto en la encuesta como en los grupos focales, procedían de diversas disciplinas académicas y tenían distintos niveles de experiencia en la impartición de sus clases y su evaluación en inglés. Los hallazgos más significativos de la investigación se centraron en la distinción entre la evaluación implícita y explícita del profesorado emI sobre el uso del inglés por parte del alumnado en sus trabajos y exámenes. Palabras clave: Inglés como medio de instrucción, educación superior, internacionalización, percepciones del profesorado y del alumnado. 1. Introduction – emI in Higher education Over the last two decades, there has been an exponential growth both in the practice of and the demand for english medium Instruction (emI) in Higher education throughout the non-english speaking world. As far back as 2006, Coleman established seven driving forces that underlie the expansion of emI at universities: internationalisation of higher education, student exchanges, teaching and research materials, staff mobility, graduate employability, the market in international students, and Content and Language Integrated Learning (2006, p. 4). more recently, further studies (Costa & Coleman, 2013) have confirmed the upward trend in demand for english-taught programmes in european universities. A particularly notable and influential case of this exponential growth of emI in higher education is that of maastricht University in the Netherlands. This institution began life in 1976 and opted immediately to combine the approaches of Problem Based Learning (PBL) and emI and now teaches over half of its degree programmes and the vast majority of its doctoral programmes in english to a student body which has around 55% of international students. There has also been a long and strong emI tradition at universities in the Nordic Countries (Finland, for example, is currently ranked first in terms of the percentage of higher education institutions that offer degree programmes in english) as well as in the Baltic nations and the aforementioned case of the Netherlands. On the other hand, emI in higher education is still a much newer and, to some extent, disputed phenomenon in Southern europe (Pulcini & Campagna, 2015),1 even leading some authors such as Dimova, Jensen and 346 Hultgren (2015) to postulate a North-South emI divide in the european Higher education Area. In the specific case of Spain there has been a vertiginous increase in university degrees offered in english over the last five years, in both public and private institutions, although most notably in the latter. This has happened, to some extent, as a direct consequence of the Bologna Process and the Internationalisation of Higher education which, at least in Spain, has tended to mean the de facto Anglicisation of this field due to economic, social and political reasons, all linked intrinsically to the demand for this type of education manifested by students and their parents. Nevertheless, some Spanish university lecturers have expressed their reservations about this process and their concerns about their own command of english, or at least their students’ perception of their linguistic competence (Ball & Lindsay, 2013). The university at which this study of emI assessment was conducted is located in the Community of madrid which, unlike other autonomous regions of Spain such as Catalonia and the Basque Country, does not have its own co-official language and instead is officially a monolingual, Spanish- speaking region. The higher education institution where this research took place offers a number of degree programmes which are taught either completely or partially through emI. In the academic year of 2019-2020, just before the onset of the COvID-19 pandemic and its severe restrictions on international travel and student mobility, the university had 7,814 undergraduate students, 4,028 graduate students and also welcomed 1,358 international students (11.46%).2 Although emI has been widely implemented de facto throughout the institution, and also bearing in mind that many of the undergraduates and particularly the postgraduate students are not native Spanish speakers, it is perhaps surprising to find that there is currently no official Language Plan at this university setting guidelines for emI. This represents a significant contrast to Catalan universities with a comparable size and academic profile which, given the fact that they work in a bilingual context, tend to attach great importance to these plans and, more specifically, to the positioning of english and emI within their own linguistic objectives. In this regard, the following statement from the 2016-2020 Language Plan of the Autonomous University of Barcelona (an institution whose research performance has ensured that it has consistently been the highest ranked Spanish university in international tables) is rather eloquent: “english has become the lingua franca of the international academic community and is essential for attracting and retaining talent, achieving the “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 347 University’s objectives of internationalisation and excellence.”3 It would appear that universities based in the Community of madrid and the academics who research and teach there do not tend to share the concerns about domain loss due to the spread of emI (Lasagabaster, 2015) experienced in the autonomous regions of Spain with other co-official languages (Doiz et al., 2013). On the contrary, the use of emI at madrid universities is not perceived as a threat (Phillipson, 2015) to the autochthonous language and instead is promoted: encouraging internationalisation is regarded by the universities as a key strategy (Dafouz et al., 2014). The widespread implementation of emI at any institution of higher education evidently has major implications in terms of the selection of shared syllabus contents, materials and methodology, cooperation among teaching staff, as well as a considerable degree of english language support, which has been one of the keys to successful emI implementation (Breeze & Sancho guinda, 2021). The implementation of emI by universities in non- english-speaking countries pursues several discrete but complementary objectives such as attracting international students (particularly at postgraduate level), maintaining a high position in terms of international global academic ranking, and offering a globalised form of education to local students that will stand them in good stead in a labour market which increasingly demands a multilingual rather than a monolingual skill set. Thus, as well as, the obvious market forces of supply and demand, the exponential spread of emI responds to the ever increasing need to focus on the strategic positioning of institutions. This article proposes a study of assessment in the context of emI based on a broad-ranging survey and two focus groups conducted among lecturers who teach on a wide variety of degrees (Humanities, Law, Business, engineering and Nursing) at a medium-sized, private university in the city of madrid (Spain). This study aimed to find out whether these lecturers also assess english skills in addition to the content of their courses and, if so, to collect, analyse, and interpret their criteria regarding assessing, correcting, and grading their students’ use of both written and spoken english in those subjects which are taught, continuously assessed, and finally examined in english. mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368348 2. Background –Assessment in emI There is now an ample and growing bibliography on emI in higher education, both in the specific case of Spain, and in the broader international context (Shohamy, 2012; Dimova et al., 2015). However, in our view, one of the aspects of this phenomenon which has currently received comparatively scant attention is assessment, particularly in relation to the language assessment criteria that could be applied in the case of use of academic english as a second language by non-native students and lecturers. Lo has very recently called attention to this lack of specific research and consequent bibliography in this specific aspect of emI, stating that: with the increasing of english medium Instruction (emI) in bilingual and multilingual contexts, plenty of research has been conducted to examine students’ achievements, classroom interaction and, more recently, teacher education. However, research focusing on assessment issues in emI remains scarce […] there is an urgent need to address the research gap on emI assessment (2022, p. 61). The same author also points out that “considering the fact that students are inevitably assessed of their english knowledge in conjunction with their content knowledge in emI assessment, there are concerns about whether students’ learning progress is accurately measured” (2022, 61), and Lo also points to a need to address both “the validity and fairness” of emI assessment, questions which have been under-researched (Shaw & Imam, 2013; massler et al., 2014; Otto & estrada, 2019). Since assessment is a key element in the validation of any academic programme of studies whether it be continuous, formative or final, it is a sine que non in terms of both the prestige of these university studies and the ultimate employability of the graduates who attain them. Therefore, it is surprising that it has thus far merited so little research interest in the field of emI. This attention has largely focused on questions of didactics, language policy matters and broader issues related to the inherent tensions between the rise of english as a lingua franca in the academic context and the protection of mother tongues and the promotion of multilingualism. All of these issues will continue to merit attention and will undoubtedly generate further research, but this article seeks rather to provide a contribution towards addressing that very research gap in language assessment highlighted by Lo. Through an empirical study of emI assessment practices in a “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 349 representative european higher education institution, we aim to provide some practical insights into these questions, formulate some observations regarding the issues they generate, and identify some broad and overarching tendencies in terms of emI assessment in higher education that can expand this specific aspect of the field. Despite this aforementioned lack of research into emI assessment, there is an enormous and growing bibliography on assessment in higher education and constitutes the main topic of certain high impact academic journals.4 There is a consensus regarding the vital role played by assessment in Higher education in general, and as Higher education is increasingly delivered through emI it is somewhat surprising to find this lack of academic attention to what is undoubtedly an extra layer of complexity added to the already difficult but essential task of evaluating students’ learning and progress. moreover, one of the fundamental purposes of assessment is not only to validate this learning and provide students with officially recognised qualifications to help them enter the job market, but also to provide them with valuable feedback through formative assessment. This makes the lack of research in the field doubly frustrating. In order to improve assessment methods and outcomes in emI, it is clearly essential to pay critical attention to the methodology and the criteria used in this practice. Indeed, in the aforementioned voluminous bibliography regarding assessment in higher education, critical attention has been paid to questions such as “unconscious bias” in university assessment. In the multilingual, multicultural context of an emI university classroom, in which there are frequently quite diverse levels of english language competence between the students and the lecturers (Dafouz milne & Núñez Perucha, 2010), the existence of often quite deep- seated linguistic and cultural prejudices that lead to “unconscious bias” on both sides is an issue that inevitably tends to inform and perhaps even distort assessment, and one which clearly needs to be addressed. Particularly in the cases of business and law schools, there are a series of additional challenges involved in english language assessment which essentially lie in the tension between preparing students properly for the workplace and offering graduate students “value for money”, whilst simultaneously respecting the integrity of robust academic assessment (Dafouz et al., 2014). mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368350 3. methodology This paper reports on the results of an online survey and two focus groups carried out in the second quarter of 2021 which sought to inquire about language assessment in english mediated undergraduate and graduate instruction at a representative Spanish university. This twofold approach stems not only from the need to gather more detailed information and feedback than a traditional online survey can necessarily provide, but also from the wish to delve more deeply into the study of emI assessment by examining what individual teachers consider to be more relevant to their personal classroom experiences in emI. The methodology used was based on the analysis of an online survey circulated among the emI teaching staff at this institution, and then two focus groups composed respectively of five and four lecturers chosen from the institution’s four main faculties (Humanities, Law and Business, engineering, and Nursing). we believe that this study can provide insights into the nature of the emI assessment conducted by lecturers at a representative european university and serve to offer some recommendations for improvement in terms of methodology and coordination. Our research questions were aimed at finding out how the lecturers in our study sample approached emI assessment. what criteria do they use to assess their students’ lexical range and use of appropriate register and grammar in writing? In oral presentations, do they grade more highly for correct pronunciation or for overall communicative competence? Do they distinguish between native and clearly non-native pronunciation? As this university has a mix of native and non-native teaching staff, we were interested in finding out whether native-speaking teachers are more tolerant of their students’ mistakes than their non-native colleagues, and we wanted to test this hypothesis with the respondents to our survey and also with the subsequent focus groups. This article also sought to observe if there were any notable differences in these assessment criteria between the Sciences and the Humanities, and also if there was any significant disparity based on variables such as age, gender, length of emI teaching experience, accredited level of english on the CeFr, and also experience in teaching and research abroad in an english-speaking environment. we also asked the participants about the degree of homogeneity regarding criteria that might eventually exist between Departments, Faculties, or individual degree programmes for grading in emI subjects, what rubrics they use, if any, to assess both writing and speaking, and if these are co-created and shared with their colleagues. “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 351 The anonymous survey was conducted by means of a web-based questionnaire generated by microsoft Forms, and potential participants were selected among those teachers whose subjects were taught in english, excluding general english language courses, to ensure a fair distribution across disciplines. The teaching staff were contacted initially by e-mail and asked to participate in the survey and also to indicate their willingness to volunteer for one of the two subsequent focus groups. They were informed that the aim of this research study was to compare how students’ language competences were assessed depending on whether they and their teachers had english or Spanish as their first language in emI courses across disciplines at this university. They were also clearly instructed to reply to the questionnaire only if they taught in english. The questionnaire consisted of 29 items, of which Questions 1-16 were intended to obtain an accurate description of the populational sample in terms of age, gender, nationality, working languages, teaching experience, official english qualifications/accreditations, and the percentage of their classes that were taught in english. Questions 17 to 25 inquired about the use of rubrics and the importance that participants attached to the assessment of issues such as grammatical correction, appropriate register, pronunciation, fluency, spelling, cohesion and coherence, textual organisation, and the use of correct terminology. Finally, Questions 26 to 29 referred to the degree of tolerance that participants attached to and perceived from the students, depending on whether their first language was english or not. Nine respondents signed up for the subsequent focus groups, which were conducted through the Teams platform on two separate occasions, depending on the participants’ availability and seeking to ensure a fair and representative distribution across faculties and disciplines. The group discussions were structured around the following questions: 1. Do you approach teaching/assessing differently depending on the language of instruction? 2. Do you think there is any difference in the way the Spanish students and the international students respond to the classes you teach in english/Spanish? 3. Are you more or less tolerant of the language mistakes of a) native speakers b) non-native speakers? 4. Leaving aside the knowledge of the subject, which language assessment criteria do you think are the most important? mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368352 5. In your opinion, what is the purpose of using english as a medium of instruction in the current context of the Spanish university? Both sessions were recorded with the microsoft Stream tool and then transcribed, revised, annotated and uploaded to the NVivo qualitative data- analysis software, through which we proceeded to create nodes to house relevant excerpts from the transcripts and cases for each participant in our focus groups. 4. Results of the survey Description of the sample Thirty-four respondents completed the questionnaire. The vast majority (30) of participants were Spanish (these represented 85.29% of the total sample) and also declared Spanish to be their mother tongue. Two respondents were Italian, one was german and one British. All of them declared a second working language, and not surprisingly in this case english was the most frequent option (30 respondents), although two participants specified that they had the same command of both english and Spanish. Nineteen respondents indicated a third working language, but in three of these cases a lack of fluency or other limitations were mentioned. In terms of gender, the sample was balanced with 50% male and 50% female participants. Their most frequent age range was from 46 to 55 (12 participants) and their length of emI teaching experience was notably heterogeneous, with the greatest number of respondents citing 3-5 years of teaching experience in english (9 participants) or 6-10 years (9 participants). eighteen (52.94%) of them also reported that they had teaching experience outside Spain in various contexts. The majority of participants (17) belonged to the Faculty of Law and economics, and most of the respondents (20) taught courses on more than one degree (in fact, there was such a wide variety of answers to Question 5, What degrees do you teach on, that we decided not to take this item into consideration in our analysis). eighteen respondents held official english language qualification and, in this respect, Cambridge english Advanced C1 (5 respondents) and Proficiency C2 (6 respondents) were the most frequent examples mentioned, although other options such as IeLTS, TeFL or the language certificates awarded by the Escuela Oficial de Idiomas (Official Language Schools which in Spain are run by regional governments) were also reported. It is worth mentioning that almost half (47.05%) of the “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 353 participants (16 in total) did not possess any english language certificate. A large proportion of respondents stated that they taught at least 75% of their courses in english. Table 1 presents a breakdown summarising those descriptive statistics that are more relevant for the purposes of this study. Table 1: Participant background information. Rubrics and assessment criteria Questions 17 to 25 sought to ascertain whether the participants used rubrics to assess english language related parameters, and what importance they attributed in their global assessment to six specific language criteria on a scale from 1 to 10 (1 being not important and 10 extremely important). what is striking at first glance is that 20 out of 34 respondents reported that they did not include in their rubrics any other criteria other than those mentioned in Questions 17 to 25, thus implying that they did assess language-specific criteria in their rubrics. Only two participants pointed out that their rubrics did not include any language criteria at all. when we compared this result mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368354 ! Question 1 (N = 34) 25-35 36-45 46-55 56-65 Over 65 1. Age 4 (11.76%) 11 (32.35%) 12 (35.29%) 6 (17.64%) 1 (2.94%) Question 2 (N = 34) Male Female 2. Gender 17 (50%) 17 (50%) Question 3 (N=34) Spanish Italian Hungarian British 3. Nationality 30 (85.29%) 2 (5.88%) 1 (2.94%) 1 (2.94%) Question 4 (N = 34) Law/Econ. Engineering Social / Hum. Sc Health Sc Theology 4. Faculty 17 (50%) 5 (14.70%) 10 (29.41%) 1 (2.94%). 1 (2.94%) Question 6 (N = 34) Spanish English German Hungarian Italian 6. Mother tongue 29 (85.29%) 1 (2,94%) 2 (5.88%) 1 (2.94%) 1 (2.94%) Question 9 (N = 34) -3 years 3-5 y. 6-10 y. 11-15 y. 16-20 y. +20 years 9. Teaching experience in English 8 (23.52%) 9 (26.47%) 9 (26.47%) 3 (8.82%) 1 (2.94%) 4 (11.76%) Questions 12/13 (N = 34) C1 C2 Others None 12/13. English language qualification 5 (14.70%) 6 (17.64%) 7 (20.58%) 16 (47.05%) Question 14 (N = 34) 100% 75-95% 50-75% -50% 14. Percentage of classes in English 8 (23.52%) 13 (38.23%) 4 (11.76%) 9 (26.47%) with the answers elicited in the focus groups, our conclusion was that Question 25 (Do your assessment rubrics contain any concepts not mentioned above? If so, which ones?) must have been misunderstood by some of the participants. Those respondents who mentioned other non-language criteria referred to parameters such as knowledge of the subject content, citation and referencing, critical thinking, and argumentation. Table 2 presents the replies for Questions 17 to 25 regarding rubrics and language assessment criteria. Table 2. Questions 17 to 25. when comparing these results with the remarks presented in the focus group discussions, we observed certain inconsistencies. On the one hand, and as we will explore in further detail in the next section, the participants in the focus groups stated quite categorically that they did not assess according to language criteria. On the other hand, when faced with the task of attributing “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 355 ! Question 17 (N = 34) 1 2 3 4 5 6 7 8 9 10 17. Grammatical correction 2 1 0 2 2 4 6 5 8 4 Question 18 (N = 34) 1 2 3 4 5 6 7 8 9 10 18.Appropriate register 1 0 1 0 3 3 6 8 7 5 Question 19 (N = 34) 1 2 3 4 5 6 7 8 9 10 19. Pronunciation 4 2 0 0 5 4 9 5 4 1 Question 20 (N = 34) 1 2 3 4 5 6 7 8 9 10 20. Fluency 1 1 0 1 5 3 7 6 5 5 Question 21 (N = 34) 1 2 3 4 5 6 7 8 9 10 21. Spelling 2 2 1 4 4 2 5 7 7 3 Question 22 (N = 34) 1 2 3 4 5 6 7 8 9 10 22. Cohesion & coherence 0 0 0 0 3 1 5 2 13 10 Question 23 (N = 34) 1 2 3 4 5 6 7 8 9 10 23. Textual organisation 0 0 0 0 2 1 3 8 10 10 Question 24 (N = 34) 1 2 3 4 5 6 7 8 9 10 24. Correct terminology 1 0 0 0 0 1 6 7 8 11 Question 25 (N = 34) Yes No Rubrics do not include any language criteria 25. Rubrics include other criteria 22 20 2 a value to the importance of language criteria in their own system of overall assessment, more respondents selected values over 5 than under 5 and, in some cases, they even assigned medium or high scores to certain highly relevant linguistic parameters, such as correct grammar, where half of the participants (17) gave values of 8 to 10, or pronunciation, to which 9 respondents (26.47%) attributed a value of 7 and ten (29.41%) marked even higher with values of 8, 9 and 10. None of the respondents awarded a score of below five for two criteria, namely cohesion and coherence, and textual organisation. And only one respondent attributed a value of 1 to the importance attached in their assessment to correct terminology, which is in line with the findings of the focus groups. Table 3 summarises the information collected for the importance of language assessment criteria, aggregated into three items: low (scores 1 to 4), medium (scores 5 to 7) and high (scores 8 to 10). Table 3. Summary importance of language criteria (in bold ≥ 50% respondents gave attributed values from 8 to 10). All in all, more than 50% of the respondents attributed a high degree of importance (scores ranging from 8 to 10) to six of the selected language assessment criteria: correct grammar, appropriate register, spelling, cohesion and coherence, textual organisation and correct terminology, but also scored extremely high on the last three criteria, which are related to conveying subject-appropriate meaning. This could indicate that content and language aspects are combined in the assessment. Tolerance of language mistakes Questions 26 to 29 were specifically aimed at ascertaining whether participants were more or less tolerant of the students’ language mistakes depending on the first language of the students. In the case of respondents who named english as their first language we observed a clear inconsistency, mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368356 ! Questions 17 to 24 (N = 34) Low importance (1-4) Medium importance (5-7) High importance (8-10) 17. Grammatical correction 4 (11.76%) 12 (35.29%) 17 (50%) 18.Appropriate register 2 (5.88%) 12 (35.29%) 20 (58.82) 19. Pronunciation 6 (17.64%) 18 (52.94%) 10 (29.41%) 20. Fluency 3 (8.82%) 15 (44.11%) 16 (47.05%) 21. Spelling 9 (26.47%) 11 (32.35%) 17 (50%) 22. Cohesion & coherence 0 (0%) 9 (26.47%) 25 (73.52%) 23. Textual organisation 0 (0%) 6 (17.64%) 28 (82.35%) 24. Correct terminology 1 (2.94%) 7 (20.58%) 26 (76.47%) since 14 participants declared that english was their first language in their answers to Question 26, when only one respondent had reported english as their mother tongue in Question 6. In contrast, the same number of respondents (29) reported Spanish as their mother tongue in Questions 6 and 28. In the case of Question 27, 28 respondents also stated that english was not their first language, which in conjunction with Question 26, does not add up to the number of total participants (34). we could speculate here that, once again, the question was misunderstood and that what respondents meant by their “first language” was that in which their courses were taught. Perhaps the question should have been phrased more explicitly. In spite of this incongruity, the majority of participants stated that they were less tolerant of language mistakes, both in Spanish and english, when students were using their first language, as shown in Table 4. Table 4. Tolerance of mistakes. 5. Results of the focus groups Description of participants and outputs The two focus groups were conducted at two different times and on two different days, based on the availability of the participants, but nevertheless ensuring adequate distribution of specialisations and subjects represented. An interesting feature of the dynamics of the two groups was the language in which the participants chose to communicate. In both cases, they were given the freedom to express themselves in either Spanish or english, “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 357 ! Question 26 (N = 14) Less tolerant of EN language mistakes Students’ first language EN Students’ first language not EN 26. If English your first language 10 4 Question 27 (N = 30) Less tolerant of EN language mistakes Students’ first language EN Students’ first language not EN 27. If English not your first language 26 4 Question 28 (N = 29) Less tolerant of ES language mistakes Students’ first language ES Students’ first language not ES 28. If Spanish your first language 28 1 Question 27 (N = 12) Less tolerant of ES language mistakes Students’ first language ES Students’ first language not ES 29. If Spanish is not your first language 11 1 although the questions that guided the discussion were only asked in english. However, in the first group the language used in all but one of the contributions was Spanish, while the opposite happened in the second group. The length of the discussions was very similar in both: 43:41 minutes in the first case, and 42:00 minutes in the second. In the first focus group, 5 lecturers (2 male and 3 female) from three different faculties participated: Human and Social Sciences, Law, and engineering. The second focus group was composed of 4 lecturers (2 male and 2 female) from: Human and Social Sciences, economics, and Law. regarding the outputs, in the first case the number of words uttered by the participants was 6,730 and in the second case 5,363, which is consistent with the number of participants in each group. As explained above (Section 3), the moderators structured the discussion around 5 questions, but participants were free to add information if they considered it relevant. NVivo processing and results Once the verbatim transcripts of the recordings of both focus groups were made, they were loaded into the qualitative data analysis software NVivo and coded. In total, 5 main nodes were created (different approaches depending on language, different response depending on students, language assessment criteria, purpose of emI and tolerance to language mistakes), corresponding to the questions that structured the discussions. These were, in turn, split up into 20 sub-divisions: • different approaches depending on language - different approach to teaching and assessment - undifferentiated approach to teaching and assessment • different response depending on students - American students - Asian students - european students - French – Belgian – german - Spanish students • language assessment - communication mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368358 - handwriting - implicit bias - only content - organisation - register - soft skills • purpose of emI - cultural exchange - english skills - internationalisation - professional • tolerance of language mistakes - students’ tolerance of teachers - teachers’ tolerance of students Figures 1 and 2 show the breakdown of the nodes according to their frequency in each focus group transcript. As we can see, in both cases and despite the fact that there was no communication between the participants in the two focus groups and these issues were not raised by the moderators, the participants introduced two new aspects into the discussion, namely the differences in learning approaches between students of various nationalities and the students’ tolerance of the teacher’s language errors. Comments on the differences in learning approach observed between American, Asian, Spanish and european students are derived from Question 2, which asked about international or non-international students’ responses to classes taught in english. In both focus groups, teachers reported marked differences in the learning style and approach of American students, noting that they participate more, prefer to be given work to do outside the university classroom, and require an adaptation of the methodology towards less rote and more collaborative learning. As Figures 1 and 2 show, comments on this issue reached 4.43% and 4.66% coverage respectively, and in both cases these comments were more frequent than those made about assessment criteria and implicit bias, amounting to 13 explicit and abundantly developed “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 359 references in both transcripts. This statement from Participant 2 of group 2 illustrates this perception: […] Americans they participate much more than Spaniards. you know in a completely different way. Spanish students are always participating like “I’m not completely sure if what I’m gonna say is right or wrong” so, they are more cautious when participating. Americans they are like “whatever I have in my head, I’m gonna say” And if it is wrong, Ok, I will learn, and if it is right, Ok. Incidentally, the learning problems of Asian (Focus group 1) or French- speaking (Focus group 2) students are also explicitly mentioned, but in both cases related to what was perceived to be a poor level of english which prevents them from keeping up properly with the pace of the classes, a problem compounded by the fact that they also do not speak enough Spanish to understand instructions given in that language. german students were also mentioned, but in this case, they were described as ‘very stimulant (sic) and participative’ (Participant 1, Focus group 2). Figure 1. Frequency of references related to NVivo subdivisions in Focus Group 1. Furthermore, the comments about the students’ level of tolerance concerning their teachers’ english language mistakes could be interpreted as the result of a misunderstanding of questions 27 to 29, related to the level of tolerance of language mistakes. This question actually sought to find out whether teachers were more tolerant of the language errors made by students using either their native language or a second language, not the mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368360 ! ! T ! S ! I ! T ! a ! p ! f ! n ! n ! a ! n ! g ! s ! o ! w ! s ! e ! a ! y ! ! ! ! ! k ! ! ! ! ! ! ! ! other way around. However, it is significant that in both groups the parallel issue was spontaneously mentioned, with a coverage of 3.43% in the first case and 2.68% in the second. In the two transcripts, teachers’ tolerance of pupils’ mistakes was mentioned 13 times, whereas pupils’ tolerance of teachers’ mistakes was mentioned 8 times. Figure 2. Frequency of references related to NVivo subdivisions in Focus Group 2. with regard to the language assessment criteria used by the teachers, and despite having previously responded to a survey in which these were broken down in great detail and in which high values were given to parameters such as correct grammar (see Table 3), the participants in both groups agreed that their main concern when assessing was content and that they did not attach much importance to linguistic criteria apart from appropriate register or correct textual organisation. This is in line with previous research on emI lecturers’ beliefs, where it has often been found that emI lecturers state they are not language teachers (mancho-Barés & Aguilar-Pérez, 2020; roothooft, 2019). Some also highlighted the importance of soft skills, mainly verbal and non-verbal communication. Figure 3 shows the division of the main node devoted to language assessment criteria, comparing the number of references for each sub-division with the rest in both focus groups. As we can see, the participants stated with the same frequency that they only assess on the basis of content and that they, therefore, do not use rubrics to assess language, but they also agree that, if they were used, they would be common and shared. They also recognised that their assessment has an implicit bias that may be related to linguistic factors (5 references in each case) and, four “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 361 ! ! ! ! S ! ! R ! ! e ! ! h ! ! u ! ! g ! ! r ! ! r ! ! g ! ! w ! ! ! ! ! ! ! ! d ! ! ! ! ! ! ! ! g ! ! ! ! ! ! ! ! n ! ! t ! ! h ! ! c ! ! a ! ! e ! ! r ! ! n ! ! u ! ! f ! ! u respondents even asserted that this implicit bias is related to the readability of students’ handwriting: I don’t assess the english at all, let’s say explicitly […] And there are many times that implicitly you are assessing the english, because they are not expressing pretty well. So, whatever they are writing, it’s pretty difficult to understand. (Participant 2, Focus group 2) Figure 3. References to Language Assessment in Focus Groups 1 and 2. Apart from content, communicative success seems to be a determining factor in assessing both written and oral students’ assignments: The same with me. It’s communicative success. If the student is able to transmit what needs to be transmitted, I take no points off whatsoever for language use. I think I can’t really, because of the guía docente [syllabus] does not specifically include english language production as a part of the assessment criteria. I think it will be highly unfair to me to establish that as part of the grade if really, honestly, that has not been written anywhere. […] They know that they’ll need to communicate in english, so my answer is no, I do not assess language in those cases. (Participant 2, Focus group 1) At any rate, implicit bias related to linguistic criteria was mainly mentioned with reference to the teacher’s general understanding of students’ answers in tests. If this answer is not understood due to a lack of language proficiency, then the mark will be lower or, as Participant 3 in Focus group 2 put it: mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368362 ! ! T ! S ! I ! T ! g ! , ! y ! e ! o ! ! ! ! ! ! ! ! ! ! ! f ! F ! ! 2 ! n ! b ! h ! ] ! n ! g ! n ! r ! n ! s ! h […] of course, when you have 50 exams to mark. I’m very sorry, but this whole paragraph I can’t understand it. Ok, better, I cannot say if you were thinking something that was correct, but you wrote something incorrect. Therefore, this is wrong. Know what I mean? So, I would say explicitly no, implicitly yes. Because if they do not express themselves well, you cannot understand their answer. Finally, when asked about their opinion of the purpose of the use of emI in the current Spanish university context, the participants in both groups declared that the main objectives are internationalisation (8 references), followed by training students for future work in a professional environment where english is a lingua franca (6 references) and, in third place, improving students’ english language skills, although in 3 of the 5 references where this was mentioned, the participants explicitly stated that, although this may have been the initial objective, it is probably not the most important nowadays. 6. Discussion The majority of participants in the survey (17) were lecturers from the Faculty of Law and economics, suggesting not only greater implementation of emI in these parallel disciplines, but also stronger interest in reflecting upon emI practices in general and assessment in particular. we found no evidence that native-speaking teachers are more tolerant of students’ mistakes than their non-native colleagues. In fact, the vast majority of our respondents were non-natives but expressed the same standards and linguistic expectations in their assessment in english as in Spanish. However, this criterion did not apply, and was in fact radically different, when the students were writing or speaking in their own first language, be that Spanish or english. In this scenario, all of the respondents were notably less “tolerant”. Nor did we observe any significant disparity in the findings based on variables such as age, gender, or length of emI teaching experience. Several participants in the focus groups pointed to the importance of sharing and using common rubrics for assessment in emI. Another common strand of thought focused on the perceived need to apply these rubrics and overall assessment criteria to both english and Spanish, invoking the need for coherence and their insistence on the use of a proper academic and professional register in both cases. As other authors have found in their analysis of the international classroom, there are considerable differences in “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 363 learning style preferences (ramburuth & mcCormick, 2001) among students who are not only from different nationalities but also from radically different social and educational cultures. In this study, one of the broadest divisions was observed between european students, who were found to be rather passive (with some exceptions like the germans) and students from the USA, whose high degree of participation and willingness to ask questions and generally engage in the class were notable. The latter understood the ‘point’ of receiving classes from an expert who happens not to be a native english speaker and therefore feel no need to judge this expert’s level of english, but rather the quality of the teaching and the contents. Nevertheless, the most striking point of convergence which was elicited from the focus groups was the recognition that, at least implicitly, they were always somehow assessing their students’ use of language in areas such as register or cohesion and coherence of written assignments. Interestingly, most participants agreed that this was exactly what they do when they teach in Spanish, and they all admitted upon reflection that they were not prepared to “lower the bar” beyond a certain point in terms of written expression in either language, starting from the basic assumption that if a text is not comprehensible, the student who wrote it simply cannot pass that exam. 7. Conclusions This study aimed to explore the attitudes and practices of Spanish university lecturers towards their assessment of the students’ use of english in emI. Our findings reveal that the survey respondents attach great importance to linguistic coherence and consistency, textual organisation and correct terminology, but not to pronunciation or spelling. No differences in their assessment approach were revealed in terms of their emI teaching experience or their accredited level of english. It was also not possible to identify differences in language assessment approach between lecturers from different faculties, as most of the participants in the research were from the Faculty of Law and economics. rubrics used only covered content and not linguistic criteria. On the other hand, although the lecturers declared that they only assessed content and not language, the focus groups revealed an implicit bias in their assessment depending on the students’ communicative ability in english, mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368364 with respect to the textual organisation and clarity of their assignments and exams, and their general capacity to adequately convey content meaning. regarding pedagogical implications, we recommend that universities should review their emI assessment policies, in order to attain several key objectives: provide students with better feedback and formative assessment; position themselves better in terms of their international strategies; enhance the quality of assessment and provide greater transparency for all of the stakeholders involved, above all students and their prospective employers. Post-Covid-19, the lessons learnt about assessment and feedback under the extreme strains of universal lockdown must be used to enhance both greater quality and reliability in this field. This is a pressing and timely question, as it encompasses fundamental issues related to equality, diversity and inclusion. As future areas of research, it would also be worth considering: the need to implement shared rubrics for assessment which are also made available to the students in the interests of formative assessment; a unified system of accreditation of the english language level of the teaching staff who are going to deliver their classes and carry out their assessment through emI; a comprehensive emI training programme on an institutional level; a system of regular classroom observations, both by an external observer and also through peer reviews of recorded classes. Acknowledging that english is indeed a core skill for the university educated professionals of tomorrow, it is incumbent upon us to improve not only the overall standard of the emI that they receive at their institutions, but also very specifically the quality and reliability of the assessment that they receive through emI, an assessment that will inevitably condition their professional prospects. In this sense, this article has sought to shed some light on emI lecturers’ attitudes and practices in relation to this question and provide some broad recommendations about how this issue needs to be addressed in the ever more globalised and competitive context of the Internationalisation of Higher education. Article history: Received 31 May 2022 Received in revised form 02 September 2022 Accepted 22 September 2022 “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 365 References mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368366 Ball, P., & Lindsay, D. (2013). Language demands and support for English-Medium Instruction in tertiary education: Learning from a specific context. In A. Doiz, D. Lasagabaster & J. M. Sierra (Eds.), English-Medium Instruction at universities. Global challenges (pp. 44-61). Multilingual Matters. Breeze, R., & Sancho Guinda, C. (2021). Teaching English Medium Instruction courses in Higher Education: A guide for non-native speakers. Bloomsbury. Coleman, J. A. (2006). English-medium teaching in European Higher Education. Language Teaching, 39(1), 1-14. Costa, F., & Coleman, J. A. (2013). A survey of English-medium instruction in Italian higher education. International Journal of Bilingual Education and Bilingualism, 16(1), 3-19. Dafouz, E., Camacho, M., & Urquía, E. (2014). “Surely they can’t do as well”: A comparison of business students’ academic performance in English medium and Spanish-as-first-language medium programmes. Language and Education, 28(3), 223-236. Dafouz Milne, E., & Núñez Perucha, B. (2010). Metadiscursive devices in university lectures: A contrastive analysis of L1 and L2 teacher performance. In C. Dalton-Puffer, T. Nikula, & U. Smit (Eds.), Language use and language learning in cLIL classrooms (pp. 213-232). John Benjamins. https://doi.org/10.1075/aals.7.11daf Dimova, S., Hultgren, A. K., & Jensen, C. (Eds.) (2015). English Medium Instruction in European Higher Education. De Gruyter Mouton. Dimova, S., Hultgren, A. K., & Jensen, C. (2015). English-Medium Instruction in European Higher Education: Review and future research. In D. Dimova, A. K. Hultgren & C. Jensen (Eds.), English Medium Instruction in European Higher Education (pp. 317-324). De Gruyter Mouton. Doiz, A., Lasagabaster, D., & Sierra, J. M. (2013). English-Medium Instruction at universities. Global challenges. Multilingual Matters. Doiz, A., Lasagabaster, D., & Sierra, J. M. (2013). English as L3 at a bilingual university in The Basque Country, Spain. In A. Doiz, D. Lasagabaster, & J. M. Sierra (Eds.), English- Medium Instruction at universities. Global challenges (pp. 84-105). Multilingual Matters. Lasagabaster, D. (2015). Multilingual language policy: Is it becoming a misnomer at university level? In S. Dimova, A. K. Hultgren, & C. Jensen (Eds.), English Medium Instruction in European Higher Education (pp. 115-136). De Gruyter Mouton. Lo, Y. Y. (2022). Researching Assessment in EMI. In J. K. H. Pun & S. M. Curle (Eds.), Research methods in English Medium Instruction (pp. 61- 74). Routledge. Mancho-Barés, G., & Aguilar-Pérez, M. (2020). EMI lecturers’ practices in correcting English Resources for language teaching? Journal of Immersion and content-Based Language Education, 8(2), 257-284. Massler, U., Stotz, D., & Queisser, C. (2014). Assessment instruments for primary CLIL: The conceptualization and evaluation of set tasks. The Language Learning Journal, 42(2), 137-150. Otto, A., & Estrada, J. L. (2019). Towards an understanding of CLIL assessment practices in a European context: Main assessment tools and the role of language in content subjects. cLIL Journal of Innovation and Research in Plurilingual and Pluricultural Education, 2(1), 31-42. Phillipson, R. (2015). English as threat or opportunity in European Higher Education. In S. Dimova, A. K. Hultgren, & C. Jensen (Eds.), English Medium Instruction in European Higher Education (pp. 19-42). De Gruyter Mouton. Pulcini, V., & Campagna, S. (2015). Internationalisation and the EMI controversy in Italian Higher Education. In S. Dimova, A. K. Hultgren, & C. Jensen (Eds.), English Medium Instruction in European Higher Education (pp. 65- 88). De Gruyter Mouton. Ramburuth, P., & McCormick, J. (2001). Learning diversity in higher education: A comparative study of Asian international and Australian students. Higher Education, 42, 333-350. Roothooft, H. (2019). Spanish lecturers’ beliefs Dr. maría Dolores Rodríguez melchor is a lecturer at Comillas Pontifical University (madrid, Spain), where she is director of the Translation, Interpreting and multilingual Communication Department. She is an accredited conference interpreter for the european Institutions. Her research interests include ICTs in conference interpreter training, emI at university level and english as a lingua franca. She has recently co-edited a volume of the Peter Lang series on New trends in Translation Studies (The role of technology in conference interpreter training, 2020). Dr. Andrew Samuel Walsh is a lecturer at Comillas Pontifical University and holds a degree in Hispanic Studies from the University of Liverpool (UK) and a PhD in Spanish Philology from the University of granada. He previously taught at the University of granada in the Departments of Translation and Interpreting and english Philology. He is a member of the european Language Council working group on “Languages and Science”, focusing his research on multilingualism in science and emI in Higher education. He is also interested in literary translation and is the co-editor of Literary Retranslation in Context (Peter Lang, 2017). His latest book is Lorca in English. A History of Manipulation through Translation (routledge, 2020). noTeS 1 In this respect, it is worth highlighting the recent judgment of the State Council of Italy which prohibited the Polytechnic of milan from offering all its graduate courses in english, which was supported by the Accademia della Crusca, the body responsible for safeguarding the Italian language. http://www.universityworldnews.com/article.php?story=2018020304275939 (Accessed 30 November 2021) 2 In the first semester of the academic year 2021-2022, when international students were once again able to study at the university, the figure for incoming students was 550. 3https://universitatsirecerca.gencat.cat/web/.content/16_llengues_a_la_universitat/les_llengues_a_la_uni versitat/Documents/plans_de_llengues_de_les_universitats/UAB-Plan-of-Languages-2016-2020-eN.pdf (Accessed 30 November 2021) “I’m reALLy NOT ASSeSSINg THe LANgUAge” - emI LeCTUrerS’ PerCePTIONS OF THeIr ImPLICIT AND exPLICIT ASSeSSmeNT OF STUDeNTS’ LANgUAge PrODUCTION ibérica 44 (2022): 345-368 367 about English medium instruction: STEM versus Humanities. International Journal of Bilingual Education and Bilingualism, 25(2), 627-640. Shaw, S., & Imam, H. (2013). Assessment of international students through the medium of English: Ensuring validity and fairness in content- based examinations. Language Assessment Quarterly, 10(4), 452-475. Shohamy. E. (2012). A critical perspective on the use of English as a medium of instruction at universities. In A. Doiz, D. Lasagabaster & J. M. Sierra (Eds.), English-Medium Instruction at universities. Global challenges (pp. 196-212). Multilingual Matters. 4 For example, Assessment and Evaluation in Higher Education which is devoted specifically to this aspect of university education. mAríA DOLOreS rODrígUez meLCHOr & ANDrew SAmUeL wALSH ibérica 44 (2022): 345-368368