Promoting Linguistic Diversity and Inclusion: Incorporating Machine Translation Literacy into Information Literacy Instruction for Undergraduate Students The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, https://jps.library.utoronto.ca/index.php/ijidi DOI: 10.33137/ijidi.v5i3.36159 Promoting Linguistic Diversity and Inclusion: Incorporating Machine Translation Literacy into Information Literacy Instruction for Undergraduate Students Lynne Bowker, University of Ottawa, Canada Abstract Using a lingua franca for scholarly communication offers some advantages, but it also limits research diversity, and there is a growing movement to encourage publication in other languages. Both approaches require scholars to access material through other languages, and more people are turning to machine translation to help with this task. Machine translation has improved considerably in recent years with the introduction of artificial intelligence techniques such as machine learning; however, it is far from perfect and users who are not trained as professional translators need to improve their machine translation literacy to use this technology effectively. Machine translation literacy is less about acquiring techno-procedural skills and more about developing cognitive competencies. In this way, machine translation literacy aligns with the overall direction of the Association of College & Research Libraries’ (2015) Framework for Information Literacy for Higher Education, which encourages a conceptual, rather than a skills-based, approach. This case study presents a pilot project in which machine translation literacy instruction was incorporated into a broader program of information literacy and delivered to first-year students—both Anglophone and non- Anglophone—at a Canadian university. Students were surveyed and, overall, they found the machine translation literacy module to be valuable and recommended that similar instruction be made available to all students. Academic librarians are well-positioned to participate in the delivery of machine translation literacy instruction as part of a broader information literacy program, and in so doing, they can promote linguistic diversity and better enable students and researchers from all regions to participate in scholarly conversations. Keywords: academic librarians; inclusion; information literacy; linguistic diversity; machine translation; machine translation literacy Publication Type: case study Introduction lthough numerous specific definitions exist, the concept of information literacy is generally understood to refer to the ability to identify, find, evaluate, and use information effectively. As noted by Polkinghorne and Julien (2018), high-quality instruction in information literacy is crucial for helping students to succeed academically. Thus, information literacy instruction has become a staple offering for undergraduates at A https://jps.library.utoronto.ca/index.php/ijidi Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 universities, where students of all disciplines need to learn how to research, weigh, and apply information in their field of study. In the context of undergraduate education, common aspects of information literacy instruction include helping students to develop effective research strategies, to navigate both relatively constrained academic databases and the much broader internet, to distinguish between high- quality sources and more dubious materials, and to appropriately cite and reference their selected sources (Julien et al., 2018). The 2015 Framework for Information Literacy for Higher Education by the Association for College & Research Libraries (ACRL) encourages academic librarians to look beyond a skills-based approach to information literacy instruction and to adopt a more conceptual approach within the following six concepts or “frames”: 1. Authority Is Constructed and Contextual 2. Information Creation as a Process 3. Information Has Value 4. Research as Inquiry 5. Scholarship as Conversation 6. Searching as Strategic Exploration (ACRL, 2015, p.8) Since the release of this framework, researchers in various countries have surveyed academic librarians with a view to better understanding their information literacy instruction practices within a contemporary, digital information world. As part of a 20-year longitudinal study in Canada, Polkinghorne and Julien (2018) enquire whether information literacy instruction is changing in response to the contemporary information retrieval environment. According to the results of Polkinghorne and Julien’s (2018) fifth and most recent survey conducted in 2017, the situation in Canada appears to have remained relatively stable, although “numerous respondents [academic librarians] are teaching in timely conceptual areas such as scholarly communication” (p. 84), which includes open access publishing and open education resources within the context of the survey. Julien et al. (2018) adapted the instrument used in the Canadian study to survey academic librarians in the U.S., and they observed that U.S. information literacy instruction focuses on using databases, search strategies, general library use, and use of the online catalogue. Julien et al. (2018) concluded that while these topics reflect needed skills, they also represent the mainstay of traditional instructional pedagogies. However, the researchers acknowledge that the instruction does appear to be integrating information technology to a greater extent, and they note that some respondents reported addressing topics such as social media, open access publishing, images and fair use, and citation metrics in their pedagogy. Meanwhile, Aharony et al. (2020) also used the same instrument to survey academic librarians in Israel and found that in this country too, the focus of information literacy instruction tends to be on database use, search strategies, library use in general, and use of the online catalogue. Once again, the majority of respondents suggested that they were integrating more information technology into their instruction. Overall, the surveys carried out in Canada, the U.S., and Israel reveal that information literacy instruction in these countries still place considerable emphasis on skills development, although there are some encouraging signs to suggest a movement towards 128 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 greater incorporation of critical thinking and information evaluation as put forward by ACRL’s (2015) framework. While each survey reveals several challenges to be overcome or areas of improvement that are needed, none specifically mentioned the issue of language, which can be a potentially complicating factor that is not always explicitly addressed as part of information literacy training. Similarly, while all of the survey results suggest greater incorporation of technology in information literacy instruction, none mentioned machine translation technology specifically. In response to this concern, the overarching research question to be explored in this article is: Can information literacy instruction be usefully broadened to include machine language translation in order to help university students become more critical and informed users of this technology within an academic context? Language does come up indirectly in the ACRL’s (2015, p. 20) framework, which presents the concept of “scholarship as conversation”. Within this frame, it is noted that learners who are not fluent in “the language and process of a discipline” may be hampered in their ability to engage and participate in the conversation (ACRL, 2015, p. 21). In this context, I interpret “language” to mean the specialized discourse of a given field rather than meaning a language such as English, French, or Spanish. However, I believe that both readings of the term language are valid here since a learner who does not know the language (e.g., English) in which the sought-after information is expressed will not be able to participate in the scholarly conversation. Unfortunately, this situation is all too common. In recent decades, English has become increasingly entrenched as the language of science, research, scholarly communication, and higher education, even though less than 5% of the world’s population speaks English as a native language (Montgomery, 2013). Because of the importance of English as a lingua franca, Wächter and Maiworm (2014) posit that many universities around the world offer programs taught in English, even when English is not an official language of their country. In their research, Wächter and Maiworm (2014) report that there were 725 English-taught programs offered in continental Europe in 2002, and by 2014 this number had jumped to 8,089, representing an increase of more than 1,000 percent. In addition, in countries where English is an official language, universities are welcoming growing numbers of non-Anglophone international students. For example, in the U.S., the number of international students rose by 63% between 2008 and 2018 (IEE, 2019); in Australia, it jumped by 97% during this same period (Ferguson & Spinks, 2019); while in Canada, this number shot up by 175% between 2007 and 2017 (IRCC, 2017). On the other side of the coin, we also see an increasing recognition that the dominance of any one language in scientific research hinders diversity (Pérez Ortega, 2020). For instance, any research published in a language other than English risks being overlooked, which contributes to biases in our understanding (Konno et al., 2020). In this way, the English language has become a sort of gatekeeper of science, excluding other opinions and perspectives, which makes this a critical issue not only for non-Anglophones but for English speakers too. In response to this phenomenon, there is a growing movement to diversify science and make it fairer and more equitable by encouraging researchers to report their findings not only in English but also in other languages. 129 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 Indeed, UNESCO has been consulting with member nations for input in the process of preparing a document entitled Recommendation on Open Science (UNESCO, 2021). In the preliminary report on the first draft of the Recommendation on Open Science, UNESCO (2020) notes that some nations have called for the recommendation to promote multilingualism, stating: Promoting multilingualism, to embrace worldwide inclusiveness, information-sharing, collaborative knowledge construction, and equity, by enabling global interaction with multinational and multidisciplinary researchers, and other OpenScience actors. (UNESCO, 2020, p. 11) In either case, whether it is non-Anglophones needing to use English as a lingua franca, or more researchers choosing to disseminate their work in other languages, there is a need for some type of interlingual transfer on the part of the information creator or seeker. Professional translation is costly and time-consuming, and it is simply not feasible to expect that all the information that is created in the world can be professionally translated into every one of the world’s languages in an affordable and timely fashion. Much of the time, people need to find their own solutions for processing information in other languages. Increasingly, this challenge is being tackled with the help of machine translation tools, such as Google Translate, DeepL Translator, Baidu Translate, and similar software. Indeed, as part of their trend report and insights document, the International Federation of Library Associations and Institutions (IFLA) identified machine translation technology as one of the five, key high-level trends to watch in the global information environment, noting that it has the potential to give researchers and others unprecedented access to information from other countries (IFLA, 2013). Given that there is a growing need for everyone, everywhere to be able to access and use information in multiple languages, we believe it would be useful to extend the information literacy instruction offered to university students to include training in how to use machine translation tools critically. Indeed, when Johnston et al. (2018) investigated information literacy instruction for international students, they found that both reading and writing in the local language posed significant barriers for many students. However, a systematic review of library and information science (LIS) literature on information literacy and international students conducted by Houlihan et al. (2017) uncovered just 23 articles published on this topic between 1990 and 2014. Some of those articles included language-based recommendations such as encouraging library instructors to speak clearly and to use plain language or embedding information literacy instruction within English-as-a-Second-Language (ESL) composition courses. However, none of these studies investigated the use of machine translation in the context of information literacy instruction or the need for machine translation literacy. If, as noted above, we essentially understand key elements of information literacy to be an ability to think critically and make balanced judgements about any information we find and use, then being able to make informed decisions about the information that has been accessed via machine translation technology will be a useful and increasingly necessary skill for speakers of all languages. The goal of this article is to describe and evaluate a pilot project where Anglophone and non- Anglophone students in the Faculty of Arts at the University of Ottawa were taught to use and reflect on machine translation tools as part of a machine translation literacy module that was integrated into a first-year undergraduate course on “New Literacies for the Digital Age.” Although the data was gathered from a single institution, the situation is not unique to this institution. As emphasized above, students all around the world will benefit from being able to 130 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 engage with information in or through another language. Therefore, this project serves as a useful case study whose findings may inspire similar machine translation literacy activities at other institutions. Following this introduction, the article is divided into four main sections. We begin with a brief presentation of machine translation, and the concept of machine translation literacy, establishing why it is needed. Next, we describe the pilot study, including a description of the broader course in which the machine translation literacy module was embedded, the demographic profile of the students in the course, the general content of the machine translation literacy module, and some of the pedagogical practices used to deliver and evaluate it. Then, we present student feedback on the module that was collected via an anonymous online survey. Finally, we offer some concluding remarks and suggestions for the next steps. Machine Translation and Machine Translation Literacy Efforts to develop machine translation systems date back to the period following World War II; the earliest approaches involved programming computers to apply grammar rules and look up words in large bilingual dictionaries (Hutchins & Somers, 1992). This so-called rule-based approach to machine translation had limited success, and the quality of the translations produced by these systems was usually quite poor, meaning that few people used them. Half a century later, around the year 2000, researchers began to experiment with a data-driven approach to machine translation known as statistical machine translation (Koehn, 2010). This approach played more to the strengths of computers, which include number crunching and pattern matching, which led to a considerable improvement in the quality of machine- translated texts overall, although this quality remained far from perfect. Even more recently, since around late 2016, there has been another paradigm shift in the underlying approach to machine translation. Now, researchers have introduced artificial intelligence techniques, such as machine learning, in an approach that is referred to as neural machine translation (Forcada, 2017). This approach, which also relies on providing the computer with enormous collections of previously translated texts, incorporates an artificial neural network that can be trained to “learn” from these examples. Whereas both the original rule-based approach and the later statistical approach produced translations that often contained semantic errors and sounded awkward, the translations produced by neural machine translation systems are much more accurate and natural sounding. Though still not perfect, the output of neural machine translation systems is more viable as a starting point than was the output of the older systems. Since the results may be usable for some purposes (e.g., basic knowledge acquisition), or may at least provide a viable first draft that can then be improved (e.g., for academic writing), more and more people are beginning to use machine translation, which is now freely available online and thus easily accessible to anyone with an internet connection. Adding to their appeal is the fact that, from a user perspective, machine translation tools are very straightforward to use because they require users to do little more than copy/paste a text and select the language for translation. Among the group of active machine translation users, we can find students who, as noted above, may need to engage with information in or through multiple languages as part of their studies. Indeed, numerous researchers, including Mundt and Groves (2016), Alhaisoni and Alhaysony (2017), Nurminen and Papula (2018), and Lee (2020), have identified university students as avid 131 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 users of machine translation tools. However, just because machine translation technology is relatively easy to access, very easy to use, and capable of producing a reasonable quality first draft, this does not mean that students inherently adopt a critical mindset when using it. In order to become informed and critical users of machine translation tools, and of their translated output, students need to develop machine translation literacy. As noted above, the how-to skills of using machine translation tools are very easy to acquire since they consist of little more than copying and pasting a text, choosing a language pair, and clicking the “Translate” button. Using machine translation is easy, but using it critically requires some thought. Therefore, as will be explained in more detail in an upcoming section, machine translation literacy is mainly about developing cognitive competencies, rather than techno- procedural skills. In this way, machine translation literacy is in line with the more conceptual approach to information literacy instruction that is encouraged by ACRL’s (2015) framework. What is Driving the Need for Machine Translation Literacy? Professional translators are trained both in translation and in how, when, and whether to use translation technology, as well as how to recognize and compensate for its limitations. However, people outside the translation professions do not typically receive such training. For non-language professionals, the popular media play a key role in reflecting and shaping people’s perceptions of machine translation. Several researchers have investigated the way that the media from various countries present this technology. Bowker (2020) considers the press in Canada between 1990 and 2015, noting that in this period there were 44 articles about translation technology in the English-language press and 70 in the French-language press. According to Bowker (2020), these news reports often invoke science fiction references (e.g., Star Trek and Hitchhiker’s Guide to the Galaxy), and they frequently take a humorous tone and regularly poke fun at machine translation’s inability to cope with translations of poetry or songs—tasks that are particularly challenging even for professional human translators. Meanwhile, Wang and Ping (2020) examine 50 articles collected from Chinese media outlets between 2017 and 2019 and note that a majority (68%) of the articles focus on the excitement about progress made in machine translation in recent years following the introduction of the neural approach. However, Wang and Ping (2020) also comment that half of the articles contain what they describe as “hype,” such as when the Chinese media generally positions machine translation as doing away with the need for human translators (p. 11). Finally, Nuñes Vieira (2020) conducted an international study by searching the Newspapers section of the Westlaw database. This search identified 284 English-language articles, from 1986 to 2019, about machine translation coming from the U.K., the U.S., India, Japan, China, and South Korea. Nuñes Vieira (2020) established that the coverage of machine translation in the written press tends to emphasize the positive aspects of the technology; however, at times these reports “inflated the capabilities of the technology by comparing it to human translators or by implying that MT had the power to make users speak and understand any language without the prospect of encountering any issues” (p. 13). Meanwhile, those articles which took a negative view were more narrowly focused on the poor quality of the output and used a sensationalist tone to draw particular attention to errors that were either humorous or shocking. Taken as a whole, these three investigations into news coverage about machine translation from various regions suggest that the reporting lacks nuance and tends either to overstate the capabilities of this technology or to position it as being good mainly for a laugh. The truth of the matter lies somewhere between these two extremes, and people who are outside the language professions need help 132 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 to increase their awareness of how machine translation works, when and where it can be usefully employed, and how to work with it more effectively to optimize the output. Nuñes Vieira (2020) notes, MT [machine translation] has great potential to facilitate and promote multilingualism, but its speed and usefulness may also prompt end-users to underestimate the complexities of translation while overestimating the capabilities of the technology, which in turn may lead to its misuse. (pp. 98-99) In addition to pointing out how the popular media may distort the view of non-translators concerning machine translation and its capabilities, the literature also provides examples of machine translation being used by non-translators with varying degrees of success. For instance, Anazawa et al. (2013) describe how practicing nurses in Japan use machine translation to stay on top of the latest developments in international nursing literature. Bowker and Buitrago Ciro (2019) investigate the use of this tool by researchers who want to publish in other languages, Nurminen (2020) explains that patent professionals use this technology to search for international patents, and O’Brien and Ehrensberger-Dow (2020) observe that machine translation may be used to facilitate communication in a crisis (e.g., following an earthquake). In each case, the researchers emphasize that some form of machine translation literacy instruction is needed to allow users to make better decisions about employing the technology and to optimize its use. In line with these researchers, I believe that expanding information literacy instruction to include training in machine translation literacy—whether for journalists, health practitioners, educators and information professionals, patent professionals, crisis communicators, or for the public at large—can represent a meaningful step in this direction. In the next section, I describe a pilot project where I integrated a machine translation literacy module into a broader information literacy course aimed at first-year undergraduate university students. Pilot Project: Teaching Machine Translation Literacy to First-Year Undergraduate Arts Students To explore the viability and value of training university students in machine translation literacy, I built and incorporated a module on the topic into a course called Introduction to Interdisciplinary Study in the Arts at the University of Ottawa. At the University of Ottawa, all first-year students in the Faculty of Arts are required to take a minimum of four full-semester (12-week) courses from a pool that includes several philosophy courses, English courses, and the aforementioned interdisciplinary course. The high-level goal of all the courses in this pool is to help students develop skills in critical reading and academic writing. The interdisciplinary course must be team-taught by two or three faculty members who come from different departments, and it must respond to the following official course description, which appears in the course catalogue: AHL 1100 Introduction to Interdisciplinary Study in the Arts (3 units) Exploration of at least two disciplines in the Faculty of Arts whose conjunction illuminates contemporary situations and debates. Development of critical reading and academic writing. This course has variable themes. Students may take this course twice with different topics. (University of Ottawa, 2021) 133 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 The theme or topic of the course changes from one semester to the next, depending on who will be teaching. Each year, professors are invited to pitch an interdisciplinary teaching team and a topic for consideration. For the Winter 2021 semester, I successfully pitched the topic “New Literacies for the Digital Age” to be co-taught by a professor from the School of Information Studies and a professor from the School of Translation and Interpretation. Along with modules on more traditional aspects of information literacy, the course also contained instruction on media literacy (e.g., fake news) and scholarly communication literacy (e.g., predatory publishing), as well as a specially designed module on machine translation literacy. A total of 80 students registered for the course, coming from 13 different programs within the Faculty of Arts as summarized in Table 1. Table 1. Disciplinary backgrounds of students in the “New Literacies for the Digital Age” course Program Number of Students Communication 35 Visual Arts 12 Linguistics 8 Environmental Studies 6 Second-language Teaching 4 History 4 Art History 2 English 2 General Arts 2 Philosophy 2 Creative Writing 1 Music 1 Interdisciplinary Studies 1 Total 80 As the table shows, 35/80 (44%) students were from the Communication program, which is the largest program in the Faculty of Arts. Many students in the Communication program go on to work as journalists or as other media professionals, meaning that this machine translation literacy module had the potential to reach a population that may later be able to portray machine translation in the media in a way that is more nuanced than that found in the news reports studied by Bowker (2020), Wang and Ping (2020), and Nuñes Vieira (2020). In addition, 134 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 six students are from the Environmental Studies program, which may not seem like a typical program from the Arts; however, at the University of Ottawa, this interdisciplinary program is coordinated through the Department of Geography, which is housed in the Faculty of Arts. Overall, the student population in the course is relatively diverse with regard to their program of study and this group represents a cross-section of students from programs in the Faculty of Arts. Students in the class were asked to voluntarily share their native language and their responses are summarized in Table 2. Table 2. Native language of students in the “New Literacies for the Digital Age” course Language Number of Students English 44 Chinese 14 French 8 Arabic 5 Hindi 3 Spanish 3 Persian 1 Ukrainian 1 Vietnamese 1 Total 80 Out of 80 total students in the course, 55% (44/80) identified as native English speakers, while the remaining 45% (36/80) indicated that their native language was a language other than English. While this may appear to be a high percentage of non-Anglophones, there are several factors to consider. Firstly, Canada is an officially bilingual (English-French) country and the University of Ottawa, which is located in the nation’s capital on the border of Ontario (an English-speaking province) and Quebec (a French-speaking province), is an officially bilingual university. Approximately 30% of the overall student population of the university is Francophone, and many of these students are bilingual and take courses taught in English (University of Ottawa, 2020). In addition, Canada is a country with a strong history of immigration, meaning that many people living in Canada have come from elsewhere and may speak a heritage language in addition to English or French. According to the most recent census, 21.8% of people living in Canada report using a language other than English or French at home (Statistics Canada, 2020). Finally, owing to the COVID-19 pandemic, the vast majority of programs offered by the University of Ottawa in the 2020/2021 academic year were delivered online, meaning that students did not need to be present on campus to take courses and could 135 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 be located anywhere in the world. In this particular academic year, international students made up 17% of the university’s total undergraduate population (University of Ottawa, 2020). Machine Translation Literacy Module The module on machine translation literacy was integrated into the “New Literacies for the Digital Age” course as a module for one week of the 12-week course. This module comprised three in-class contact hours; students were also expected to do some before-class preparation (e.g., consulting introductory texts and videos on how machine translation systems work), as well as some after-class homework (e.g., exercises on revising machine translation input and output, quiz). As previously noted, machine translation literacy has a strong cognitive or conceptual element. As such, the content of the module is less about “how to” use machine translation (i.e., which buttons to push) and more about whether, when, and why to use the tool. The learning module focuses more on critical thinking tasks, such as evaluating the suitability of a text for translation by machine or weighing the benefits and risks of using machine translation against other translation solutions. The overall aim of the machine translation literacy module is to teach students to approach machine translation use in a critical way where students will be able to: • Explain the general concept of machine learning and the overall neural approach to machine translation • Appreciate the type, quantity, and quality of data required for data-driven neural machine translation, identify how machine translation systems can be sensitive to data, and articulate the potential consequences of data insufficiency • Describe the need for transparency around machine translation use • Conduct basic risk assessment regarding machine translation use • Compare and evaluate the results produced by a selection of free online machine translation systems • Modify input texts to reduce ambiguity and improve the quality of the machine translation output • Apply basic post-editing techniques to improve machine translation output according to fit-for-purpose principles To enable students to achieve these learning outcomes, the machine translation literacy module was divided into four main parts: Data-driven Approaches to Machine Translation With the help of some introductory videos and readings (viewed/read before class), as well as a short lecture from the professor, students learned how data-driven machine translation systems (neural machine translation systems) are built and operate. This knowledge is 136 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 important for helping students to understand the strengths and limitations of these tools. For example, understanding the concept of sensitivity to training data can help users to realize why machine translation systems could be more or less useful for different language combinations, different domains, or different text types. A machine translation system operating between widely used languages (e.g., English and French) may produce better results than one operating between less widely used languages (e.g., Finnish and Polish) because a greater volume of training data is available for the former group. Likewise, a machine translation system that has been trained using texts from one domain, such as medicine, may not do a good job translating documents from another domain—for example, the law. An understanding of how data-driven machine translation systems are trained also alerts students to the fact that each machine translation system is likely to produce different results. For instance, Google Translate is a well-known free online machine translation tool, but it is not the only one. Since each tool is trained using a different corpus, the various tools are unlikely to produce identical results. In particular, different tools may perform better or worse depending on the language in question. For instance, Baidu Translate is a tool that is strong in Chinese, while Yandex.Translate works well with Russian, and DeepL Translator might be a better choice for the more commonly used European languages (e.g., French, German, Spanish). In addition, neural machine translation systems are always ‘learning,’ so their results may improve from one day to the next. Students learn that it is a good idea to try different systems and to refrain from writing off a tool altogether because it performs poorly on one occasion. Finally, knowledge about sensitivity to training data also alerts students to the potential for different types of algorithmic bias, such as gender or racial bias, if the training data is not well selected. For instance, when translating from a language that has gender-neutral, third-person singular pronouns (e.g., Finnish, Hungarian) into a language where third-person, singular pronouns are marked for gender (e.g., English, French), there are reports that Google Translate generally skews toward choosing male pronouns (e.g., he) for words like “strong” or “doctor” and female pronouns (e.g., she) for “beautiful” and “nurse” (Monti, 2020). Transparency To help students learn the importance of transparency regarding machine translation use, they are presented with a number of mini-case studies to analyze. For example, in an academic context, one important ethical consideration is academic integrity. Using machine translation for coursework may not be accepted if it is contrary to the learning objectives of the course. In contrast, it may be perfectly fine to use machine translation as an academic writing aid to help produce an assignment for a geography or history course, where the objective is to demonstrate an understanding of the concepts in question, rather than to demonstrate a mastery of linguistics. However, when using machine translation to translate academic works, it is still essential to cite and reference the original ideas that are borrowed, even if these are being presented in a different language. Beyond academic contexts, it is also worth considering the issue of fair use of machine translation and sustainability, which includes encouraging users of machine translation not to use this technology in a way that might harm the language industries. For example, users should be transparent about their use of machine translation (e.g., clearly label machine- 137 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 translated texts as such) as this will allow readers to take this information into account when deciding how much they should trust the content. Risk Assessment Mini-case studies are also used to help students develop their judgement about assessing the benefits and risks of using machine translation for a particular task. One key piece of information to share with machine translation users is the fact that information pasted into a free online machine translation tool does not disappear when the window is closed. Rather, companies that own the tools can keep the data and use it for other purposes. Therefore, users should not enter sensitive information into free online machine translation tools. However, a risk assessment needs to take place on another level also, such as determining whether the translation task is a high-stakes or a low-stakes task. For example, using machine translation to help read a manga comic book or to send a casual email to a friend is a low- stakes scenario because a poor translation is unlikely to have serious consequences. In contrast, using machine translation to translate texts in healthcare or in a legal setting is a high-stakes scenario because translation errors could lead to serious consequences (Nuñes Vieira et al., 2020). Translation can be undertaken for different purposes. Using raw machine translation to help understand texts in a subject area that is already familiar to the reader may be a good use of this tool, whereas if the purpose is to publish a document, then a machine-translated text may first need to be revised before being distributed more widely. Using machine translation for short texts, such as translating keywords to search in a database (Bowker, 2018) may produce different results than translating longer texts, such as an abstract or even a complete article. On the one hand, translating isolated words presented out of context may be problematic (e.g., without some context, it is impossible to know whether the word “bank” is referring to a financial institution or the side of a river). On the other hand, a long and complex sentence offers more opportunities for the machine translation system to stumble. Likewise, as noted above, a machine translation system that has been trained using texts from one domain (e.g., medicine) may not do a good job translating documents from another domain (e.g., law), and similarly, a system trained to handle a particular text type, such as contracts, may not perform well with another text type, such as user manuals. The conditions of the job may also come into play. The well-known “triple constraint”(Wright & Lawlor-Wright, 2018, p. 1), where competing parameters such as time, cost, and quality come into play, could help to determine whether it makes sense to use machine translation or whether the services of a professional translator are needed. Machine translation is almost always faster and cheaper than professional translation, but language professionals are usually capable of producing higher-quality text if this is what the job requires. Determining the level of quality needed and figuring out how to best achieve this while respecting the specified budget and deadline is a skill that students work on developing as part of their machine translation literacy training. In summary, it is important to consider the stakes, purpose, content, form, and audience of the translation and to conduct a sort of risk assessment (e.g., Canfora & Ottmann, 2020) before deciding whether machine translation is the right option. 138 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 Interacting with Machine Translation Finally, students engage in several text editing exercises to learn how they can interact with machine translation systems to improve their output. The biggest challenge for machine translation is the inherent ambiguity of natural language. One key way to improve machine translation output is to ensure that the input text is written in plain language with little to no ambiguity (e.g., using the active voice, short sentences, consistent terminology). A text that is easier to read is also easier to translate, and when users are translating from their dominant language to a language that they know less well, it may be easier for them to improve the input text rather than to fix up the translated text. Another way for users to interact with machine translation is by revising the output. In many cases, the raw output produced by a machine translation system must be fixed up or improved (such as before submitting a text to a professor or a journal). Although the types of errors made by a machine translation system will likely differ from one system to the next, from one text type or domain to the next, and from one language to the next, it is worth learning some basic techniques to spot and fix errors. In particular, neural machine translation systems are known to produce texts that sound very plausible, even though they may not be correct (Way, 2020), and such errors may be easily overlooked if users are not vigilant or do not know where to look. By practicing both pre- and post-editing and comparing the results of different machine translation systems, students become more comfortable with manipulating texts to achieve a desired level of quality. Pedagogical Approaches and Evaluation of Learning Outcomes As noted above, a variety of pedagogical approaches were incorporated into the machine translation literacy module. For example, students were asked to watch some short introductory videos on neural networks and machine learning and to read to an introductory text on data-driven machine translation before the class. This material was supplemented with a short lecture that focused on explaining the type of data needed for successful data-driven approaches, an analysis of the ways in which machine translation algorithms are sensitive to data, and the potential problems that can arise if there is not enough data or the data is not of high quality. The students’ knowledge of these issues was evaluated with a 20-question quiz containing a variety of multiple-choice, true/false, fill-in-the-blank, matching, and ordering questions. The quiz was developed using the university’s learning management system and could therefore be auto-corrected. The class average for the quiz was 85% and the median grade was 84%, indicating that the majority of students competently absorbed the material. Meanwhile, for the parts of the machine translation literacy module that focused on issues of transparency and risk assessment, the main teaching method used was a short lecture combined with a series of mini-case studies. Following the lecture, students were divided into smaller groups (10 students per group) and presented with a series of short scenarios. Each group was tasked with analyzing one scenario and presenting the results of their analysis to the rest of the class for a brief discussion. The rapporteur for each group also posted the main points of their analysis to the online discussion forum and students from other groups were encouraged to comment on the analyses for homework. Because the discussion forum was an optional, rather than a required activity, there was less discussion than I would have liked to see; however, the 17 comments that were posted showed a meaningful engagement with the material, including the presentation of counterarguments and the development of some mini- 139 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 debates about whether or not machine translation was the best choice for a given scenario. After seeing the results of the online discussion and observing its value for promoting critical reflection among the students, I am considering either making the discussion item compulsory in a future iteration of the course or modifying the presentation-style format. Future iterations could include converting at least some of the mini-case studies into more of a debate format where different groups take “for” and “against” positions and argue their merits. For the parts of the machine translation literacy module that focused on interacting with the input and output, students worked on exercises that involved both pre-editing and post-editing. For the pre-editing exercise, students were invited to select a short extract of approximately 150-200 words from a general news text in their strongest language and to run it through a machine translation system to obtain output in another language that they know reasonably well. Next, they were asked to study the machine translation output and to identify those areas where there were problems. Then, the students were told to return to the original source text and try to rephrase or clarify the elements that had not been translated well. For example, this could mean shortening a long sentence, replacing an ambiguous term with a more precise term, or clarifying the relationship between two elements (e.g., inserting an optional relative pronoun such as “that”). Finally, students were instructed to run their modified text through the machine translation system again to see whether the modifications had the desired effect of improving the output. This process could be repeated in multiple iterations. Owing to the limited amount of class time available for the module, students did not work on post-editing during class time but instead did this activity as a graded assignment. For the post- editing assignment, students were asked to once again find a news article extract of approximately 150-200 words, but this time, the text should be in their second or less dominant language. Students were invited to run this text through at least two free online machine translation systems into their strongest language. Students were asked to compare the results produced by the different tools. Next, they were asked to take these two different output texts, and use a “track changes” tool to edit each text differently according to the guidelines provided by the Translation Automation User Society (TAUS, 2011). One text was to be edited to a “good enough” quality, meaning it was essential to correct any errors of meaning, but not necessary to improve the style. Meanwhile, the second text was to be edited to a “publishable quality,” wherein both errors of meaning and stylistic problems had to be corrected. Finally, students were asked to explain the nature of the changes that they implemented and their motivation for doing so. This work was scored and incorporated into their overall grade for the “New Literacies for the Digital Age” course. The class average for the assignment was 81%, and the median was 83%, indicating that the majority of students performed competently with regard to post-editing—although it must be acknowledged that this was a very small-scale assignment. Feedback on the Machine Translation Literacy Module To obtain some feedback on the machine translation literacy module, we invited the students to participate in a voluntary and anonymous 10-question online survey distributed one week after the module using the online tool Survey Monkey. The survey was designed to gather information about the students’ general use of machine translation as well as to obtain their feedback on the efficacy of the machine translation literacy module (see Appendix A). In total, 140 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 67 of the 80 students completed the survey, representing a completion rate of approximately 84%. When asked how often they use machine translation, 85% of students claim to use this technology more than once a month, 15% of this group state that they use it every day, 21% indicate that they use it three to four times per week and another 16% reply that they use it at least once per week. The students indicate that they use machine translation in multiple areas of their lives, with 72% of respondents using machine translation for their studies, 64% using it for leisure activities, and 15% using it for their job. In answer to the question about whether they use machine translation to help them with reading/understanding a text or with writing/producing a text, 66% of students claim to use it for both tasks, while 13% indicate that they use it only when they need to write in another language, and 21% only for reading in another language. When asked to comment on what they would do if they did not have access to free online machine translation, 85% of the students said they would ask a friend or colleague to help them translate the text, while 15% said they would not bother to translate the text at all. No student indicated a willingness to hire a language professional to translate or edit the text. With regard to the language for which they use machine translation, 9% of respondents indicate that they tend to use this technology when working with a language that they already know quite well, while 27% said they use it for a language they know a bit, and 31% use it for a language that they don’t know at all. The remaining 33% specified that they use machine translation with a language that they are actively learning. Finally, in answer to the question about how satisfied they are with the results of machine translation overall, 4% of the students stated that they were completely satisfied, 49% indicated that they were very satisfied, 42% noted that they were moderately satisfied, and 5% claimed that they were not very satisfied. No student claimed to be not at all satisfied. Of the key items covered in the machine translation literacy module, students were asked to indicate which item they found to be the most surprising or novel thing that they did not previously know. Forty-two percent of the students selected the issue of privacy/confidentiality, 24% were surprised to learn about potential algorithmic bias, 15% were surprised to learn that different tools could produce different results, 9% had never previously considered that changing the input could improve the output, 6% found the information regarding academic integrity to be new to them, while just 4% had not previously thought about how the nature of a translation task might influence its suitability for machine translation. No students reported being surprised to learn about the likelihood of needing to revise the output of a machine translation system. Likewise, no students indicated that they had not learned anything new. Respondents were asked to weigh in on whether they thought that machine translation literacy instruction was valuable for undergraduate students. In response, 21% identified it as being essential, 61% said it was very important, and 18% suggested it was moderately important. No students selected the categories not very important or not at all important. The members of the class were asked whether, in their opinion, the University of Ottawa should make machine translation literacy instruction available to all students. In response, 43% of the students replied definitely, 40% answered probably, and 15% said maybe. Only 1% of the respondents replied probably not, while no students selected the category definitely not. Finally, students had the option of providing any other comments that they wished to share about the machine 141 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 translation literacy module, and 27 students availed themselves of this opportunity. Their comments fall into three main categories: 1) observations about former naïve use of machine translation, 2) indications of intent to change behaviour following the completion of the module, and 3) expressions of gratitude for the module. These are summarized in Table 3. Table 1. Comments shared by students about the machine translation literacy module Theme Number of comments Sample comments Former naïve use of machine translation 3 • I never realized how much I was on auto-pilot when I used MT before. This was a wake-up call! • I see I was too trusting of results. I need to pay more attention. • I did not think to make a better start text to get a better translation text. This is a great idea! Intent to change behaviour following the machine translation literacy module 4 • Machine translation is only as good as its users. I feel like a smarter user now and I hope this tool will be even more useful to me now too. • I use Google Translate more often than I’d like to admit… It’s just so convenient so I always figured there was nothing to lose. But sometimes the results are frustrating so now I’ll try and be more selective about what I use it for because I see that some tasks don’t make sense for machines. • I feel like I should not have been surprised to learn that Google is keeping my data, but I was. I’m going to be more careful in future. • I want to know more how I can make my texts better after machine translation. Can you give us even more information after this class? Expressions of gratitude 20 • Thank you for teaching us about machine translation literacy, professor! • I learned a lot of new things – thanks! • This was a very interesting lesson • I do believe each person should know the key essentials of machin [sic] translation literacy. I am very glad to know now. 142 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 Discussion The survey confirms that undergraduate students—both Anglophones and non-native speakers of English—are frequent users of machine translation and that they use this technology in various areas of their lives to support comprehension as well as text production tasks. Moreover, it is not too surprising to learn that students who need translation services are not willing/able to pay for them, meaning that free online machine translation is a key tool for this group and so it is in their best interests to learn how to incorporate it more effectively into their activities. It also means that this type of machine translation use is not taking business away from professional translators, so it falls into the category of fair and sustainable use of this technology. Interestingly, nearly 70% of students report using machine translation for languages that they know to some degree or are actively learning, and comparatively few use it for languages that they don’t know at all. This suggests that even students who appear to be reasonably competent in a given language—which is likely the case for most international students, who are usually required demonstrate a minimum level of competence in the language of instruction before being admitted to a program—turn to machine translation for some of their needs. It is well known that operating in a non-native language has a higher cognitive load than working in a native language, and some people may turn to machine translation because it could help them to either work more quickly or more confidently, allow them to verify their understanding, or reduce their cognitive load in some other way (O’Brien & Ehrensberger-Dow, 2020). In other words, even if students demonstrate a certain level of competence in a language, they may still turn to machine translation for support. Failing to recognize that even reasonably competent speakers use this technology represents a missed opportunity for providing better support to this group since improved machine translation literacy could help to lighten a user’s cognitive load. In terms of satisfaction, students report that machine translation does a reasonable job of meeting their needs overall, but there is still room for improvement since only 4% of respondents were completely satisfied, and nearly half the respondents were only moderately or not very satisfied. What’s more, since no students reported being completely dissatisfied, this suggests that students will continue to use machine translation, imperfect as it is. Although the technology continues to improve, there are steps users can take to use it more critically and get better results, and so machine translation literacy instruction could potentially help students make better decisions about its use and to get more out of this technology. The responses to the question about which element of the machine translation literacy instruction was most novel or surprising were diverse and covered six elements discussed in the module, which suggests that people’s knowledge on this topic is varied, partial, or fragmented and that there is still lots to be learned by students about effective use of machine translation. Although some of these things may be obvious to language professionals, they are not obvious to non-translators and need to be taught. These observations seem to be confirmed by the fact that all of the respondents felt that machine translation literacy instruction was at least moderately important for university students, and more than 80% emphasized that it was at least very important. Similarly, a combined 83% of the respondents felt that the University of Ottawa should probably or definitely make machine translation literacy instruction available to all students. Finally, students appear to have engaged with the material and their comments reveal a general appreciation for the module. Some students explicitly recognized that their 143 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 prior approach to using machine translation may have been naïve, while others signaled an intention to do better by approaching this technology from a more critical and informed perspective moving forward. Many students acknowledged their appreciation for the machine translation literacy module overall. Although the survey captured some simple factual data (how often machine translation is used and what types of tasks it is used for) and perceptual responses (satisfaction with machine translation and whether the module should be available to all students), this format did not allow for the collection of more substantive data on whether or how the students applied some of the content taught through specific assignments or tasks (although some evidence has been provided in the section on pedagogical approaches and feedback above). In the pilot study described above, the machine translation literacy module came towards the end, taking place in week 10 of the 12-week course, so there was no time to implement a more longitudinal assessment, although it would be interesting to integrate this into a future iteration of the course. For instance, if the machine translation literacy module came earlier in the course, it could be interesting to survey students later in the course to investigate whether there had been any changes to the frequency or nature of their interactions with this technology, or with their level of satisfaction with it. Similarly, it could be interesting to build translation-related elements into other aspects of the course, or into other assignments for the course, so that elements of machine translation literacy would be integrated throughout the course rather than being concentrated in a single module. For example, in the module on media literacy and fake news, it could be beneficial to incorporate some news texts that have been translated from other languages with the help of machine translation and to consider whether this presents additional challenges to the evaluation of the material. Similarly, in the module on internet search strategies, it could be useful to assess the extent to which machine translation can help translate keywords. Conclusion Machine translation looks set to stay, and if used properly, it has the potential to help people access or produce information in multiple languages. In the context of higher education and research, English is currently a dominant lingua franca, meaning that many students and researchers must read and write in English, even though it may not be their native language. Meanwhile, organizations such as UNESCO, among others, have signaled a strong need to make the education and research arena more linguistically diverse, which means that everyone— including English speakers—will need to be able to access information in other languages. Machine translation technology has improved considerably in recent years with the introduction of artificial intelligence techniques such as machine learning; however, this technology is far from perfect and must be approached and applied with a critical eye. At present, universities—and particularly academic libraries—invest considerable effort in providing information literacy instruction to students, and while this instruction is increasingly integrating information technology, machine translation literacy does not yet seem to be widely taught. Machine translation literacy is less about learning how to use this tool (i.e., which buttons to push) and more about critical thinking, such as deciding whether, when, or why to use this technology, and how to interact with it meaningfully (e.g., human-computer interaction). In this way, machine translation literacy instruction aligns with the overall direction of ACRL’s (2015) Framework for Information Literacy for Higher Education, which encourages a conceptual rather than a skills-based approach. The pilot project to offer 144 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 machine translation literacy instruction to first-year undergraduates at the University of Ottawa as part of a broader course on “New Literacies for the Digital Age” was well received by both Anglophone and non-Anglophone students from a diverse range of programs in the Faculty of Arts. Overall, students found the module on machine translation literacy to be valuable, and they recommended that similar instruction be made available to all students. This pilot project would seem to confirm that machine translation literacy instruction can find a comfortable home within a broadened approach to information literacy instruction on university campuses. Academic librarians have a long history of delivering information literacy instruction in a university environment, and some are also stepping up to address other literacy needs in our evolving society, such as offering instruction in artificial intelligence literacy (Wheatley & Hervieux, 2020). With this in mind, academic librarians could become key partners for offering machine translation literacy instruction to students across all disciplines moving forward. In doing so, librarians could support and promote linguistic diversity and multilingual access to information. This in turn will diversify research and promote fairness and equity by making it possible for students and researchers from all corners of the globe to participate in scholarly conversations, regardless of the language they speak. Finally, given the interest in and relevance of machine translation for first-year university students, it is quite reasonable to accept that high school students, and likely even primary school students, are using this technology also (Stapleton & Leung, 2019), and may similarly benefit from some basic machine translation literacy at that level too. Courses, where information literacy is introduced in high schools or primary schools, could be a good place to consider incorporating elements of machine translation literacy, and school librarians may also have a role to play here. Other possibilities include working teachers or with offices on campus that do outreach activities in the community, such as the University of Ottawa’s Outreach unit, which offers a variety of experiential learning opportunities for children and teens, including clubs, workshops, summer camps, and credited high-school courses. Of course, whether we are considering academic librarians, school librarians, university professors, school teachers, or outreach teams as potential instructors for delivering machine translation literacy training, an important next step is considering how to “train the trainers.” In other words, members of these groups will first need to become machine translation literate themselves before being able to engage in meaningful instruction for others. I hope this report on a pilot project to deliver machine translation literacy instruction to first-year undergraduate students will provide some food for thought to be carried forward into future investigations of training the trainers and enhancing the content of machine translation literacy instruction. Acknowledgements This work was supported by an Insight Grant from the Social Sciences and Humanities Research Council of Canada (Grant number 435-2020-0089). Thanks are owed to the students who participated in the voluntary survey to offer feedback on the module. Appendix A: Survey on Machine Translation Use and Machine Translation Literacy Q1: I use machine translation to help me with the following tasks: • Read / understand a text in a non-dominant language 145 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 • Write / produce a text in a non-dominant language • Both read and write in a non-dominant language Q2: I use a machine translation tool (e.g., Google Translate, DeepL Translator, Baidu Translate or a similar tool) • Every day • 3 or 4 times a week • Once a week • 1 or 2 times a month • Less than once a month Q3: I use machine translation to translate texts for activities related to my [Select all that apply] • Job • Leisure activities • Studies Q4: I am MOST likely to use machine translation to help me with a language that I • Don't know • Know a bit • Know quite well • Am actively learning Q5: On the whole, how satisfied are you with the results of machine translation? • Completely satisfied. It meets all my needs. • Very satisfied. It meets most of my needs. • Moderately satisfied. It meets a reasonable number of my needs. • Not very satisfied. It meets a few of my needs. • Not at all satisfied. It does not meet any of my needs. Q6: If I did not have access to a free online machine translation system, I would do the following instead 146 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 • Pay a professional translator to translate (or edit) the text. • Translate the text myself. • Ask a friend or colleague to translate the text. • Nothing. I simply would not translate that text. Q7: Which of the key elements of machine translation literacy was the most surprising to you (i.e., something that you didn’t know before or hadn’t thought of in your previous experience of using online machine translation tools)? • Privacy/confidentiality • Academic integrity • Potential for algorithmic bias • Awareness of different tools • Awareness of different translation tasks • Improving the output by changing the input • Improving the output through revision • None. I already knew everything covered in the module. • Other (please specify) Q8: "Machine translation literacy" means learning about how machine translation tools work and how we can interact with these tools to get better results (such as by pre-editing the source text or post-editing the target text). For people who are not language professionals, I think that machine translation literacy is • Essential • Very important • Moderately important • Not very important • Not at all important Q9: Do you think that the University of Ottawa should make machine translation literacy training available to everyone? • Definitely. Machine translation literacy training will be useful to all students regardless of what they are studying. 147 https://jps.library.utoronto.ca/index.php/ijidi/index Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 • Probably. Machine translation literacy training could be useful to many students. • Maybe. Machine translation literacy training could be useful to some students. • Probably not. Machine translation literacy training would be useful to just a few students. • Definitely not. This topic is not relevant or interesting for any students unless they specifically want to become language professionals. Q10: Is there anything else that you would like to share with us on the topic of machine translation literacy? References Aharony, N., Julien, H., & Nadel-Kritz, N. (2020). Survey of information literacy instructional practices in academic libraries. Journal of Librarianship and Information Science, 52(4), 964–971. https://doi.org/10.1177/0961000619891762 Alhaisoni, E., & Alhaysony, M. (2017). An investigation of Saudi EFL university students’ attitudes towards the use of Google Translate. International Journal of English Language Education, 5(1), 72–82. https://doi.org/10.5296/ijele.v5i1.10696 Anazawa, R., Ishikawa, H., & Takahiro, K. (2013). Use of online machine translation for nursing literature: A questionnaire-based survey. Open Nursing Journal, 7(1), 22–28. https://doi.org/10.2174/1874434601307010022 Association of College and Research Libraries (ACRL). (2015). Framework for information literacy for higher education. http://www.ala.org/acrl/standards/ilframework Bowker, L. (2018). Machine translation and author keywords: A viable search strategy for authors with limited English proficiency? Proceedings of the 29th ASIS SIG/CR Classification Research Workshop. http://dx.doi.org/10.7152/acro.v29i1.15455 Bowker, L. (2020). From science fiction to reality: Representations of machine translation in the Canadian press. Circuit, 146. https://www.circuitmagazine.org/dossier-146/from- science-fiction-to-reality-representations-of-machine-translation-in-the-canadian-press Bowker, L., & Buitrago Ciro, J. (2019). Machine translation and global research: Towards an improved machine translation literacy in the scholarly community. Emerald. Canfora, C., & Ottmann, A. (2020). Risks in neural machine translation. Translation Spaces, 9(1), 58–77. https://doi.org/10.1075/ts.00021.can Ferguson, H., & Spinks, H. (2019). Overseas students in Australian higher education: A quick guide. Parliamentary Library Research Paper Series. Parliament of Australia. https://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;query=Id%3A%22librar y%2Flcatalog%2F01249150%22;src1=sm1 148 https://jps.library.utoronto.ca/index.php/ijidi/index https://doi.org/10.1177/0961000619891762 https://doi.org/10.5296/ijele.v5i1.10696 https://doi.org/10.2174/1874434601307010022 http://www.ala.org/acrl/standards/ilframework http://dx.doi.org/10.7152/acro.v29i1.15455 https://www.circuitmagazine.org/dossier-146/from-science-fiction-to-reality-representations-of-machine-translation-in-the-canadian-press https://www.circuitmagazine.org/dossier-146/from-science-fiction-to-reality-representations-of-machine-translation-in-the-canadian-press https://doi.org/10.1075/ts.00021.can https://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;query=Id%3A%22library%2Flcatalog%2F01249150%22;src1=sm1 https://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;query=Id%3A%22library%2Flcatalog%2F01249150%22;src1=sm1 Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 Forcada, M. L. (2017). Making sense of neural machine translation. Translation Spaces, 6(2), 291–309. https://doi.org/10.1075/ts.6.2.06for Houlihan, M., Wiley, C. W., & Click, A. B. (2017). International students and information literacy: A systematic review. Reference Services Review 45(2), 258–277. https://doi.org/10.1108/RSR-06-2016-0038 Hutchins, W. J., & Somers. H. L. (1992). An introduction to machine translation. Academic Press. Immigration, Refugees & Citizenship Canada (IRCC). (2017). International students with a valid permit on December 31st by gender and age, 1997 to 2017. http://www.cic.gc.ca/opendata-donneesouvertes/data/IRCC_FF_TR_2017_21_E.xls Institute of International Education (IEE). (2019). Open doors 2019. https://web.archive.org/web/20200712192127/https://p.widencdn.net/ye4ezw/Inter national-Students-Enrollment International Federation of Library Associations and Institutions (IFLA). (2013). Riding the waves or caught in the tide? Insights from the IFLA trend report. https://trends.ifla.org/insights-document Johnston, N., Houlihan, M., & Neindorf, J. (2018). International students’ expectations of information literacy instruction. In S. Kurbanoğlu, J. Boustany, S. Špiranec, E. Grassian, D. Mizrachi, & L. Roy (Eds.), Information literacy in the workplace (pp. 728–737). Springer Verlag. https://doi.org/10.1007/978-3-319-74334-9_74 Julien, H., Gross, M., & Latham, D. (2018). Survey of information literacy instructional practices in U.S. academic libraries. College & Research Libraries, 79(2), 179–199. https://doi.org/10.5860/crl.79.2.179 Koehn, P. (2010). Statistical machine translation. Cambridge University Press. Konno, K., Akasaka, M., Koshida, C., Katayama, N., Osada, N., Spake, R., & Amano, T. (2020). Ignoring non-English-language studies may bias ecological meta-analyses. Ecology and Evolution 10(13), 6373–6384. https://doi.org/10.1002/ece3.6368 Lee, S. (2020). The impact of using machine translation on EFL students’ writing. Computer Assisted Language Learning, 33(3), 157–175. https://doi.org/10.1080/09588221.2018.1553186 Monti, J. (2020). Gender issues in machine translation: An unsolved problem? In L. von Flotow & H. Kamal (Eds.), The Routledge Handbook of Translation, Feminism and Gender (pp. 457–468). Routledge. Montgomery, S. L. (2013). Does science need a global language? University of Chicago Press. Mundt, K., & Groves, M. (2016). A double-edged sword: The merits and the policy implications of Google Translate in higher education. European Journal of Higher Education 6(4), 387–401. https://doi.org/10.1080/21568235.2016.1172248 149 https://jps.library.utoronto.ca/index.php/ijidi/index https://doi.org/10.1075/ts.6.2.06for https://doi.org/10.1108/RSR-06-2016-0038 http://www.cic.gc.ca/opendata-donneesouvertes/data/IRCC_FF_TR_2017_21_E.xls https://web.archive.org/web/20200712192127/https:/p.widencdn.net/ye4ezw/International-Students-Enrollment https://web.archive.org/web/20200712192127/https:/p.widencdn.net/ye4ezw/International-Students-Enrollment https://trends.ifla.org/insights-document https://doi.org/10.1007/978-3-319-74334-9_74 https://doi.org/10.5860/crl.79.2.179 https://doi.org/10.1002/ece3.6368 https://doi.org/10.1080/09588221.2018.1553186 https://doi.org/10.1080/21568235.2016.1172248 Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 Nuñes Vieira, L. (2020). Machine translation in the news: A framing analysis of the written press. Translation Spaces 9(1), 98–122. https://doi.org/10.1075/ts.00023.nun Nuñes Vieira, L., O’Hagan, M., & O’Sullivan, C. (2020). Understanding the societal impacts of machine translation: A critical review of the literature on medical and legal use cases. Information, Communication & Society, 24(11), 1515-1532. https:/doi.org/10.1080/1369118X.2020.1776370 Nurminen, M. (2020). Raw machine translation use by patent professionals. A case of distributed cognition. Translation, Cognition & Behavior, 3(1), 100–121. https://doi.org/10.1075/tcb.00036.nur Nurminen, M., & Papula, N. (2018). Gist MT users: A snapshot of the use and users of one online MT tool. In J. A. Pérez-Ortiz (Ed.), Proceedings of the 21st annual conference of the European Association for Machine Translation: 28-30 May 2018 (pp. 199–208). European Association for Machine Translation. http://rua.ua.es/dspace/handle/10045/76049 O’Brien, S., & Ehrensberger-Dow, M. (2020). MT literacy: A cognitive view. Translation, Cognition & Behavior 3(2), 145–164. https://doi.org/10.1075/tcb.00038.obr Pérez Ortega, R. (2020, October 28). Science’s English dominance hinders diversity—but the community can work toward change. Science Magazine. https://www.sciencemag.org/careers/2020/10/science-s-english-dominance-hinders- diversity-community-can-work-toward-change Polkinghorne, S., & Julien, H. (2018). Treading water: Results from the longitudinal study of information literacy instruction in Canadian academic libraries, 1995–2017. Canadian Journal of Information and Library Science, 42(1–2): 69–93. https://muse.jhu.edu/article/717388 Statistics Canada. (2020). Table 15-10-0009-01. English, French and non-official languages spoken at home by geography, 2001 to 2016. https://doi.org/10.25318/1510000901- eng Stapleton, P., & Leung Ka Kin, B. (2019). Assessing the accuracy and teachers’ impressions of Google Translate: A study of primary L2 writers in Hong Kong. English for Specific Purposes, 56, 18–34. https://doi.org/10.1016/j.esp.2019.07.001 Translation Automation User Society (TAUS). (2010,). MT Post-editing guidelines. https://www.taus.net/academy/best-practices/postedit-best-practices/machine- translation-post-editing-guidelines UNESCO. (2020). Preliminary report on the first draft of the recommendation on open science. https://unesdoc.unesco.org/ark:/48223/pf0000374409.page UNESCO. (2021). Draft text of the UNESCO recommendation on open science. https://unesdoc.unesco.org/ark:/48223/pf0000376893.locale=en University of Ottawa. (2020). Quick Facts. https://www.uottawa.ca/institutional-research- planning/resources/facts-figures/quick-facts 150 https://jps.library.utoronto.ca/index.php/ijidi/index https://doi.org/10.1075/ts.00023.nun https://doi.org/10.1080/1369118X.2020.1776370 https://doi.org/10.1075/tcb.00036.nur http://rua.ua.es/dspace/handle/10045/76049 https://doi.org/10.1075/tcb.00038.obr https://www.sciencemag.org/careers/2020/10/science-s-english-dominance-hinders-diversity-community-can-work-toward-change https://www.sciencemag.org/careers/2020/10/science-s-english-dominance-hinders-diversity-community-can-work-toward-change https://muse.jhu.edu/article/717388 https://doi.org/10.25318/1510000901-eng https://doi.org/10.25318/1510000901-eng https://doi.org/10.1016/j.esp.2019.07.001 https://www.taus.net/academy/best-practices/postedit-best-practices/machine-translation-post-editing-guidelines https://www.taus.net/academy/best-practices/postedit-best-practices/machine-translation-post-editing-guidelines https://unesdoc.unesco.org/ark:/48223/pf0000374409.page https://unesdoc.unesco.org/ark:/48223/pf0000376893.locale=en https://www.uottawa.ca/institutional-research-planning/resources/facts-figures/quick-facts https://www.uottawa.ca/institutional-research-planning/resources/facts-figures/quick-facts Promoting Linguistic Diversity and Inclusion The International Journal of Information, Diversity, & Inclusion, 5(3), 2021 ISSN 2574-3430, jps.library.utoronto.ca/index.php/ijidi/index DOI: 10.33137/ijidi.v5i3.36159 University of Ottawa (2021). Interdisciplinary Study in Arts (AHL). University of Ottawa course catalogue. https://catalogue.uottawa.ca/en/courses/ahl/ Wang, B., & Ping, Y. (2020). Perceptions of machine translation and computer-aided translation by professionals and the general public: A survey study based on articles in professional journals and in the media. International Journal of Translation, Interpretation and Applied Linguistics 2(2), 1–14. https://doi.org/10.4018/IJTIAL.20200701.oa1 Way, A. (2020). Machine translation: Where are we at today? In E. Angelone, M. Ehrensberger- Dow, & G. Massey (Eds.), The Bloomsbury companion to language industry studies (pp. 311–332). Bloomsbury. https://doi.org/10.5040/9781350024960.0018 Wheatley, A., & Hervieux, S. (2020). Artificial intelligence in academic libraries: An environmental scan. Information Services & Use, 39(4), 347–356. https://doi.org/10.3233/ISU-190065 Wright, A., & Lawlor-Wright, T. (2018). Project Success and Quality. Routledge. Lynne Bowker (lbowker@uottawa.ca) is a Full Professor at the University of Ottawa, she holds a cross-appointment between the School of Information Studies and the School of Translation and Interpretation. She holds a Ph.D. in Language Engineering and has published on various aspects of language technologies, including co-authoring Machine Translation and Global Research (Emerald, 2019). 151 https://jps.library.utoronto.ca/index.php/ijidi/index https://catalogue.uottawa.ca/en/courses/ahl/ https://doi.org/10.4018/IJTIAL.20200701.oa1 https://doi.org/10.5040/9781350024960.0018 https://doi.org/10.3233/ISU-190065 mailto:lbowker@uottawa.ca Introduction Machine Translation and Machine Translation Literacy What is Driving the Need for Machine Translation Literacy? Pilot Project: Teaching Machine Translation Literacy to First-Year Undergraduate Arts Students Machine Translation Literacy Module Data-driven Approaches to Machine Translation Transparency Risk Assessment Interacting with Machine Translation Pedagogical Approaches and Evaluation of Learning Outcomes Feedback on the Machine Translation Literacy Module Discussion Conclusion Acknowledgements Appendix A: Survey on Machine Translation Use and Machine Translation Literacy References