Selected Papers of #AoIR2023: The 24th Annual Conference of the Association of Internet Researchers Philadelphia, PA, USA / 18-21 Oct 2023 Suggested Citation (APA): Medina Perea, I., Bates, J., Fratczak, M., Kennedy, H., Ochu, E. (2023, October). Beliefs, values and emotions in practitioners’ engagements with learning analytics in higher education. Paper presented at AoIR2023: The 24th Annual Conference of the Association of Internet Researchers. Philadelphia, PA, USA: AoIR. Retrieved from http://spir.aoir.org. BELIEFS, VALUES AND EMOTIONS IN PRACTITIONERS’ ENGAGEMENTS WITH LEARNING ANALYTICS IN HIGHER EDUCATION Itzelle Medina Perea The University of Sheffield Jo Bates The University of Sheffield Monika Fratczak The University of Sheffield Helen Kennedy The University of Sheffield Erinma Ochu University of the West of England Philanthropists, internet entrepreneurs, EdTech companies, AI enthusiasts, and policymakers around the world have promoted the idea that big data and learning analytics (LA) have the potential to revolutionise education (Roberts-Mahoney et al., 2016) - and not necessarily in ways that many ‘revolutionaries’ would wish. LA, defined as the continuous measurement, collection, analysis and reporting of data about learners and their context (Gašević et al., 2015, p. 1), is increasingly being used to track and evaluate what students do in internet-mediated environments (Williamson, 2017). The movement supporting the adoption of AI-based solutions in education is in sync with the belief that technology has the power to bring progress. Some seem to embrace the belief that the adoption of AI-based solutions is the remedy to old-fashioned systems that limit students’ potential (Microsoft, 2021). A number of benefits have been attributed to the adoption of AI-based solutions in the education sector. It has been argued, for example, that they help enhance student success (Jisc, 2018) and minimise inequalities (Microsoft, 2021). It has also been claimed that by identifying patterns in student data, educators can promptly identify students’ needs or weaknesses and act based on these data to improve their learning experience (Luckin et al., 2016). A growing body of literature has questioned these benefits, reflecting on the potential negative implications of adopting AI-based solutions and raising a number of concerns about the current developments in the field (Jarke & Macgilchrist, 2021; Prinsloo, 2020; Williamson, 2017; Yu & Couldry, 2020). Studies of a critical nature have explored, for example, issues such as surveillance and loss of privacy, bias and discrimination, and the marketisation of education (Castañeda & Selwyn, 2018; Hartong & Förschler, 2019; Roberts-Mahoney et al., 2016; Yu & Couldry, 2020). Despite this growing interest among researchers, we know little about how the beliefs, values and feelings of different groups of educational practitioners shape how they engage with AI-driven learning analytics technologies and influence the evolution of the cultures of practice shaping adoption of learning analytics. In this paper, we report on research that asks: how do culturally situated beliefs, values and emotions shape practitioners’ engagements with narrow AI in different contexts of practice? The research project as a whole examines these cultures of practice across three contrasting contexts: higher education, science, and arts practice. Here we will discuss early findings from one of these contexts – learning analytics in higher education. The theoretical approach underlying the study recognises the interconnected nature of values, beliefs and emotions in the constitution of social phenomena (Ahmed, 2014; Bericat, 2016; Ustek-Spilda et al., 2019), in contrast to approaches that theorise reason and emotion as binary entities. Drawing on work by e.g. Bates (2017) and Kitchin and Lauriault (2014), we understand the values and beliefs within a given context of practice to be shaped by external social forces. We understand beliefs as ideas that people assert to be true, recognising widely accepted beliefs as “loose ideologies” (Harrison & Boyd, 2018) or fragmented common sense (Hall, 1987). Values are specific types of beliefs that people hold about, for example, what is right or wrong. Finally, we understand emotions to be “social feelings…conditioned by the culture of society..its norms, values, ideas, beliefs” (Bericat, 2016). Previous research has identified a range of beliefs people have about data and AI, such as “dataism” (van Dijck, 2014) and “data-centric relationality (Ricaurte, 2019), values around emergent tech practices e.g. ‘human’ values such as liberal values, human-centric values (Pasquale, 2020), or emotional responses to data and machine learning systems (Eubanks, 2018; Kennedy & Hill, 2018). However, values, feelings and beliefs have not been considered in interaction with one another. Our data collection includes observations, focus groups, and interviews. We interviewed 33 participants in different roles: staff from Jisc, a provider of digital services for UK higher education (n~5), and staff from three different UK universities, including academic staff from different disciplines (n~17), learning technologists and others responsible for LA system implementation (n~7), and student advisors (n~4). We also conducted four focus groups with 3-5 people in different roles and participant observation in Jisc LA network events and mailing list. Data are being analysed using a combination of thematic analysis (Braun & Clarke, 2006), and close critical reading around key findings. Ethical approval for the study was gained from The University of Sheffield. It is clear that the beliefs, values and emotions of practitioners about the use of LA in higher education, and resultantly in their own work, are strongly influenced by participants’ perceptions of what is happening externally to their organisation. While educational practitioners recognise the potential benefits of LA, their belief systems do not seem to embrace the revolutionary discourse promoted by some EdTech providers and other stakeholders such as internet entrepreneurs, in which tools are promised to improve retention, increase efficiency, and ‘fix’ education. For the educational practitioners we engaged with, learning analytics is a tool that, if used carefully and responsibly, has the potential to enhance educators' work, but they are averse to the belief that it is “magic or [an] easy solution” (Interviewee, learning technologist, Oct-2022). Practitioners in all roles express frustration about the proliferation of the revolutionary discourse that tends to over-claim the benefits of these tools. They believe that more critical voices in the field willing to challenge the values and beliefs being promoted by EdTech providers and other powerful stakeholders are needed. Practitioners' beliefs and values regarding the use or adoption of LA reflect the human-centric values (Pasquale, 2020) they hold in relation to their work in education. Participants reflect on the importance of keeping students at the forefront of their practice and seeing beyond the data. Some teachers avoid looking at data before engaging with students because they worry it will influence their relations with them or will ‘dehumanise’ their students, which goes against their values. Furthermore, they value interactions with students over obtaining insights from LA tools; for them, systems should not replace meaningful conversations or undermine the possibility of developing meaningful and positive student-teacher relationships. One of the big promises of LA dashboards is that their outputs provide new insights about learners. However, teachers believe that in most cases, these tools tend to confirm what they already know about students and only on rare occasions, indicate something new. For some, getting confirmation from the system about their existing beliefs about students reassures them that their “thinking is right” (Interviewee, academic staff, Nov-2022) while others believe it may be a better idea for institutions to prioritise exploring other strategies to support learners’ journeys, even if such strategies do not involve the use of AI or other new technologies. Findings of this research highlight that education practitioners believe there are potential benefits of using internet-mediated learning analytics, however do not embrace the idea that the mere implementation of these tools will improve education. This challenges the revolutionary discourse promoted by EdTech companies, internet entrepreneurs, AI enthusiasts, and other powerful stakeholders. Through enhancing our understanding of the beliefs, values and emotional engagements of education practitioners with data and learning analytics systems, we aim to contribute to empower practitioners in higher education and relevant stakeholders to foster the development of critical and reflective data cultures that are able to exploit the possibilities of learning analytics, while being critically responsive to their societal implications and limitations. References Ahmed, S. (2014). Cultural Politics of Emotion. Edinburgh University Press. Bates, J. (2017). Data cultures, power and the city. In Data and the City (pp. 189– 200). Routledge. https://eprints.whiterose.ac.uk/120825/3/14_Bates%20v3- 2%20%28final%20edited%20JB261016%29.pdf Bericat, E. (2016). The sociology of emotions: Four decades of progress. Current Sociology, 64(3), 491–513. https://doi.org/10.1177/0011392115588355 Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology,. 3(2), 77–101. Castañeda, L., & Selwyn, N. (2018). More than tools? Making sense of the ongoing digitizations of higher education. International Journal of Educational Technology in Higher Education, 15(1), 22. https://doi.org/10.1186/s41239-018-0109-y Eubanks, V. (2018). Automating Inequality: How high-tech tools profile, police, and punish the poor (First edition). St. Martin’s Press. Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014- 0822-x Hall, S. (1987). Gramsci and Us. Marxism Today, June 1987, 16–21. Harrison, K., & Boyd, T. (2018). The role of ideology in politics and society. In Understanding political ideas and movements (pp. 135–153). Manchester University Press. https://www.manchesteropenhive.com/view/9781526137951/9781526137951.00011. xml Hartong, S., & Förschler, A. (2019). Opening the black box of data-based school monitoring: Data infrastructures, flows and practices in state education agencies. Big Data & Society, 6(1), 2053951719853311. https://doi.org/10.1177/2053951719853311 Jarke, J., & Macgilchrist, F. (2021). Dashboard stories: How narratives told by predictive analytics reconfigure roles, risk and sociality in education. Big Data & Society, 8(1), 20539517211025560. https://doi.org/10.1177/20539517211025561 Jisc. (2018, September 20). Can artificial intelligence transform the teaching and learning experience? Times Higher Education (THE). https://www.timeshighereducation.com/hub/jisc/p/can-artificial-intelligence-transform- teaching-and-learning-experience Kennedy, H., & Hill, R. (2018). The Feeling of Numbers: Emotions in Everyday Engagements with Data and Their Visualisation. Sociology. https://journals-sagepub- com.sheffield.idm.oclc.org/doi/full/10.1177/0038038516674675 Kitchin, R., & Lauriault, T. (2014). Towards Critical Data Studies: Charting and Unpacking Data Assemblages and Their Work (SSRN Scholarly Paper ID 2474112). Social Science Research Network. https://papers.ssrn.com/abstract=2474112 Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence unleashed: An argument for AI in Education. Pearson. https://www.pearson.com/content/dam/one-dot-com/one-dot-com/global/Files/about- pearson/innovation/open-ideas/Intelligence-Unleashed-v15-Web.pdf Microsoft. (2021). Understanding the new learning landscape: Accelerating learning analytics and AI in education. Microsoft. https://edudownloads.azureedge.net/msdownloads/Microsoft-Accelerating-Learning- Analytics-and-AI-in-Education.pdf Pasquale, F. (2020). New Laws of Robotics—Frank Pasquale. https://www.hup.harvard.edu/catalog.php?isbn=9780674975224 Posner, Z. (2017, November 21). Artificial Intelligence Comes to Learning. McGraw Hill. https://www.mheducation.com/ideas/artificial-intelligence-comes-to-learning.html Prinsloo, P. (2020). Of ‘black boxes’ and algorithmic decision-making in (higher) education – A commentary. Big Data & Society, 7(1), 2053951720933994. https://doi.org/10.1177/2053951720933994 Ricaurte, P. (2019). Data Epistemologies, The Coloniality of Power, and Resistance. Television & New Media, 20(4), 350–365. https://doi.org/10.1177/1527476419831640 Roberts-Mahoney, H., Means, A. J., & Garrison, M. J. (2016). Netflixing human capital development: Personalized learning technology and the corporatization of K- 12 education. Journal of Education Policy, 31(4), 405–420. https://doi.org/10.1080/02680939.2015.1132774 Ustek-Spilda, F., Powell, A., & Nemorin, S. (2019). Engaging with ethics in Internet of Things: Imaginaries in the social milieu of technology developers. Big Data & Society, 6(2), 205395171987946. https://doi.org/10.1177/2053951719879468 van Dijck, J. (2014). Datafication, dataism and dataveillance: Big Data between scientific paradigm and ideology. Surveillance & Society, 12(2), 197–208. https://doi.org/10.24908/ss.v12i2.4776 Williamson, B. (2017). Big data in education: The digital future of learning, policy and practice. SAGE. https://ebookcentral.proquest.com/lib/sheffield/detail.action?docID=4923811 Yu, J., & Couldry, N. (2020). Education as a domain of natural data extraction: Analysing corporate discourse about educational tracking. Information, Communication & Society, 1–18. https://doi.org/10.1080/1369118X.2020.1764604