Standards and affordances of 21st-century digital learning: Using the Experience Application Programming Interface and the Augmented Reality Learning Experience Model to track engagement in extended reality ACTA IMEKO ISSN: 2221-870X September 2022, Volume 11, Number 3, 1 - 6 ACTA IMEKO | www.imeko.org September 2022 | Volume 11 | Number 3 | 1 Standards and affordances of 21st-century digital learning: Using the Experience Application Programming Interface and the Augmented Reality Learning Experience Model to track engagement in extended reality Jennifer Wolf Rogers1, Karen Alexander2 1 People Accelerator, The Woodlands, Texas, USA 2 XRconnectED, Pittsburgh, Pennsylvania, USA Section: TECHNICAL NOTE Keywords: xAPI; ARLEM; augmented reality; virtual reality; data transformation; capability development; training; learning; education Citation: Jennifer Wolf Rogers, Karen Alexander, Standards and affordances of 21st-century digital learning: Using the Experience Application Programming Interface and the Augmented Reality Learning Experience Model to track engagement in extended reality, Acta IMEKO, vol. 11, no. 3, article 7, September 2022, identifier: IMEKO-ACTA-11 (2022)-03-07 Section Editor: Zafar Taqvi, USA Received March 1, 2022; In final form September 16, 2022; Published September 2022 Copyright: This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Corresponding author: Karen Alexander, PhD, Karen@XRconnectED.com 1. INTRODUCTION As the delivery of learning has been increasingly digitized over the past few decades, standards such as SCORM, the Sharable Content Object Reference Model, have helped to ensure interoperability across Learning Management Systems (LMS). But the 21st century has brought new, powerful tools for learning that enable educational and training experiences far richer than those available via a desktop or laptop computer. Virtual, Augmented, and Mixed Reality (VR, AR, MR) bring affordances to learning that far surpass what was previously available. Immersive, embodied learning in 3D environments, with interactive 3D objects and collaborative engagements with teachers or other learners, will revolutionize education and training as we know it. For these reasons, new standards that can aid in the capture of data about a learner’s experience have arisen, notably the Augmented Reality Learning Experience Model (ARLEM) and the Experience Application Programming Interface (xAPI). With xAPI and ARLEM, specific learner behaviour can be directly tracked and measured as it is shaped and/or changes in a specific interaction, thus permitting predictions of transfer from knowledge to demonstrable skill. Adoption of these standards is key to avoiding silos of information and data around associated learner development and behaviour change encoded in different systems and formats that make communication across them difficult. In Section 2 we will discuss how new technologies for learning demand new means of assessment. Section 3 introduces xAPI, its structure, and design principles for interoperable data structures. In section 4, we describe the Augmented Reality Learning Experience Model, or ARLEM. A case study illustrates xAPI in Virtual Reality in section 5. In section 6, the challenges and opportunities in using AR and VR and xAPI in early ABSTRACT The development of new extended reality (XR) technologies for learning enables the capture of a richer set of data than has previously been possible. To derive a benefit from this wealth of data requires new structures appropriate to the new learning activities these immersive learning tools afford. The Experience Application Programming Interface (xAPI) and the Augmented Reality Learning Experience Model (ARLEM) standards have been developed in response, and their adoption will help ensure interoperability as XR learning is more widely deployed. This paper briefly describes what is different about XR for learning and provides an account of xAPI and its structures as well as design principles for its use. The relationship between environmental context and ARLEM is explained, and a case study of a VR experience using xAPI is examined. The paper ends with an account of some of the promises for collecting data from early childhood learning experiences and an unsuccessful attempt at a study using Augmented Reality with young children. mailto:Karen@XRconnectED.com ACTA IMEKO | www.imeko.org September 2022 | Volume 11 | Number 3 | 2 childhood education are outlined. In conclusion, we point toward possibilities for future developments in the fields of extended reality (XR) for training and interoperable standards for data collection. 2. NEW WAYS OF LEARNING, NEW WAYS OF MEASURING Among the affordances of Virtual Reality that make it such an effective tool for learning is the way that it engages the senses of proprioception and presence while permitting movement and gesture as the learner interacts with the virtual world and content within it. Research has shown that large gestures and bodily movements enhance the acquisition and retention of knowledge, and VR is able to deliver experiences that allow such movement, unlike the small, hands-only interactions we have when using a computer keyboard interface for digital learning [1]. AR and MR deliver digital content within a real-world environment, and such content appears to the learner as present in space alongside them. The learner can modify the size of the virtual objects with which they are interacting or rotate them for a different view, walk around them, and experience the content in a fully embodied, present manner. XR technologies engage the body as it is inhabited in space and tap into embodied cognition. As researcher Mina Johnson- Glenberg notes, “Human cognition is deeply rooted in the body’s interactions with the world and our systems of perception.” But Johnson Glenberg also acknowledges that “For several decades, the primary interface in educational technology has been the mouse and keyboard; however, these are not highly embodied interface tools” [1]. XR permits the collection of data from bodily movements and interactions. One of the benefits of this can be seen in recent research that shows that the combination of physical movement and a gaming element in learning experiences for children not only enhances their ability to learn, but in fact also enhances cognitive function [2], [3]. With such new tools that can engage the body, “we need better Learning Analytics for real-time and real-world interaction” [4]. 3. THE EXPERIENCE API 3.1. Experience (xAPI) Data The experience (xAPI) data is a modern data standard that is uniquely positioned to measure and track human behaviour in learning scenarios in a manner that more closely approximates their behaviour in everyday situations and experiences. The xAPI differs from traditional, compliance-based, linear forced-choice methods of measuring learning (such as scoring responses to a finite series of multiple-choice questions and distractors). Instead, it takes a more dynamic and analytical approach, acknowledging the often less-than-predictable nature of human response, and allowing learners to immerse themselves in a variety of contexts and respond organically (and, in some cases, automatically, without an intentional initiation of cognitive processing) to a variety of stressors and/or motivators in a manner that more closely approximates their true everyday behaviour in the real world. Additionally, xAPI provides a standard JavaScript Object Notation (JSON) syntax that facilitates the collection and subsequent analysis of these learner behaviours across learning scenarios and experiences, providing assurances around degree of predictability of a learner response over time, as well as the opportunity to correlate behaviour in practice/simulated environments with behaviour from the same individual/group of individuals, and potential real-world outcomes arising from that behaviour, in the physical world. 3.2. Key Considerations for Structuring Data Collection using xAPI Though xAPI is a broad standard and syntax aimed at increasing the efficacy of all modalities associated with learning experience, immersive environments are, in particular, well- suited to this type of measurement. In XR experiences, there is a unique opportunity for learners to embody specific roles and interact with specific environments and objects, as well as demonstrate behaviours that are highly analogous to the physical world. When structuring xAPI data collection in these immersive scenarios, it is important to consider the ways in which people, objects, and actions/behaviours might interact with one another, the outcomes and/or consequences that may arise from these interactions, and how these interactions and interdependencies might be captured. Ideally, data collection will be structured using the base syntax to answer the following types of questions: ● What is the environmental context? ● Who/what is present with the learner in this environment? ● How does the environment (and/or people or objects within it) trigger a learner behaviour? ● How may learner behaviour be described in such a way that it may be generalized across other immersive environments? Is this behaviour tracked in the physical world in the form of metrics and/or key performance indicators (KPIs)? How is it described/reported? ● Does the environmental context change as a result of the learner’s behaviour(s)? Is there a natural resolution to a specific problem/tension? If so, how is this described? 3.3. Key design principles for human metrics in XR Beyond the decisions driving specific ontologies and/or taxonomies to describe JSON statement syntax within specific interactions/scenarios, there are also key design principles to consider when designing an overall interoperable data structure meant to support and analyse this syntax: 1. Human metrics design should be scalable. When constructing data schemas and frameworks to support xAPI, it is imperative that human-centred design is Table 1. Overview of styles and font sizes used in this template. JSON Syntax Field Field Description Adult Learner Example Child Learner Example Subject/ Actor Identifiers and/or descriptors for individual learners/groups Jane Jill Verb Action that learner takes in the scenario released donned Object Object and/or person that learner interacts with the pressure valve a coat Context Optional Extension - information to describe the context of the behaviour (e.g. location coordinates, physical barriers, emotional factors, etc.) …when operating parameters [began to trend out of limits] …when snow fell from the sky Results Optional Extension - information to describe results and/or resolution of the initial context, based upon learner action ….which resulted in pressure stabilization ….which resulted in health stats increasing ACTA IMEKO | www.imeko.org September 2022 | Volume 11 | Number 3 | 3 employed to ensure that the resulting metrics include verbs, results, and contexts that may be shared across a wide variety of XR scenarios/experiences for the same user(s) over time. In this case, the human/individual is the constant as they “travel” through a variety of XR simulations/scenarios and their behaviour in these different contexts must be described in a manner that lends itself to trend analysis and/or situational anomalies. Additionally, as humans are not truly “compartmentalized” within a particular function/domain of their life, frameworks should be constructed in such a way that provides for longevity around the description of behaviour across broad ecosystems an individual may transverse, in both the physical and virtual world (e.g. familial networks, social networks, academic networks, professional networks, etc.). 2. Actions/verb selection should minimize bias and/or interpretation, wherever possible. As the range of human behaviour is vast, and “appropriate” responses may vary across simulations/scenarios, every effort should be taken to avoid introduction of verbs into an ontology/taxonomy and/or individual interaction that imply some sense of judgement and/or right and wrong responses. Instead, the verb portion of the syntax itself should merely describe the user behaviour in the statement itself, utilizing the results and context extensions where appropriate to contextualize the nature of appropriateness of the response in light of the conditions at a particular point in time in an interaction. 3. Human metrics data should be interoperable and machine-readable. Deviation from JSON format and xAPI-specific context should be avoided, as both are open source standards that ensure that data recorded during immersive experiences/scenarios may be created, managed, and analysed broadly across existing and future platforms and technology, particularly those that pertain to measurement of human development, capabilities, and performance (e.g. student/workforce management systems, capability management and workforce planning systems, learning management systems, assessment management systems, performance management systems, etc.). Additionally, metrics should be constructed in such a manner that lends them to be analysed by machine learning principles such as Natural Language Processing and, in some cases, subsequently actioned further by artificial intelligence in subsequent interactions, as described below. 4. Individual actors should be differentiated. Appropriate and differentiated identifiers should be utilized in actor fields that, though anonymized where necessary, may be correlated back to a specific learner. This standardized practice ensures that interaction data is cumulative across a variety of immersive scenarios/simulations and that resulting data may be analysed for specific behavioural trends in a variety of contexts. Additionally, this convention provides an affordance whereby user experience and/or nuances of a specific scenario’s context in future trials may be personalized to target specific growth/performance targets of the specific individual. 5. Overall data sets, and the ontologies and/or taxonomies inherent within them, should be human-readable. To enable the proper analysis of human behaviour within specific contexts and the subsequent growth and performance of people over time, particularly as it pertains to informed practice for educational and human resources professionals responsible for guiding academic and career growth, it is imperative that virtual interaction descriptors are chosen that most closely approximate observable human behaviours in the physical world. 4. AUGMENTED REALITY LEARNING EXPERIENCE MODEL One recent iteration of the xAPI standard as it pertains to XR specifically lies in the Augmented Reality Learning Experience Model, otherwise known as ARLEM [5]. In this formulation, as described by Secretan, Wild, & Guest, Performance = Action + Competence. “An action is the smallest possible step that can be taken by a user in a learning activity,” they say, and “competence becomes visible when learners perform” [4]. Accordingly, ARLEM aims to solve many of the challenges faced by colleagues in industrial and/or operational environments, whereby a worker's ability to take the right action at the right time is paramount. Traditional measurement protocols were previously oriented toward providing flat, two- dimensional “performance support” to these colleagues, in the form of viewing a schematic or standard operating procedure, watching a video of a colleague performing a process step, etc. and were centred around the passive “completion” of learning, which inferred only that an individual had accessed content and, in some cases, progressed through to the end. In some cases, additional measurement in the form of more formal assessment required learners to answer knowledge-based questions around the operational process. These assessments, often based on rote memorization, are incapable of measuring what actions a worker will take when faced with a particular situation. With the ARLEM and xAPI standards, actions performed may now be prioritized over memorization of facts. 4.1. Environmental triggers A hallmark of the ARLEM standard itself is its unique application of environmental “triggers” prompting human behaviour or experience; in this case, in the workplace specifically. By monitoring the environmental context and its operational parameters, ARLEM provides an association between situational context and appropriate supports provided to the end user to assist them in taking the appropriate action. 4.2. Operational excellence key performance indicators Utilizing the differentiators between environmental triggers and human activity, and further applying the results extension in the xAPI syntax allows for a unique opportunity to assess the effectiveness of augmented reality in the flow or work. In some instances, IoT data associated with operational processes and/or equipment that is trending out of limits/acceptable parameters may “trigger” an augmented reality support layer, comprised of appropriate schematics, videos, and even remote operations assistance from a geographically distant subject matter expert, over the real world and deployed to the correct individual in proximity to prompt human intervention to normalize the operational process. The specific IoT parameters may be recorded in the “context” extensions in xAPI syntax so that the ACTA IMEKO | www.imeko.org September 2022 | Volume 11 | Number 3 | 4 resulting human action can be understood in relation to operational KPIs. Furthermore, resolution of the IoT data associated with the operational process, where available, may be recorded. 5. XAPI IN VIRTUAL REALITY - AN OPERATIONAL CASE STUDY In light of the success with xAPI in the ARLEM standard, this paper’s authors hypothesized that utilization of the xAPI base syntax, associated extensions, and concept of operational triggers would also lend themselves quite well to measurement of learning experiences and other immersive experiences, such as virtual reality. To test this theory, an existing 360 video-based virtual reality interaction, centred around operational safety and risk in a heavy-manufacturing environment, and for which existing learning measurement data was available, was selected. In the selected VR scenario, an operator at a plant was given a job task to complete, and, in so doing, a set of dangers/risks in the environment were layered throughout the scenes (see Figure 1). The scenario itself had been created to simulate the work environment and to increase plant operators’ ability to recognize and mitigate risks present in their working environment without overt prompts to do so. Existing dashboards associated with the scenario were then examined to ascertain the likelihood of correlation with actual plant operator behaviour in the workplace. Analysis of this information demonstrated that, though the immersive environment lent itself quite well to application of the xAPI syntax, more traditional learning measurement methods, such as scoring of responses as right/wrong were employed, along with a scoring mechanism that subtracted points for each incorrect response (or lack thereof) within the simulation. Elapsed time in the interaction was also reported (see Figure 2). Because the original measurement design did not approximate the type of behaviours observed, collected, and reported in the workplace, the effectiveness of the immersive learning experience, as compared to other traditional learning modalities, was difficult to determine across various learners and/or similar scenarios. As expected, the nature of the existing data did not give a great deal of information that would allow inference as to the immersive scenario’s ability to shape and/or change human behaviour in a manner that one could reasonably assume would make them safer and/or more productive in their actual jobs in the real world. 5.1. Enhanced data set - VR - safety/risk In order to enhance the measurement of human interaction within the simulation in a manner that could be more closely correlated to human behaviour in the workplace, we obtained a sample of the historical data and, along with a careful analysis of the operational risks present in the design of the VR scenario, completed a data transform that allowed for xAPI syntax to be expressed in a manner that identified environmental context, as well as subsequent user action/behaviour, and results of this behaviour in relation to the original “triggering” context itself (see Figure 3). xAPI-formatted JSON statements were then created, using existing available data fields, as well as additional data available and yet not previously recorded (see Figure 4). These statements were subsequently loaded into a Learning Record Store (LRS), and subsequent dashboards were created to demonstrate the relationships between environment and corresponding user behaviour, as well as relationships with existing competency frameworks related to health and safety and designed by the U.S. Department of Labor [6] (see Figure 5). As compared to the original, more traditional measurement design, these dashboards contextualized human behaviour in conditions of varying risk in a manner similar to the way health and safety behaviours are measured in real operational environments and gave a clearer indication of human behaviour that might be demonstrated in these operational environments, based upon actions measured and tracked in the immersive simulation. 5.2. Key findings The results of this case study, as well as reference implications of the ARLEM standard, as previously described by Wild et al., demonstrate that it is possible to map additional information to existing XR datasets to infer “shaping of human behaviour”, as well as potentially correlate actions in simulated and/or augmented environments to those observed and recorded in the real world. Furthermore, the original “triggering condition/context” metaphor suggested in the ARLEM standard was found to extend more broadly to all experiential-based learning experiences and work in harmony with the xAPI data syntax associated with learning experiences. To enable effective measurement across scenarios and users, we created the following taxonomies for use in the xAPI syntax, designed to be utilized broadly across XR scenarios measuring hazard identification and mitigation of operational risk: Figure 1. Selected VR scenario, designed to measure learner’s ability to identify and mitigate operational risk in a simulated environment. Figure 2. Measurement of non-xAPi-compliant learner behaviour in a VR scenario. ACTA IMEKO | www.imeko.org September 2022 | Volume 11 | Number 3 | 5 1. Stimulus (Context) - Taxonomy to describe starting condition(s) within a scenario. 2. Trigger (Context) - Taxonomy to describe expected levels of perceived risk, competing human priorities (dilemmas), and/or intentional environmental distractor(s). 3. Response (Results)- Taxonomy to describe user roles and levels within the scenario, ending condition(s), correlation to open competency frameworks, haptics, eye gaze, biometrics. While these taxonomies were created to facilitate historical data transformation from an existing VR scenario into an xAPI- compliant dataset, it is acknowledged here that actual ontologies are more desirable. In order to promote interoperability and the broad reach of xAPI as a measurement standard in XR learning scenarios, it is recommended that representative ontologies and/or taxonomies be developed that can inform widespread historical data transformation in the XR space, as well as serve as guides around measurement designs for additional immersive scenarios that are designed in future. These advancements, we believe, will provide a powerful measurement tool designed to effectively measure, describe, and perhaps even predict human behaviour in context in a manner that speaks to not only the knowledge that individuals possess, but also to their performed skill/competency level across a variety of contexts. This has major implications for the ways in which we measure and assess specific and generalized learnings in individuals over time. 6. DESIGNING AUGMENTED REALITY LEARNING EXPERIENCES FOR EARLY CHILDHOOD EDUCATION: OPPORTUNITIES AND CHALLENGES In a literature review of VR, AR, and MR in K-12 education, Maas and Hughes [7] noted that studies found increased attitude, motivation, engagement, performance/learning outcomes, and 21st Century skills and competencies with the use of these XR technologies. But they also emphasized that it was challenging to find studies on XR in K-12 education in comparison to such research in higher ed. The reasons they theorized such research was difficult to find were the rapid development of the technology, which means that it has only recently become available; the lack of XR content for this age group; and a lack of resources, making XR unevenly available within countries and around the globe. With early childhood education, these challenges are even more daunting. When the current paper was first proposed, the intention was to discuss research on an early childhood AR application built on a platform compatible with xAPI. That research was to be informed by work indicating that bodily engagement enhances learning outcomes and improves executive function. Research on children’s learning by Eng et al. using a modified VR game combined with a cognitive task suggests that the combination of “exergames” with a learning task not only improves performance on that task, but can actually improve cognitive ability and executive function [2], [3]. While Eng et al. studied children using VR along with measurements in fMRI as well as teacher assessments, in the case of the proposed new research, AR was to be used. Augmented reality is much more accessible than virtual reality for early childhood and K-12 education, which increases the opportunities to study it. For example, tablets were already in use at the lab school where the study is expected to take place. More AR platforms and experiences are becoming compatible with xAPI. Thus, we still believe there is great promise for the study in development and for other such studies. In addition, AR- compatible smartphones and tablets are also quite common in households in 2022, which means that many children would be able to participate in AR learning experiences from home. Ironically, such AR learning experiences could greatly benefit the many children who have not been able to attend school in person in the time of Covid and who have missed precious learning time at a key moment in their young lives. This proved challenging on many fronts. The challenges included ensuring that the design principle around differentiation of individual actors be followed. In practical terms, this would involve ensuring that each child be supplied with a device assigned to and associated with that particular child to avoid relying on young children to log into the device with the appropriate credentials, which would alleviate pressures on teachers in the classroom context. For young children who are just beginning to learn to read, audio prompts may need to be incorporated into the experience, or each student may need to be guided by the teacher, making the process resource intensive. The continuing effects of the pandemic also had a significant impact on the project because the early childhood education lab where the research was to be conducted experienced repeated closures and disruptions. As a result, the study has not yet formally begun. Figure 3. Measurement of learner behaviour in the same VR scenario, utilizing the xAPI standard, in conditions of Medium Risk (context). Figure 4. Measurement of predicted results based upon learner behaviour in the same VR scenario, mirroring operational consequences in the workplace. Figure 5. Number of user actions in VR scenario showing evidence of competency in U.S. Department of Labor standard: “Assessing material equipment, and fixtures for hazards”. ACTA IMEKO | www.imeko.org September 2022 | Volume 11 | Number 3 | 6 7. CONCLUSIONS AND NEXT STEPS Some of the early childhood learning that we would like to measure and that are appropriate for structuring with xAPI include self-concept skills such as proprioception, awareness of others in space, and geometry and spatial sense skills such as recognition of 3D shapes and their persistent form when rotated, for example. Learning of appropriate behaviours for self-care and social responses to others give situational triggers that are also targets for studies of the use of xAPI with young children. In general, future steps include continuing to explore specific elements of xAPI profiles appropriate for XR with regards to experiential learning, considering the relationship between knowledge-based concepts and experiential skills, and examining xAPI profiles for XR across the talent pipeline (from Pre-K to adult) in order to hypothesize about implications for the future of education. ACKNOWLEDGEMENT Special thanks to colleagues at Warp VR, namely Guido Helmerhorst and Menno van der Sman for contributing scenarios, historical data subsets, and transformation support. This effort would not have been possible without your spirit of innovation and commitment to open source-initiatives and practices. REFERENCES [1] M. Johnson-Glenberg, Immersive VR and Education: Embodied Design Principles that Include Gesture and Hand Controls, Frontiers in Robotics and AI, July 2018. DOI: 10.3389/frobt.2018.00081 [2] C. M. Eng, M. Pocsai, F. A. Fishburn, D. M. Calkosz, E. D. Thiessen, A. V. Fisher, Adaptations of Executive Function and Prefrontal Cortex Connectivity Following Exergame Play in 4- to 5-year old Children. [3] C. M. Eng, D. M. Calkosz, S. Y. Yang, N. C. Williams, E. D. Thiessen, A. V. Fisher, Doctoral Colloquium - Enhancing brain plasticity and cognition utilizing immersive technology and virtual reality contexts for gameplay. 6th International Conference of the Immersive Learning Research Network (iLRN), San Luis Obispo, CA, USA, 21-25 June 2020, pp. 395-398. DOI: 10.23919/iLRN47897.2020.9155120 [4] J. Secretan, F. Wild, W. Guest, Learning Analytics in Augmented Reality: Blueprint for an AR / xAPI Framework, in TALE 2019, Yogyakarta, Indonesia, 10-13 December 2019, pp. 1-6. DOI: 10.1109/TALE48000.2019.9225931 [5] IEEE Standard for Augmented Reality Learning Experience Model, IEEE Std 1589-2020, April 2020, pp. 1–48. DOI: 10.1109/IEEESTD.2020.9069498. [6] Credential Finder. Online [Accessed 15 September 2022] https://credentialfinder.org/competencyframework/1928/Enginee ring_Competency_Model_U_S__Department_of_Labor_(DOL) [7] M. J. Maas, J. M. Hughes, Virtual, augmented, and mixed reality in K-12 education: A review of the literature. Technology, Pedagogy, and Education 29 (6) 2020. DOI: 10.1080/1475939X.2020.1737210 https://doi.org/10.3389/frobt.2018.00081 https://doi.org/10.23919/iLRN47897.2020.9155120 https://doi.org/10.1109/IEEESTD.2020.9069498 https://doi.org/10.1109/IEEESTD.2020.9069498 https://credentialfinder.org/competencyframework/1928/Engineering_Competency_Model_U_S__Department_of_Labor_(DOL) https://credentialfinder.org/competencyframework/1928/Engineering_Competency_Model_U_S__Department_of_Labor_(DOL) https://doi.org/10.1080/1475939X.2020.1737210