Australasian Journal of Educational Technology, 2017, 33(6). 69 Improving paramedic distance education through mobile mixed reality simulation James Birt Bond University Emma Moore and Michael Cowling Central Queensland University There is growing evidence that the use of simulation in teaching is a key means of improving learning, skills, and outcomes, particularly for practical skills. In the health sciences, the use of high-fidelity task trainers has been shown to be ideal for reducing cognitive load and leading to enhanced learning outcomes. However, how do we make these task trainers available to students studying at a distance? To answer this question, this paper presents results from the implementation and sustained testing of a mobile mixed reality intervention in an Australian distance paramedic science classroom. The context of this mobile mixed reality simulation study, provided through a user-supplied mobile phone incorporating 3D printing, virtual reality, and augmented reality, is skills acquisition in airways management, focusing on direct laryngoscopy with foreign body removal. The intervention aims to assist distance education learners in practising skills prior to attending mandatory residential schools, building a baseline equality between those students who study face to face and those at a distance. Outcomes from the study showed statistically significant improvements in the use of the simulation across several key performance indicators in the distance learners, but also demonstrated problems to overcome in the pedagogical method. Introduction As educators, we are increasingly surrounded by a new breed of individual who tackles problems in new and different ways through technology (Clark & Mayer, 2016; Corrin, Bennett, & Lockyer, 2013). In fact, Jones, Ramanau, Cross, and Healing (2010) point out that these students expect to be engaged by their environment through simulation, with participatory, interactive, sensory-rich, experimental activities (either physical or virtual), and opportunities for input. They are more oriented to visual media than previous generations and they prefer to learn visually, by doing, rather than by telling or reading. In health education, simulation can assist with student skills (Cook et al., 2013), especially in task training (Wickens, Hutchins, Carolan, & Cumming, 2013). Given that paramedic science is seeing a shift away from face-to- face lectures towards blended learning and distance education (Williams et al., 2011), this presents an opportunity to explore methods to provide simulation task training to distance education students. Literature review Simulation use in health disciplines and paramedic science In health education, there is growing evidence that simulation improves learners’ safety (Abraham, Wade, O’Connell, Desharmaus, & Jacoby, 2011), competence, and skills (Cook et al., 2013), especially when compared to traditional didactic methods and or no simulation training (Cook et al., 2012). Of significant importance to the health profession is airways management (Baker, Weller, Greenland, Riley, & Merry, 2011), where inadequate skill and poor judgment can lead to patient complications and death (Cook, Woodall, & Frerk, 2011). Airways management simulation education and hands-on training builds essential skills (Baker et al., 2011; Kennedy, Cannon, Warner, & Cook, 2014) and changes attitudes and behaviour for all health professionals. This is especially true for trainee paramedics studying high-priority invasive skills such as direct laryngoscopy with foreign body removal in pre-emergency care (Butchart, Tjen, Garg, & Young, 2011), where students require confidence and experience to execute skills correctly (Youngquist et al., 2008). Task trainers are an ideal setting for novice paramedics to train these important hands-on airways skills as they isolate specific tasks to enhance procedural or surgical techniques using three-dimensional (3D) parts Australasian Journal of Educational Technology, 2017, 33(6). 70 of the body or limb to represent a part of the whole (Scalese, Obeso, & Issenberg, 2008). This, in turn, focuses training on the specific task rapidly developing automatic skills (Johnson et al., 2008) by reducing cognitive load (Paas & Sweller, 2014; Plass, Moreno, & Brünken, 2010) which leads to enhanced learning outcomes (Wickens et al., 2013). However, a common discussion point across the health simulation education literature, and task training, is the concept of simulation fidelity or “the degree to which a simulation looks, feels, and acts like a human patient” (Hamstra, Brydges, Hatala, Zendejas, & Cook, 2014, p. 389). This thinking has led to a perceived need for high levels of fidelity (realism) in health education simulations requiring very expensive on-site equipment and intensive clinical residential training (Zendejas, Wang, Brydges, Hamstra, & Cook, 2013). Kuhn (2012) argues that fidelity is imprecise and focus should be on the underlying principles for effective learning including physical resemblance (tactile, visual, auditory, and olfactory features of the simulator) as this can be reduced with minimal or no loss of educational effectiveness, provided there is appropriate correspondence between functional aspects of the simulator and the applied context, for example, contextual cues and spatial arrangement of components. Hamstra et al. (2014) refers to this as “functional task alignment” (p. 387) and suggests that the choice of physical visualisation for maximum training effectiveness depends more on the human functional factors including context, task, stage of learning, learner ability, capabilities, task difficulty, and instructional features, and less on the simulator itself. This aligns with Cook et al. (2013) and the significance between comparative effectiveness of instruction design and simulation, and Norman, Dore, and Grierson (2012) who discusses the minimal relationship between fidelity and transfer of learning. Zendejas et al. (2013) examine this effectiveness by placing a cost on the methods to enhance instructional design features, transfer of learning, learner engagement, and immersion, and relates this to simulator affordability, availability, mobility, and effectiveness. Changing educational delivery for health disciplines and paramedic science In recent years, higher education and the health professions have seen a shift away from the traditional education practice of face-to-face didactic lectures and tutorials to self-directed (Murad, Coto-Yglesias, Varkey, Prokolp, & Murad, 2010) and online distance education (Clark & Mayer, 2016; Cook et al., 2010). This is notably true for paramedic science in Australia (Hou, Rego, & Service, 2013). However, universities in general are lagging behind in innovative pedagogy especially when students are studying from a distance, with most prior work formed around two-dimensional (2D) words and pictures, with less attention given to complex skills learning environments using interactive visualisations, games, and simulations (Ayres, 2015). Brydges et al. (2015) explore these issues from a self-regulated learning approach and recommend learning design should support and help prepare individuals for future learning by assisting learners with self-regulated learning through simulation including obervation (e.g., watching a video), emulation (e.g., imitating the instructional video), measuring self-control (e.g., goal setting), and observational measure of learning transfer. Given the increased impact in running face to face residential schools, cost of high-fidelity task trainers, and the general pedagogy shift towards distance education, new pedagogies and methods of education delivery are required to assist students and improve face to face and distance equality especially in regard to simulation training. To assist with these issues, technologies such as 3D printing (3DP), augmented reality (AR), virtual reality (VR), and mobile bring your own devices (BYOD) are becoming available for use commercially and thus are able to be studied in the distance classroom. Mixed reality for simulation and skills training Mixed reality (MR), a continuum of these innovative technologies, provides a framework to position real and virtual worlds (Milgram & Kishino, 1994). On the continuum (Figure 1), the real world takes the position of the extreme left, whereas VR exists on the extreme right. In the middle, between these two extremes, exists MR. MR is made up of two major components, AR and augmented virtuality (AV). AR is a system consisting mainly of the physical world with some digital components, whereas AV is a virtual world that can interact with the physical. Australasian Journal of Educational Technology, 2017, 33(6). 71 Figure 1. Milgram’s reality-virtuality continuum Use of this continuum can result in the development of new paradigms, tools, techniques, and instrumentation that allow visualisations at different and multiple scales, and the design and implementation of comparative MR pedagogy across multiple disciplines (Magana, 2014). The 2017 NMC Higher Education Horizon Report (Adams Becker et al., 2017) specifically highlights these technologies as key educational technologies leading to innovative and new technology-led pedagogy. For instance, Dalgarno & Lee (2010) outline a set of unique affordances for 3D virtual learning environments that sit on the right of the scale. These include greater opportunities for experiential learning, increased motivation/engagement, improved contextualisation of learning, and the facilitation of tasks that lead to enhanced spatial knowledge representation. Similarly, Cheng and Tsai (2013) provide insight into the use of AR-aided learning, indicating that student’s spatial ability, practical skills, and conceptual understanding is often afforded by this technology, but more work needs to be done on learning experience and learner characteristics. One approach to using simulation to solve these problems would be to use wearable solutions. Bower and Sturman (2015) reviewed the affordances of these technologies and noted that simulation was a useful pedagogical affordance of this technology, with engagement also featuring prominently in their review. However, they also noted participants highlighted cost and processing power as issues with this technology. A possible solution to this problem would be a BYOD solution, which provides the benefits of wearable technology without the cost issue. Parsons (2014) notes this allows students to take ownership of their learning, autonomously acquiring material when they need it, which is useful for distance education. Evidence therefore suggests that AR and VR technologies are mature, but the uptake in education has been hindered by cost and capability. This is changing with the recent wave of low cost immersive 3D VR technology by vendors such as Oculus Rift (www.oculusvr.com) and BYOD mobile VR by Google through Cardboard (vr.google.com/cardboard), powerful interactive 3D visualisation software platforms such as Unity (unity3d.com), and integrated AR plug-ins such as Vuforia by PTC (www.vuforia.com). However, while the latest technology in AR/VR assists with visual and spatial learning there is still an innate lack of physical haptic feedback that one gains through physical media manipulation (Fowler, 2015). 3D printing and the haptic feedback loop 3DP offers a way to bridge the gap between the virtual and the real (Lipson & Kurman, 2013) through physical haptic feedback (Birt & Horvoka, 2014; Birt, Horvoka, & Nelson, 2015). 3DP has seen an explosion in the past 5 years due to low-cost fused deposition modelling (FDM) systems by makers such as MakerBot (www.makerbot.com/). 3DP at its basic level uses an additive manufacturing process to build up objects in layers using plastic polymer. Although the process is slow, 3DP creates direct links between a virtual 3D-based model and the formation of an accurate physical representation with that model (Loy, 2014). This direct linking of object making to computer modelling changes the relationship of the learner to the making of the object and subsequent use. It also enables a haptic feedback loop for students who are not creating but rather using the physical objects on their own (Paas & Sweller, 2014), or within a MR simulation (Cowling, Tanenbaum, Birt, & Tanenbaum, 2016). This reduction in cost, and increased availability leads to increasing the potential of how these technologies can enhance classroom pedagogy, especially in distance paramedicine and airways management (Birt, Moore, & Cowling, 2017; Cowling, Moore, & Birt, 2015). Australasian Journal of Educational Technology, 2017, 33(6). 72 Research aim The aim of this paper is to explore the pedagogical possibilities of a distance education BYOD mobile mixed reality (3DP, AR, VR) task simulation trainer for paramedic airways management training. The context for this study is the skills acquisition and retention, of pre-residential school paramedic students, focusing on direct laryngoscopy with foreign body removal in a Queensland distance paramedic science course. The project aims to provide more hands-on skill practice to students, as well as increasing overall skill acquisition, and retention, answering the research question: • How does the use of mixed reality, especially 3D printing, augmented virtuality, and augmented reality simulation, affect skills development in paramedic science? Background problem Upon completion of their study, students in Australian university Bachelor of Paramedic Science programs are expected to have developed the real-world expertise and skills to work as health professionals in emergency medicine and retrieval (Hou et al., 2013). Graduates of these programs anticipate career options with the government ambulance service, private emergency services, or in industry, providing paramedic services to mine sites and other areas. Graduates employed in this capacity are eligible for membership with Paramedics Australasia and/or the Australian Registry of Emergency Medical Technicians (AREMT). Yet, despite these very practical requirements, the ability to practice practical skills in the programs can be limited for many of the students, who study the program at a distance (Williams et al., 2011). This project therefore stems from a need, identified through course evaluations at an Australian distance paramedic school, for more opportunity for distance students to practice skills (currently, they can only be practiced in the 5-day residential schools), as previously proposed (Cowling et al., 2015). Specifically, key elements of the task simulation learning experience were linked to the Queensland State Ambulance Service (QAS) Airways Management Clinical Practice Procedures (QAS, 2016) and explored the following key skills, classed as high-priority: 1. Direct laryngoscopy (large adult – using Macintosh blade, size 4): “the technique used to achieve optimal visualisation of the glottis for the purpose of oral endotracheal tube insertion or removal of a foreign body” (p. 342), and 2. Magill forceps: “removal of pharyngeal foreign bodies causing airways obstruction in an obtunded patient” (p. 354). Simulation design As the aim of the simulation was to provide distance paramedic students with a cost effective haptic, visual, and auditory feedback mechanic to assist in learning and practising the airway skills in line with the previous simulation fidelity research (Cook et al., 2013; Hamstra et al., 2014; Norman et al., 2012; Zendejas et al., 2013), the project used 3DP representations of the laryngoscope, Macintosh blade (size 4) and Magill forceps, 3D modelled on a 1:1 scale (Figure 2). Tools were split into pieces to fit onto smaller consumer level 3DP equipment. Figure 2. 3D models for printing representing the (A) laryngoscope, (B) Machintosh blade (size 4) and (C) Magill forceps Australasian Journal of Educational Technology, 2017, 33(6). 73 These physical representations allow for a user’s perception to be primed before they carry out the simulation through haptic and visual connection to the tools (Paas & Sweller, 2014). The design of these tools was refined over several experiments over several years, but they maintained a realistic shape, to enable the learner to become familiar with the size and shape of the real tools (Cowling et al., 2015). Markers attached to the tools were similarly refined over time, going from smaller cubic markers in the first pilot study (see images in Birt et al. [2017]) to a larger flat marker design (Figure 3). Figure 3. 3D-printed tools with AR image targets including ColorCross head mount for mobile phone simulation The simulation proposed in the study requires the use of both hands to practise and execute the skills. To allow the learner’s hands to be free throughout the simulation, a smart phone needs to be mounted in front of his/her face. This also allows for the correct view point – looking down the throat of the airway manikin. This design evolved over the experimental timeline, beginning with a hat-mounted device in the first pilot study (Birt et al., 2017), before moving to a more authentic hands-free stereoscopic experience that allowed the user to hold the instruments and visualise the simulation environment in VR, providing depth to the simulation scene (Figure 3). This was achieved using a low-cost ColorCross universal mobile phone VR headset (based off the Google Cardboard design). Version 5 of the Unity game engine development platform (www.unity3d.com) was used to produce the simulated task trainer in the form of a BYOD mobile mixed reality application. The editor can quickly deploy to multiple operating systems and device platforms. This allows for one software version to be produced and, through careful coding, deployed across platforms with minimal changes. The Vuforia version 5 plug-in was selected as this integrates with the Unity platform, is free to use, and is an industry- standard image recognition library. The Google Cardboard Software Development Kit (SDK) was selected as this can work with Vuforia and Unity to provide the VR stereoscopic visualisation, adding depth perception to the simulation assisting with the insertion of the physical tools. To enable the 3D-printed objects to be seen in the simulation, each tool had an AR maker attached for tracking purposes. The AR markers used on the original pilot simulation (Birt et al., 2017) were six-sided, 3cm cubes, which allowed the tools to be tracked from any orientation, as long as one of the sides was visible to the device’s camera. However, as the experiment evolved, these markers were changed to flat, 6cm square markers, which improved tracking and allowed a label for each tool to be added to the marker. This type of tracking utilised Vuforia’s Image Target marker, prefabricated in Unity. The abstract pattern used for the background of the markers was generated by the Augmented Reality Marker Generator by Brovision (www.brosvision.com). The simulation application was available for selected students to download (www.mixedrealityresearch.com/#paramedics) from both the Apple App Store for iPhone 5+ or the Google Play Store for Android Version 4.4+. The emphasis here was on mobile BYOD accessibility (Parsons, 2014), and the affordance of distance access to the finished application. Upon loading the application (see youtu.be/wIfwZFKlSQU – for video simulation), a user is presented with a menu scene, which contains three touch-enabled buttons: (i) Augmented Reality (AR) Tutorial; (ii) Foreign Body Removal; and (iii) AR Markers & Project Information. The first of these buttons loads the tutorial scene. Within this scene, the user is informed through text and audio feedback about how the technology behind the simulation works, and is stepped through a series of programmed tasks using virtually instantiated game objects and colliders designed to familiarise them with Australasian Journal of Educational Technology, 2017, 33(6). 74 using the 3DP, AR-enabled physical tools they have been given. Additionally, this task affords the user a virtual connection (perception) to the simulation and the held instruments through the AR camera feed. Once this connection is established, the AR camera is removed, giving the user a wholly virtual experience but with the physical tools represented by their virtual counterparts (AV). This transition allows the user to maintain a connection to the physical objects as the simulation enters the AV experience, removing the visible world from the user’s perspective, concentrating and focusing the user on the task at hand, and reducing cognitive load (Paas & Sweller, 2014; Plass et al., 2010) and working capacity (Clark & Mayer, 2016) (Figure 4). Figure 4. Simulation training including audio, text and interactivity, assisting mixed reality familiarisation At the end of this tutorial scene, the user is automatically taken into the Direct Laryngoscopy with Foreign Body Removal skills exercise. Alternatively, this scene can also be entered manually through the second touch-enabled button labelled Foreign Body Removal, to allow for quick access and ongoing training. In the development of this section of the app, the focus on the simulation to be functionally task aligned (Hamstra et al., 2014) meant that only the pertinent information relating to the key learning outcomes was included. For the developed simulation this refers to only the key indications from the Australian, Queensland Ambulance Service (QAS) (2016) Airways Management Guide. The simulation starts with the virtual patient presented at the end of the triple airways manoeuvre (Figure 5): “patient’s head is in the appropriate position to align the oral, pharyngeal and laryngeal axes” (QAS, 2016, p. 343). Only the virtual head is presented to the user, reducing device processing load, creating a virtual 3D task trainer for instructional enhancement (Johnson et al., 2008; Wickens et al., 2013). Figure 5. The traditional 2D (left) and mobile 3D simulated (right) illustrations of the patient at the end of the triple airways manoeuvre Australasian Journal of Educational Technology, 2017, 33(6). 75 The simulation follows the QAS (2016) direct laryngoscopy and foreign body removal steps as follows: 1. “Grip laryngoscope handle with the left hand to ensure optimal control and mechanical advantage” (QAS, 2016, p. 343) (Figure 6). The simulation detects the laryngoscope using the AR image recognition software and displays a virtual 3D model of the laryngoscope at 1:1 scale, triggering step 2. Once the scope is detected, the world disappears, presenting a VR view. If at any time the user incorrectly positions the instruments, it presents a red guide. The user has a few seconds to correct the mistake before the simulation restarts showing the real world again (via the camera) so that he/she can re-orientate his/her view of the tools before continuing. Figure 6. Traditional 2D (left) and mobile 3D simulated (right) illustrations of the grip the laryngoscope handle instruction 2. “Place [Macintosh blade] into right side of the patient’s mouth” (QAS, 2016, p. 343) (Figure 7). The simulation detects the correct path of the laryngoscope and if correct triggers step 3. Figure 7. The traditional 2D (left) and mobile 3D simulated (right) illustrations of the Macintosh blade alignment and insertion instruction 3. “Gently sweep the tongue left and position the [Macintosh blade] midline in the mouth” (QAS, 2016, p. 343) (Figure 8). The simulation detects the correct path of the laryngoscope and if correct triggers step 4. Figure 8. The traditional 2D (left) and mobile 3D simulated (right) illustrations of the Macintosh blade sweeping the tongue instruction Australasian Journal of Educational Technology, 2017, 33(6). 76 4. “Move the [Macintosh blade] down the tongue identifying relevant anatomy. Place the [Macintosh blade] top, in the vallecular. Lift upwards and forward at a 45o angle to expose the epiglottis” (QAS, 2016, p. 343) (Figure 9). The simulation detects the correct path and angle of the laryngoscope, and if correct triggers step 5. In the simulation image, the virtual patient head becomes transparent, offering the learner a view of the anatomy and foreign body. For the current simulation, a Grade 1 Cormack-Lehane airways classification (full view of glottis) has been used. Figure 9. The traditional 2D (left) and mobile 3D simulated (right) illustrations of the Macintosh blade lifting the tongue instruction 5. “Grasp the forceps in the right hand. Ensure correct grasp of laryngoscope. Insert forceps into the patient’s mouth (closed). Under direct laryngoscopy, use the Magill forceps to grasp the object. Ensure no pharyngeal or epiglottic structures are grasped” (p. 355) (Figure 10). The simulation detects the Magill forceps and tracks the insertion process. If correct, it triggers step 6. The guide turns red if it detects any touching of the forceps on the patient’s throat or if the laryngoscope is moved. Prior to this point, the simulation does not track the forceps, allowing users to concentrate on the laryngoscope placement, which was an issue in the prior pilot study. Figure 10. The traditional 2D (left) and mobile 3D simulated (right) illustrations of the Magill forceps insertion and grasping of the foreign body blocking the patient’s airways 6. Instrument removal. While the QAS guide does not explicitly define the removal process, the simulation detects the correct removal of the instruments (Figure 11). Figure 11. The mobile 3D simulated illustrations of the Magill forceps and laryngoscope removal process Australasian Journal of Educational Technology, 2017, 33(6). 77 Research method This project was funded by a faculty collaboration between two Australian universities. Approval was obtained from the universities’ ethics committees and all participants consented to be part of the study. The aim of this project was to answer the research question: • How does the use of mixed reality, especially 3D printing, augmented virtuality, and augmented reality simulation, affect skills development in paramedic science? To this end, a design-based research (DBR) methodology (Anderson & Shattuck, 2012) was selected for the project, with an underlying action research mentality (Kemmis, McTaggart, & Nixon, 2014) implemented in the conduct of the research in the classroom. Specifically, the four steps of the DBR methodology were followed through the analysis of the problem and design of the simulation solution (as detailed in the section above), followed by the iterative implementation of that solution into the classroom by a discipline expert practitioner positioned to evaluate the effectiveness of the solution who provided detailed feedback on the design in the first loop pilot study (Birt et al., 2017). This then resulted in loop back to design refinement and further iterative testing. To recruit participants for the study, an invitation was sent out to students studying the paramedics skills course and a list of potential candidates was developed, including location details for each candidate and their mobile digital device ownership. All 2nd year distance paramedic students were invited to participate across the two rounds of the intervention (N = 159). The list was checked and edited to ensure only students with compatible phones (able to install the app) were included in the study, and then a stratified sample of approximately 30 candidates from each round was selected from this list in a blind fashion without the participation of the discipline expert. These students were then shipped a kit consisting of the 3D printed tools, ColorCross headset or hat mount, and instructions for downloading and installing the app to a smartphone. The number of students selected to receive the tools was limited by the availability of 3D printed components and headsets. These students were given instructions on how to use the tools and encouraged to practice for 1 week prior to the residential school scheduled for late in the term. Two loops of the study were conducted. Details of the first loop were presented by Birt et al. (2017), with additional data from the second loop being presented here. Over the two loops, there was a total pool of 159 students, of which 137 students participated. From this pool, 55 students received the simulation for testing prior to coming to the residential school. Demographic data for these students is included in Table 1, and shows male/female ratio (with more females in the class, and more females in the study) as well as device ownership. Table 1 Participant data across the two loops of the design-based research study Loop Enrolled Sent simulation Participated in study Total No simulation Simulation BYOD iOS BYOD Android 1 85 30 64 37 (M:16, F:21) 27 (M:13, F:14) 20 7 2 74 30 73 45 (M:17, F:28) 28 (M:11, F:17) 15 13 n 159 60 137 82 (M:31, F:49) 55 (M:24, F:31) 35 20 Once they arrived at the residential school, all participants in the study (including both those who received the tools beforehand and those who did not) were pre-tested using airway mannequins by qualified on-road paramedics, with advance care paramedic level 2 or above on four separate key performance indicators across the two selected airways skills, before any face-to-face skills training took place. The performance indicators included whether or not they: (1) placed the laryngoscope in the right side before performing lateral sweep, (simulation step 3), (2) elevated the laryngoscope without levering of the teeth (simulation step 4), (3) adequately visualised the obstruction and safely removed it (simulation step 5), and/or (4) removed the laryngoscope without damaging structures (simulation step 6). Australasian Journal of Educational Technology, 2017, 33(6). 78 Participants either passed or failed each key performance indicator and were awarded a score of 1 or 0, respectively. Performances on all tasks were combined and assessed collectively with each participant receiving a score ranging from 0 (failed all four key performance indicators) to 4 (passed all four key performance indicators). Traditional skills training was then conducted as part of the residential school, with all students given access to the 3DP tools and simulation throughout the period to comply with ethics. At the end of the residential school, all students were surveyed on their experience with the tools, and both quantitative (Tables 2 & 3) and qualitative data (Tables 4, 5, & 6) was collected. Results Overall, results from the two rounds of this study show a statistically significant improvement for students who were provided with the paramedic simulation tools ahead of the residential school. Participants exposed to the simulation received higher scores on their overall key performance indicators (M = 2.53; SD = 1.45) than did participants who were not exposed to the simulation (M = 1.96; SD = 1.51). As reported in Table 2, the majority of participants who were not exposed to the simulation failed two or more key performance indicators. Conversely, the majority of participants who were exposed to the simulation passed three or more of the key performance indicators. Table 2 Performance scores in participants who were and were not exposed to the simulation Performance Score No simulation Simulation 0 20 (24.4%) 8 (14.5%) 1 13 (15.9%) 5 (9.1%) 2 20 (24.4%) 13 (23.6%) 3 8 (9.8%) 8 (14.5%) 4 21 (25.6%) 21 (38.2%) n 82 (100.0%) 55 (100.0%) Note. Percentages were calculated with respect to independent simulation groups An independent sample t-test found a significant difference on the participants’ task performance when comparing those who were and were not exposed to a simulation of those tasks beforehand, t(135) = -2.18, p = .031. Looking further into these data, it can also be seen that there was an improvement in the individual skills passing rates for students who completed the simulation, especially in the second round, after the looping nature of the DBR research method had led to refinements in the simulation design. Table 3 shows the individual skills and passing rates for students on each skill, with and without prior simulation training. Table 3 Individual skills and passing rates Skill No simulation Simulation Loop 1 Loop 2 Loop 1 Loop 2 1 15 (41%) 20 (44%) 14 (52%) 17 (61%) 2 11 (30%) 17 (38%) 12 (44%) 16 (57%) 3 21 (57%) 26 (58%) 15 (56%) 24 (86%) 4 23 (62%) 28 (62%) 17 (63%) 24 (86%) Finally, to provide further context and rich data, students were also asked to provide a personal assessment of the simulation tools through a survey conducted at the end of the residential school. Table 4 presents the questions that students were asked in this survey, and Table 5 presents the results of the survey comparison between the first and second loop. Table 6 then presents the results of this survey, split by cohort for the second round of the study, on a 5-point Likert scale, with 1 representing strongly disagree and a score of 5 representing strongly agree. It must be noted that specific simulation group data is not available for the first loop and was added to the second loop of the study, as were questions 11 and 12. Australasian Journal of Educational Technology, 2017, 33(6). 79 Table 4 Survey questions Q1 I am comfortable with technology in learning Q2 I am comfortable with smartphones/tablets/post-PC devices Q3 I consider myself "tech-savvy" Q4 I found the 3D distance skill development tools intuitive and user friendly Q5 The 3D distance skill development tools improved my skill development Q6 The 3D distance skill development tools complemented the practical sessions well Q7 I valued the addition of the 3D distance skill development tools to the course Q8 I found the use of multiple forms of media (3D plastic items plus app) effective Q9 I valued the real-world aspects of the 3D-printed plastic tools Q10 I valued the simulation aspects of the app Q11 I found watching the video of the simulation useful Q12 Using the [simulation] let me chat about airways management to others at residential school Q13 I would like to see more courses using multiple forms of media (i.e., 3DP tools and phone app) Table 5 Student personal assessment survey results grouped by loop Question Loop 1 Loop 2 1 2 3 4 5 n 1 2 3 4 5 n 1 2% 2% 15% 47% 34% 53 0% 4% 12% 40% 44% 52 2 0% 4% 15% 38% 43% 53 0% 2% 10% 33% 56% 52 3 2% 6% 34% 36% 23% 53 0% 4% 41% 41% 14% 51 4 2% 17% 64% 13% 4% 53 0% 0% 44% 51% 5% 39 5 2% 15% 68% 9% 6% 53 0% 0% 47% 50% 3% 38 6 4% 13% 67% 10% 6% 52 0% 0% 39% 55% 5% 38 7 2% 8% 56% 21% 13% 52 0% 0% 33% 59% 8% 39 8 2% 8% 69% 15% 6% 52 0% 3% 37% 53% 8% 38 9 0% 2% 71% 17% 10% 52 0% 0% 29% 54% 17% 41 10 0% 6% 70% 13% 11% 53 0% 0% 37% 55% 8% 38 11 - - - - - 0 0% 3% 23% 55% 20% 40 12 - - - - - 0 2% 2% 39% 49% 7% 41 13 0% 4% 50% 21% 25% 52 0% 0% 4% 46% 50% 46 Table 6 Student personal assessment survey results by group (no simulation and simulation) – Loop 2 Question Loop 2 - No simulation training Loop 2 – Simulation training 1 2 3 4 5 n 1 2 3 4 5 n 1 0% 3% 9% 34% 53% 32 0% 5% 15% 50% 30% 20 2 0% 3% 6% 28% 63% 32 0% 0% 15% 40% 45% 20 3 0% 3% 39% 48% 10% 31 0% 5% 45% 30% 20% 20 4 0% 0% 58% 37% 5% 19 0% 0% 30% 65% 5% 20 5 0% 0% 61% 39% 0% 18 0% 0% 35% 60% 5% 20 6 0% 0% 56% 39% 6% 18 0% 0% 25% 70% 5% 20 7 0% 0% 47% 47% 5% 19 0% 0% 20% 70% 10% 20 8 0% 0% 56% 44% 0% 18 0% 5% 20% 60% 15% 20 9 0% 0% 48% 52% 0% 21 0% 0% 10% 55% 35% 20 10 0% 0% 61% 39% 0% 18 0% 0% 15% 70% 15% 20 11 0% 5% 40% 40% 15% 20 0% 0% 5% 70% 25% 20 12 5% 5% 57% 29% 5% 21 0% 0% 20% 70% 10% 20 13 0% 0% 8% 46% 46% 26 0% 0% 0% 45% 55% 20 Australasian Journal of Educational Technology, 2017, 33(6). 80 These results show a general acceptance of the tools, especially by those students who spent more time with them in the pre-residential school simulation training, with scores for question 6, “The 3D distance skill development tools complemented the practical sessions well”, and question 7, “I valued the addition of the 3D distance skill development tools to the course” being especially favourable. The responses to question 13, “I would like to see more courses using multiple forms of media (i.e., 3D-printed tools and smartphone simulation applications)” are also promising for future work. Discussion The results highlighted (Table 2) indicate that participants who engaged in the simulation did perform significantly better on their assessment tasks (M = 2.53; SD = 1.45) than did participants who were not exposed to the simulation (M = 1.96; SD = 1.51), when tested prior to the residential school. This was true for both the overall simulation and the specific tasks students were asked to complete. As shown (Table 3) in all cases, those students who received the simulation out-performed those who did not receive the simulation prior to the residential school. This result was significant, as highlighted by an independent samples t-test, which found a significant difference in the participants’ task performance when comparing those who were and were not exposed to a simulation of those tasks beforehand, t(135) = -2.18, p = .031. The results would appear to support the assertion that this type of task trainer, which focuses training on the specific task, can help students to more rapidly develop automatic skills (Johnson et al., 2008) as evidenced by enhanced learning outcomes, and as supported in previous studies by Wickens et al. (2013). This also supports previous work on using simulations to produce enhanced learning outcomes, as outlined in Abraham et al. (2011), Baker et al. (2011), Johnson et al. (2008), and Wickens et al. (2013). Additionally, this would support the assumptions as presented by Dalgarno and Lee (2010) that 3D virtual learning environments allow for experiential learning, improving contextualisation and facilitation of tasks that lead to enhanced spatial knowledge representation, and Cheng and Tsai (2013) who show that AR-aided learning improves conceptual understanding and is afforded by the underlying technology. This positive enhancement to learning was also reflected in the student reactions to the technology, with the results (Table 6) showing 100% of students who used the simulation indicating that they agreed or strongly agreed that this type of pedagogical approach, should be used in coursework in the future. This was highlighted by Bower and Sturman (2015) who showed that mobile simulation solutions are a useful pedagogical technology. Analysing the participant feedback in detail, showed that 75% of students indicated via the survey that they “found the use of multiple forms of media (3DP and app) effective”. This is supportive of not only work in paramedics, but the general work in mixed reality that indicates combining of multiple forms of media can be helpful to students’ learning (Ayres, 2015). With regards to simulation design, many of the difficulties (Table 5) highlighted in the first round of the study (Birt et al., 2017) related to simulation dissonance and ease of use. These difficulties were partially solved (Table 6) with the stereoscopic heads-up display mount and transition from AR to VR. Marker pickup was improved by increasing the size and position of the markers. It is possible that these changes resulted in the improved results from loop 1 to loop 2 (Tables 3, 5, & 6), but more research work will need to be done to investigate whether this was the case. Regardless of this work, the results of this DBR process would indicate that researchers in the future should look to use stereoscopic mounts and large markers to ensure optimum performance of their simulation. Focusing on mixed reality learning in general, we suggest that anyone who wants to embark on this pedagogy must firstly understand their problem domain and focus work on significant problems. The time and cost involved in developing the simulations means that the tasks need to be primary and difficult to solve using traditional forms of distance education pedagogy such as 2D videos and animations. The main issue we had in this study was not being able to observe the students using the simulation at a distance and the usability problems they may have experienced in real-time. Future work should continue in this area, although further improvement in the simulation is required. Whilst results for students’ pre-residential school were improved by the simulation, they also show that the simulation by itself is not enough to give the hands-on training required, with only 38.2% of students able to pass all four skills based on the simulation practice alone. Discussions with the discipline expert indicate that there is still more work that could be done to improve the look and feel of the app. This is in line with Australasian Journal of Educational Technology, 2017, 33(6). 81 results reported from studies in Cook et al. (2013), which indicated that simulation design and realism was an important factor in acceptance. Students and the discipline expert indicated that future investigations could look to simulate more of the physical aspects of the real skill training, with physical components representing the airways dummy including the feeling of the throat as well as the tools used to aid the students haptic feedback. Finally, the simulation has only focused on the indications from the Australian, QAS (2016) Airways Management Guide and a Grade 1 Cormack-Lehane airways classification (p. 345) for the two focused skills. Future work will need to explore the complications and contraindications of these skills using serious game mechanics together with artificial intelligence and data analytics techniques to track and assist learner performance and skills development. Conclusion This study looked at the use of mixed reality to provide additional skills training in a distance paramedics course, exposing distance students in paramedic science to mobile BYOD task trainers using a combination of various modes of training, in line with the literature. Specifically, students were provided with a mixed reality solution incorporating 3D printed tools and a virtual and augmented reality simulation app for laryngoscopy and foreign body removal, for use on their own mobile phones. Learner evaluation was conducted to examine how students’ skill development was improved by receiving this kit prior to their traditional residential school training. Results showed a statistically significant improvement for students who received the tools before residential school, both across the skill set and within individual skills, showing promise for students to more rapidly develop automatic skills as evidenced by enhanced learning outcomes, and acceptance of this use of multiple forms of media by students. References Abraham, J., Wade, D. M., O’Connell, K. A., Desharnais, S., & Jacoby, R. (2011). The use of simulation training in teaching health care quality and safety: An annotated bibliography. American Journal of Medical Quality, 26(3), 229-238. https://doi.org/10.1177/1062860610384716 Adams Becker, S., Cummins, M., Davis, A., Freeman, A., Hall Giesinger, C., & Ananthanarayanan, V. (2017). NMC Horizon Report: 2017 Higher Education Edition. Austin, TX: New Media Consortium. Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16-25. https://doi.org/10.3102/0013189X11428813 Ayres, P. (2015). State-of-the-art research into multimedia learning: A commentary on Mayer’s handbook of multimedia learning. Applied Cognitive Psychology, 29(4), 631-636. https://doi.org/10.1002/acp.3142 Baker, P., Weller, J., Greenland, K., Riley, R., & Merry, A. (2011). Education in airway management. Anaesthesia, 66(s2), 101-111. https://doi.org/10.1111/j.1365-2044.2011.06939.x Birt, J., & Horvoka, D. (2014). Effect of mixed media visualization on learner perceptions and outcomes. Proceedings of the 25th Australiasian Conference on Information Systems, Auckland, 1-10. https://doi.org/10292/8097 Birt, J., Horvoka, D., & Nelson, J. (2015). Interdisciplinary translation of comparative visualisation. Proceedings of the 26th Australiasian Conference on Information Systems, Adelaide, 1-10. https://doi.org/10765/117415 Birt, J., Moore, E., & Cowling, M. (2017, April). Piloting mobile mixed reality simulation in paramedic distance education. Proceedings of the 5th IEEE Conference on Serious Games and Applications for Health, Perth, 1-8. https://doi.org/10.1109/SeGAH.2017.7939270. Bower, M., & Sturman, D. (2015). What are the educational affordances of wearable technologies? Computers & Education, 88, 343-353. https://doi.org/10.1016/j.compedu.2015.07.013 Brydges, R., Manzone, J., Shanks, D., Hatala, R., Hamstra, S. J., Zendejas, B., & Cook, D. A. (2015). Self-regulated learning in simulation-based training: A systematic review and meta-analysis. Medical Education, 49(4), 368-378. https://doi.org/10.1111/medu.12649 Butchart, A. G., Tjen, C., Garg, A., & Young, P. (2011). Paramedic laryngoscopy in the simulated difficult airway: Comparison of the Venner AP Advance and GlideScope Ranger video laryngoscopes. Academic Emergency Medicine, 18(7), 692-698. https://doi.org/10.1111/j.1553- 2712.2011.01115.x Cheng, K.-H., & Tsai, C.-C. (2013). Affordances of augmented reality in science learning: Suggestions for future research. Journal of Science Education and Technology, 22(4), 449-462. https://doi.org/10.1007/s10956-012-9405-9 Australasian Journal of Educational Technology, 2017, 33(6). 82 Clark, R. C., & Mayer, R. E. (2016). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning (4th ed.). New Jersey, NJ: Wiley. https://doi.org/10.1002/9781119239086 Cook, D., Brydges, R., Hamstra, S., Zendejas, B., Szostek, J., Wang, A., … & Hatala, R. (2012). Comparative effectiveness of technology-enhanced simulation versus other instructional methods: A systematic review and meta-analysis. Simulation in Healthcare, 7(5), 308-320. https://doi.org/10.1097/SIH.0b013e3182614f95 Cook, D. A., Hamstra, S. J., Brydges, R., Zendejas, B., Szostek, J. H., Wang, A. T., … Hatala, R. (2013). Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis. Medical Teacher, 35(1), 867-898. https://doi.org/10.3109/0142159X.2012.714886 Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2010). Instructional design variations in Internet-based learning for health professions education: A systematic review and meta-analysis. Academic Medicine, 85(5), 909-922. https://doi.org/10.1097/ACM.0b013e3181d6c319 Cook, T., Woodall, N., & Frerk, C. (2011). Major complications of airway management in the UK: Results of the Fourth National Audit Project of the Royal College of Anaesthetists and the Difficult Airway Society. Part 1: Anaesthesia. British Journal of Anaesthesia, 106(5), 617-631. https://doi.org/10.1093/bja/aer058 Corrin, L., Bennett, S., & Lockyer, L. (2013). Digital natives: Exploring the diversity of young people’s experience with technology. In R. Huang, Kinshuk, & J. M. Spector (Eds.) Reshaping learning: Frontiers of learning technology in a global context (pp. 113-138). Berlin: Springer. https://doi.org/10.1007/978-3-642-32301-0_5 Cowling, M., Moore, E., & Birt, J. (2015). Augmenting distance education skills development in paramedic science through mixed media visualisation. Proceedings of the 19th Annual Euromedia Conference. Lisbon, 113-117. Cowling, M., Tanenbaum, J., Birt, J., & Tanenbaum, K. (2016). Augmenting reality for augmented reality. Interactions, 24(1), 42-45. https://doi.org/10.1145/3019008 Dalgarno, B., & Lee, M. J. W. (2010). What are the learning affordances of 3-D virtual environments? British Journal of Educational Technology, 40(6), 10-32. https://doi.org/10.1111/j.1467- 8535.2009.01038.x Fowler, C. (2015). Virtual reality and learning: Where is the pedagogy? British Journal of Educational Technology, 46(2), 412-422. https://doi.org/10.1111/bjet.12135 Hamstra, S. J., Brydges, R., Hatala, R., Zendejas, B., & Cook, D. A. (2014). Reconsidering fidelity in simulation-based training. Academic Medicine, 89(3), 387-392. https://doi.org/10.1097/ACM.0000000000000130 Hou, X.-Y., Rego, J., & Service, M. (2013). Paramedic education opportunities and challenges in Australia. Emergency Medicine Australasia, 25(2), 114-119. https://doi.org/10.1111/1742-6723.12034 Johnson, K. B., Syroid, N. D., Drews, F. A., Ogden, L. L., Strayer, D. L., Pace, N. L., … Westenskow, D. R. (2008). Part task and variable priority training in first-year anesthesia resident education: A combined didactic and simulation-based approach to improve management of adverse airway and respiratory events. Anesthesiology, 108(5), 831-840. https://doi.org/10.1097/ALN.0b013e31816bbd54 Jones, C., Ramanau, R., Cross, S., & Healing, G. (2010). Net generation or digital natives: Is there a distinct new generation entering university? Computers & Education, 54(3), 722-732. https://doi.org/10.1016/j.compedu.2009.09.022 Kemmis, S., McTaggart, R., & Nixon, R. (2014). The action research planner: Doing critical participatory action research. Singapore: Springer. https://doi.org/10.1007/978-981-4560-67-2 Kennedy, C. C., Cannon, E. K., Warner, D. O., & Cook, D. A. (2014). Advanced airway management simulation training in medical education: A systematic review and meta-analysis. Critical Care Medicine, 42(1), 169-178. https://doi.org/10.1097/CCM.0b013e31829a721f Kuhn, T. S. (2012). The structure of scientific revolutions (4th ed.). Chicago, IL: The University of Chicago Press. https://doi.org/10.7208/chicago/9780226458144.001.0001 Lipson, H., & Kurman, M. (2013). Fabricated: The new world of 3D printing. NJ:Wiley. Loy, J. (2014). eLearning and eMaking: 3D printing blurring the digital and the physical. Education Sciences, 4(1), 108-121. https://doi.org/10.3390/educsci4010108 Magana, A. J. (2014). Learning strategies and multimedia techniques for scaffolding size and scale cognition. Computers & Education, 72, 367-377. https://doi.org/10.1016/j.compedu.2013.11.012 Australasian Journal of Educational Technology, 2017, 33(6). 83 Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. The Institute of Electronics, Information and Communication Engineers Transactions on Information and Systems Transactions on Information and Systems, 77(12), 1321-1329. Murad, M. H., Coto-Yglesias, F., Varkey, P., Prokop, L. J., & Murad, A. L. (2010). The effectiveness of self-directed learning in health professions education: A systematic review. Medical Education, 44(11), 1057-1068. https://doi.org/10.1111/j.1365-2923.2010.03750.x Norman, G., Dore, K., & Grierson, L. (2012). The minimal relationship between simulation fidelity and transfer of learning. Medical Education, 46(7), 636-647. https://doi.org/10.1111/j.1365- 2923.2012.04243.x Paas, F., & Sweller, J. (2014). Implications of cognitive load theory for multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 27-42). New York, NY: Cambridge University Press. https://doi.org/10.1017/cbo9781139547369.004 Parsons, D. (2014). The future of mobile learning and implications for education and training. In Mohamed, A., & Avgoustous, T. (Eds.) Increasing Access through mobile learning (pp. 217-229). Vancouver, BC: Commonwealth of Learning and Athabasca University Press. https://doi.org/11599/558 Plass, J. L., Moreno, R., & Brünken, R. (Eds.). (2010). Cognitive load theory. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511844744 Queensland Ambulance Service, CQPS Unit. (2016). Clinical practice procedures: Airway management. Retrieved from https://www.ambulance.qld.gov.au/CPPtable.html Scalese, R. J., Obeso, V. T., & Issenberg, S. B. (2008). Simulation technology for skills training and competency assessment in medical education. Journal of General Internal Medicine, 23(1), 46-49. https://doi.org/10.1007/s11606-007-0283-4 Wickens, C. D., Hutchins, S., Carolan, T., & Cumming, J. (2013). Effectiveness of task training and increasing-difficulty training strategies: A meta-analysis approach. Human Factors: The Journal of the Human Factors and Ergonomics Society, 55(2), 461-470. https://doi.org/10.1177/0018720812451994 Williams, B., Boyle, M., Molloy, A., Brightwell, R., Munro, G., & Brown, T. (2011). Undergraduate paramedic students’ attitudes to e-learning: Findings from five university programs. Research in Learning Technology, 19(2), 89-100. https://doi.org/10.1080/21567069.2011.586679 Youngquist, S. T., Henderson, D. P., GauscheHill, M., Goodrich, S. M., Poore, P. D., & Lewis, R. J. (2008). Paramedic self-efficacy and skill retention in pediatric airway management. Academic Emergency Medicine, 15(12), 1295-1303. https://doi.org/10.1111/j.1553-2712.2008.00262.x Zendejas, B., Wang, A. T., Brydges, R., Hamstra, S. J., & Cook, D. A. (2013). Cost: The missing outcome in simulation-based medical education research: A systematic review. Surgery, 153(2), 160- 176. https://doi.org/10.1016/j.surg.2012.06.025 Corresponding author: James Birt, jbirt@bond.edu.au Australasian Journal of Educational Technology © 2017. Please cite as: Birt, J., Moore, E., & Cowling, M. (2017). Improving paramedic distance education through mobile mixed reality simulation. Australasian Journal of Educational Technology, 33(6), 69- 83. https://doi.org/10.14742/ajet.3596 mailto:jbirt@bond.edu.au https://doi.org/10.14742/ajet.3596 Introduction Literature review Simulation use in health disciplines and paramedic science Changing educational delivery for health disciplines and paramedic science Mixed reality for simulation and skills training 3D printing and the haptic feedback loop Research aim Background problem Simulation design Research method Results Discussion Conclusion References