Australasian Journal of Educational Technology, 2021, 37(4). 68 Tele-mentoring using augmented reality technology in healthcare: A systematic review Dung Trung Bui, Tony Barnett, Ha Hoang Centre for Rural Health, School of Health Sciences, College of Health and Medicine, University of Tasmania Winyu Chinthammit Human Interface Technology Laboratory, School of Information and Communications Technology, University of Tasmania This systematic review aimed to identify how tele-mentoring systems that incorporate augmented reality (AR) technology are being used in healthcare environments. A total of 12 electronic bibliographic databases were searched using the terms “augmented reality”, “tele- mentoring” and “health”. The PRISMA checklist was used as a guide for reporting. The mixed method appraisal tool was used to assess the quality of the included experiments. The data were then analysed using a concept-centric approach and categorised primarily with regards to system performance and task performance measures. A total of 11 randomised controlled trials and 14 non-randomised designs were included for review. Both mentees and mentors assessed the system and task performance according to 25 categories. The feedback of mentees using AR tele-mentoring systems was generally positive. The majority of experiments revealed that the AR system was an effective tele-mentoring device overall and resulted in the effective performance of a procedure. Benefits included improvements in trainees’ confidence, task completion time and reductions in task errors and shifts in focus. However, the systems had limitations, including heaviness of the equipment, inconvenience, discomfort and distraction of wearing devices, limited battery life, the latency of video and audio signals and limited field of view. Implications for practice or policy: • Health practitioners can apply AR technology to receive and follow real-time annotated instructions verbally and visually from remote experts. • Technical developers may consider improving AR devices in terms of lighter weight, larger field of view, more ergonomic design, more stable network connection and longer battery life. • Further AR-related experiments may need to explore AR tele-mentoring systems’ utility across healthcare environments with larger samples, real patient populations in remote settings, cost-benefit analysis and impacts on short- and long-term patient outcomes. Keywords: augmented reality (AR), mentoring, tele-mentoring, health professional education, systematic review Introduction Tele-mentoring is a method in which a mentor interactively guides a mentee at a different geographic location using a technological communication device (Antoniou et al., 2012; Erridge et al., 2019). According to Agarwal et al. (2007), during a healthcare procedure, tele-mentoring provides instruction from a remote expert to a local, less-experienced practitioner in real time at the site of treatment. With this approach, community practitioners can benefit from remote expert advice to complete unfamiliar clinical techniques. Moehr et al. (2006) reported that tele-mentoring provided rural and remote residents with accessibility to primary and specialised health services, improved continuity of care and increased availability of patient information, resulting in decreased frequency of patient visits to healthcare specialists. Tele-mentoring has been identified as a promising method to improve patient outcomes in a remote environment, such as prolonged field care and long-distance evacuation. In particular, patients with complex trauma injuries cannot always be transported to a hospital to receive care quickly enough for the successful treatment of such injuries (Kotwal et al., 2016). Delays in transportation can reduce the Australasian Journal of Educational Technology, 2021, 37(4). 69 likelihood of patient survival. Surgical tele-mentoring may bring the specialised expertise of a surgeon onto the battlefield, contributing to a reduction in combat deaths (Andersen et al., 2017). Studies have found that tele-mentoring decreases healthcare costs for rural patients by avoiding unnecessary referrals and transfer to tertiary care centres (Doze et al., 1999; Panait et al., 2006; Schlachta et al., 2010; Sebajang et al., 2006). Aarnio et al. (2000) also indicated that tele-mentoring may reduce discomfort and further injury for the patient, which may also reduce the burden on the patient’s caregiver. Limitations of tele-mentoring systems have been identified in the literature. For example, an expert could draw visual annotations overlaid on an image of the operating field, but the local surgeon could view these annotations only by looking at a nearby computer monitor (Budrionis et al., 2013). Bogen et al. (2014) indicated that the monitors distracted the surgeon and reduced the sense of co-presence as well as resulting in delays or errors in medical treatment. Antoniou et al. (2012) reviewed 96 laparoscopic tele-mentored procedures and reported that the expense of tele-mentoring applications, their installation, maintenance and broadband services may outweigh the cost savings, especially if the number of patients is low. They also reported difficulties with video and audio latency with low transfer rates and inadequate guidance from the mentor. Other pressures can come from restrictions on residents’ working hours, the increase of patient load of attending surgeons and the increase of operating room costs (Issenberg & Scalese, 2008). Augmented reality (AR) is defined as a form of immersive experience in which the real world is enhanced by computer-generated three-dimensional content, which is tied to specific locations and/or activities (Azuma, 1997). A major strength of AR compared to visualisation on an external monitor is that it is able to show virtual objects in place (Sielhorst et al., 2004). Users do not need to look away from the place of interest to the screen. Another strength of AR is the perception of the real scene (Barsom et al., 2016). The system overlays only those pieces of information that are necessary but does not immerse the user in a totally virtual environment. It supports the intuitive integration of information, which is provided by the visualisation of objects hidden under a real surface in order to give the impression of a view inside (Sielhorst et al., 2004). An AR tele-mentoring system is divided into a local subsystem and a remote subsystem. The local subsystem refers to a station or setting designed for trainees who receive instructions, while the remote subsystem is a station operated by trainers who provide the instructions from another location. Figure 1 illustrates how the two stations communicate and interact with each other. Based on AR technology, the local mentee can communicate verbally to the remote mentor and view the operating field with the mentor providing motions or annotations overlayed on the visual field (Dey et al., 2018; Zhu et al., 2014). The AR tele-mentoring systems provide the local (mentees) with images of annotations, hand gestures, tool movements or positions, textual labels, icons of working instruments or instructional visuals created by the remote mentor. These annotations could be predefined or directly drawn in real time. Figure 1. Communication and interaction mechanism in an AR tele-mentoring system Australasian Journal of Educational Technology, 2021, 37(4). 70 There is limited systemic knowledge about the application of AR technology in tele-mentoring systems in healthcare environments. Identification of its benefits and issues that need further improvement may provide research-based evidence to help with appropriate policy directives and support healthcare system to better apply AR technology into practice. Thus, this systematic review aimed to identify how tele- mentoring systems that incorporate AR technology are being used in healthcare environment, including their benefits and limitations. Methods Search strategy A total of 12 electronic bibliographic databases in the fields of medicine, education and technology were searched using the keywords “augmented reality”, “tele-mentoring” and “health” and its medical subject headings terms. The databases consisted of ACM Transactions on Computer-Human Interaction, CINAHL Complete, Cochrane Library, Embase (via Ovid), ERIC, IEEE Xplore Digital Library, MEDLINE Complete, PsycINFO, PubMed, Science Direct, Scopus and Web of Science (all databases). We screened titles and abstracts of studies retrieved using the search strategy independently to identify studies that potentially met the inclusion criteria outlined in Table 1. After that, we retrieved and assessed individually the full texts of these potentially eligible studies for eligibility; we resolved any disagreements through discussion. Table 1 Inclusion and exclusion criteria Refinements Inclusion criteria Exclusion criteria Year Published from January 1990 to June 2019 Published before 1990 Source formats Research papers published in journals and conference proceedings Book, book chapter, expert opinions, commentary, editorials Language English Non-English Designs Quantitative, experimental (randomised controlled trials, non-randomised studies) Descriptive or observational studies. qualitative studies Technologies The system studied must: - Use AR or technologies described as the overlay of virtual objects on a real environment, and - Include real-time tele-mentoring activities - Virtual reality or technologies described to immerse users in a totally virtual environment; or - Did not include a component of real- time tele-mentoring Participants Healthcare professionals, patients, health students, or non-medical volunteers without limitations of gender or age Used robotics to perform tasks in experiments Intervention - Skills applied must be medical skills or supportive of the treatment or rehabilitation of a disease - Participants could practice the skills on real patients or people acting as patients or on mannequins or simulators - Skills applied are not in the healthcare sector or - Telecommunication only without any task performance Outcomes Must have at least one outcome related to the system or task performance Description of the development of the system only or with no evaluation of outcomes Search outcomes This review was conducted and reported in accordance with the preferred reporting items for systematic reviews and meta-analyses (PRISMA; Moher et al., 2009). The PRISMA was selected as it was designed for the systematic review of trials rather than other types of study designs (such as observational studies for which the meta‐analysis of observational studies in epidemiology checklist (Van Zuuren & Fedorowicz, 2016) may have been more appropriate). It assisted us in ensuring transparency and consistency of approach by following a fit-for-purpose evidence-based checklist. Australasian Journal of Educational Technology, 2021, 37(4). 71 Figure 2. Search strategy on tele-mentoring using AR in medicine Data extraction We developed a data extraction tool using standardised headings, and all relevant data entered on a Microsoft Excel spreadsheet (see the Appendix). The data were analysed according to the concept-centric approach (Webster & Watson, 2002), which makes it easier to compare and identify differences in study outcomes across reviewed articles. The sources were categorised primarily with regards to system performance and task performance measures (Sommerauer & Müller, 2018). Methodological quality assessment The mixed method appraisal tool (MMAT; Hong et al., 2018) is a critical appraisal tool designed for the appraisal stage of systematic reviews, for example, reviews include qualitative, quantitative and mixed methods studies. The MMAT was used to assess the methodological quality of the studies reviewed. The MMAT was chosen as it can be used to appraise different research designs, including quantitative randomised controlled trials and quantitative non-randomised. The results of quality assessment of studies using the MMAT are presented in Figure 3. Australasian Journal of Educational Technology, 2021, 37(4). 72 Figure 3. Summary of quality appraisal of studies using the MMAT Note. Y: Yes (if it met quality criterion); N: No (if it did not meet quality criterion); CT: Cannot tell (if it did not mention relevant information). *R refers to the study code listed in the Appendix. S 1 . A re t h e re c le a r re s e a rc h q u e s ti o n s ? S 2 . D o t h e c o ll e c te d d a ta a ll o w t o a d d re s s th e r e s e a rc h q u e s ti o n s ? 1 . Is r a n d o m is a ti o n a p p ro p ri a te ly p e rf o rm e d ? 2 . A re t h e g ro u p s c o m p a ra b le a t b a s e li n e ? 3 . A re t h e re c o m p le te o u tc o m e d a ta ? 4 . A re o u tc o m e a s s e s s o rs b li n d e d t o t h e in te rv e n ti o n p ro v id e d ? 5 . D id t h e p a rt ic ip a n ts a d h e re t o t h e a s s ig n e d in te rv e n ti o n ? 1 . A re t h e p a rt ic ip a n ts r e p re s e n ta ti v e o f th e ta rg e t p o p u la ti o n ? 2 . A re m e a s u re m e n ts a p p ro p ri a te r e g a rd in g b o th t h e o u tc o m e a n d i n te rv e n ti o n ? 3 . A re t h e re c o m p le te o u tc o m e d a ta ? 4 . A re t h e c o n fo u n d e rs a c c o u n te d f o r in t h e d e s ig n a n d a n a ly s is ? 5 . D u ri n g t h e s tu d y p e ri o d , is t h e i n te rv e n ti o n a d m in is te re d a s i n te n d e d ? Y Y CT Y Y N Y Y Y N Y Y CT Y Y Y N Y Y CT Y Y Y CT Y Y N Y Y Y N Y Y CT Y Y Y N Y Y CT Y Y Y N Y Y CT Y Y Y CT Y Y N Y Y Y N Y Y CT Y Y Y N Y Y CT Y Y Y N Y Y CT Y Y Y N Y Y CT Y Y Y N Y Y CT Y Y Y N Y Y CT Y Y Y CT Y Y N Y Y Y N Y Y CT Y Y Y N Y Y CT Y Y Y CT Y Y N N Y Y N Y Y CT Y Y Y CT Y Y N Y Y Y CT Y Y N Y Y Y CT Y Y N Y Y Y CT Y Y N Y Y Y CT Y Y N Y Y Y CT Y Y N Y n 25 25 0 11 11 0 10 0 14 14 0 14 % 100 100 0 100 100 0 91 0 100 100 0 100 n 0 0 0 0 0 11 1 14 0 0 0 0 % 0 0 0 0 0 100 9 100 0 0 0 0 n 0 0 11 0 0 0 0 0 0 0 14 0 % 0 0 100 0 0 0 0 0 0 0 100 0 Assessm ent criteria References (codes*) Wang et al., 2017 (R18) Yoshinaga et al., 2011 Yoshinaga et al., 2011 Masuda et al., 2011 (R6) All study Quantitative random ised controlled trials Quantitative non-random ised Andersen et al., 2017 (R1) Carbone et al., 2018 (R2.1) Carbone et al., 2018 (R2.2) Chintham m it et al., 2014 (R3) Davis et al., 2016 (R4) Hanna et al., 2018 (R5) Wagner et al., 1999 (R17) Mather et al., 2017 (R7.1) Mather et al., 2017 (R7.2) Mitsuno et al., 2019 (R8) Oyam a et al., 2010 (R9) Ponce et al., 2014 (R10) Ponce et al., 2014 (R11) Rio et al., 2018 (R12) Rojas-Munoz et al., 2018 (R13) Shenai et al., 2011 (R14) Suenaga et al., 2001 (R15) Vera et al., 2014 (R16) Andersen et al., 2016 (R20) Andersen et al., 2019 (R21) Lin et al., 2018 (R22) Total Y N CT Australasian Journal of Educational Technology, 2021, 37(4). 73 Findings The results of the literature search and screening process are presented in Figure 2. A total of 22 studies were included in the review. Of these, three included two different experiments (Carbone et al., 2018; Mather et al., 2017; Yoshinaga et al., 2011). The experiments used the same AR-based systems and had some similar outcome measures but different objectives, target population, intervention setting and medical professional skills. Because these six experiments met the inclusion criteria, they were all included in the next stages of the review. Thus, the findings from a total of 25 experiments extracted from 22 studies were synthesised and analysed. Overview Of the 25 experiments, 11 were randomised controlled trials, and 14 used non-randomised techniques. All AR-based tele-mentoring systems had been developed and trialled in developed countries. Whilst three experiments did not report sample size, the average number of participants for the remainder was 15. Sample size was determined by a statistical power calculation in only one experiment (Vera et al., 2014). The highest number of participants reported was 60 students in the experiment conducted by Yoshinaga et al. (2011). In 19 (76%) of the experiments, a new AR prototype was developed, while the remaining (6 or 24%) used or modified AR devices developed by a third party. Microsoft HoloLens version 1 (Microsoft, 2019) was used in five experiments (Hanna et al., 2018; Lin et al., 2018; Mitsuno et al., 2019; Rojas-Munoz et al., 2018; Wang et al., 2017), and Google Glass (Google, n.d.) was used in one experiment (Ponce, Menendez et al., 2014). Different technological devices had been trialled to display the physical movements or annotations of the remote mentors in the local mentees’ field of view. A total of 11 out of 25 experiments used head-mounted display (HMD) devices followed by built-in screen-based systems (10) and tablet-based systems (3). Most of the built-in screen designed systems (80%) were used in experiments related to ultrasound or laparoscopy. Findings about the system and task performance from the studies reviewed are summarised below. AR tele-mentoring system performance The users’ perspectives on the system performance were assessed in 16 out of 25 experiments (Table 2). The 13 categories were classified into the topics of overall system and video or images or audio. Table 2 Users’ perspectives on system performance Feature Experiments* Overall system Performance of the system R2.2, R12, R13, R14, R17, R18, R21 Ease of use R7.1, R7.2, R10, R13, R18 Composite accuracy R4, R12, R17 Platform invasiveness R2.1, R2.2 Headset distraction R2.1, R2.2 Webcam distraction R2.1, R2.2 System reliability R10 Concerns about weight, ocular sickness, etc. R9, R13, R18, R22 Video or images or audio Video quality R2.1, R2.2, R4, R11, R14, R16, R17 Image quality R2.1, R2.2, R10, R18 Field of view R11, R12, R13, R16, R18 Lag in motion R10 Audio quality R2.1, R2.2, R4, R11, R14, R16, R17 *R refers to the study code listed in the Appendix. Australasian Journal of Educational Technology, 2021, 37(4). 74 Overall system The overall feedback of participants assigned to use AR devices as a remote educational tool was generally positive. Users were reported to be excited about the new technology. In five experiments, they rated the system as easy to set up and use (Mather et al., 2017; Ponce, Jennings et al., 2014; Rojas-Munoz et al., 2018; Wang et al., 2017). The performance of the HMD and built-in screen-based systems was reported in the experiments. Rojas- Munoz et al. (2018) found that an AR HMD yielded smaller performance errors and fewer focus shifts in comparison with a telestrator system in which mentees used a nearby screen to retrieve the instructions. Andersen et al. (2017) developed a tablet display system similar to an HMD in which a tablet was positioned between the operating field and the local surgeon’s head. Local mentees using this design also had fewer placement errors and shifted focus away from the operating field less often than the telestrator system users. However, it did not eliminate the potential tablet collisions, which would not happen if mentees were wearing an HMD. In contrast, Carbone et al. (2018) developed a platform with a built-in screen, a video encoder and a webcam instead of a headset because they considered using an HMD was invasive and inconvenient. The participants agreed that the system created by Carbone et al. did not create clutter that could undermine the proper conduct of the diagnostic exam. Contrary to the positive response on AR systems reported by mentees, Wang et al. (2017) found that performance did not always improve, and the effectiveness of the HoloLens system in their experiment was rated low by the mentor. (Wang et al. noted that there was only one mentor in this experiment.) The mentor felt it was harder to provide guidance with this setup than a traditional multi-camera telemedicine system primarily due to malfunctions of the AR system and the poor quality of the local network connections. These difficulties resulted in a significantly longer time to complete the ultrasound exploration than if a low-cost setup using cameras to live stream and a headphone to facilitate two-way communication was used. Several other limitations of the HMDs were perceived by users. There were concerns about the heaviness of the equipment (Oyama et al., 2010; Wang et al., 2017), the inconvenience (Rio et al., 2018) and distraction (Carbone et al., 2018). In addition, Rio et al. found that the field of view available when using smart glasses was about 23°, which limited the AR visualisation area. Rojas-Munoz et al. (2018) commented that their AR HMD using HoloLens had a limited field of view. The incorrect adjustment of the images on the device could lead to user ocular fatigue and head discomfort. Wang et al. reported that user experience was influenced by the narrow field of view of the HoloLens when compared to the entire natural human view. HoloLens users also found the nose pad uncomfortable to wear. Some found that even when wearing a headband, achieving a good fit for smaller head sizes could be difficult. Ponce, Menendez et al. (2014) also pointed out divergent field of views between Google Glass and the surgeons’ view, where the Google Glass camera captured only the surgical view when the surgeons positioned their heads downward. Video or images or audio From the user perspective, the quality of video and audio was perceived as the ability to reliably see movements and instruments of the remote mentor and understand their instructions. In six out of seven experiments that measured users’ perspectives regarding video and audio quality, users perceived it to be of good quality. Although slight audio and visual delays were measured between remote and local substations, this did not interfere with performance of real-time guidance or the accuracy of the task performance (Davis et al., 2016; Ponce, Menendez et al., 2014; Shenai et al., 2011; Wagner et al., 1999). There was only one case where the surgeon wearing the Google Glass was unable to hear what the remote consultant was saying (Ponce, Menendez et al.). The authors assumed that this may have been caused by the noise of the operating room and the head fan of the battery pack used by the surgeon. Battery life was one of the main concerns of HDM devices. Ponce, Menendez et al. (2014) reported that battery life was limited when they used Google Glass to continuously send and receive real-time video, which required significant encoding or decoding and processing. Similarly, Wang et al. (2017) stated that the battery life of Microsoft HoloLens was a limitation of the system when they used it to capture the mentor’s hand gestures and virtually display these in the AR space of the HoloLens. The HoloLens could operate for about 100 minutes prior to having to be recharged, and one participant even finished the trial with a charging cable connected. Australasian Journal of Educational Technology, 2021, 37(4). 75 Health professional task performance A total of 12 experiments assessed the utility of the AR systems in performing tasks via 12 categories of users’ perspectives (Table 3). These categories were classified based on relevance with the topics of Task performance and Tele-mentoring. Table 3 Users’ perspectives on the task performance Feature Experiments* Task performance 1. Usefulness in performing tasks R2.2, R3, R4, R7.1, R7.2, R13, R14, R15, R16, R18, R22 2. User's belief and confidence R2.2, R3, R4, R7.1, R14, R16, R18, R22 3. Fatigue R4, R14, R18 4. Mental effort R18 5. Task difficulty R18 6. Personal performance R22 Tele-mentoring 1. Usefulness in mentoring R2.2, R3, R13, R16, R18 2. Usefulness in receiving instruction R7.1, R7.2, R10, R14, R15, R22 3. Ease of following the instruction R3, R7.1, R7.2, R13, R15, R22 4. Ease of following the visual instruction R7.1, R7.2, R15 5. Ease of following the verbal instruction R7.1, R7.2 6. Satisfaction with two-way interaction R7.1, R7.2, R10, R13 *R refers to the study code listed in the Appendix. Task performance Users in 11 experiments agreed that the AR system was an effective tele-mentoring device overall and resulted in the effective performance of a procedure. They believed that the tutor with a tele-consultation platform helped the mentee reach a diagnosis (Carbone et al., 2018). The experimental group using Helping Hands, an AR HMD device, demonstrated a greater improvement in confidence than the control group provided with face-to-face instruction (Mather et al., 2017). Participants using Ghostman, an AR tele- rehabilitation system, were confident that they were better able to use chopsticks than before the experiment (Chinthammit et al., 2014). Statistics also confirmed their perspective as the number of errors in performing tasks reduced significantly from pre-test to post-test. The effectiveness of an AR tele-mentoring system in performing health professional tasks was demonstrated by a statistically significant reduction in task errors and improvement in task completion time. Task completion time was measured in 15 out of 25 experiments, whilst six experiments measured the number of errors during task performance. Nine experiments showed that AR users performed the task faster than non-AR users. In contrast, the AR users were significantly slower in six experiments due to limitations related to the systems. This may have been due to the delay in the network, video and audio signals (Ponce, Menendez et al., 2014; Wang et al., 2017); the narrow field of view of the HMD; the distorted correspondence between the displayed view and the real view and the deteriorated sense of distance (Oyama et al., 2010; Wang et al., 2017); or the deficiencies in hand-eye coordination due to the lack of a fully transparent effect on the display (Andersen et al., 2016). A longer time could also have been experienced as participants tried to match their operative field to the absolute 3D position in which the annotation was supposed to be located (Rojas-Munoz et al., 2018). In addition, Andersen et al. (2017) and Rojas-Munoz et al. (2018) demonstrated that with AR systems, the task error was significantly reduced compared to conventional telestrators. Vera et al. (2014) also argued that although their AR system did not produce a significant difference in task errors, the AR group was faster without compromising accuracy than the group using traditional tele-mentoring. Tele-mentoring Ten experiments that assessed the ability of the systems to provide instruction from a distance showed positive feedback. Mentors and participants agreed that the systems were useful in mediating instruction remotely and in receiving the mentor’s guidance during the procedure via the virtual interactive presence Australasian Journal of Educational Technology, 2021, 37(4). 76 of the remote expert. They found it easy to follow both visual and verbal instruction and were also satisfied with the two-way interaction with the instructor. There was only one exception to these positive reports. A remote expert rated poorly the system using the HoloLens integrated with the Leap Motion sensor as the sensor appeared to be not working properly. However, regardless of the limitations of the technology used, participants felt it was easier to use the HoloLens application to help during an ultrasound scan procedure because the mental effort rating and task difficulty rating were lower than a full tele-medicine setup (Wang et al., 2017). The advantages and limitations of AR technology are summarised in Table 4. Table 4 Advantages and limitations of AR technology Advantages Limitations • Ease in setup and use • Reduction in shifts in focus • Reduction in skill errors • Improvement in task completion time • Improvement in task performance • Improvement in trainees’ confidence • Usefulness in remote instruction • Heaviness of the equipment • Inconvenience of wearing devices • Ocular fatigue and head or nose discomfort of headsets • Distraction of headsets • Limited field of view • Limited battery life • Latency of network, video and audio signals • System malfunctioning or not working properly Discussion This review identified a total of 25 experiments reported in 22 articles that the application of AR technology in tele-mentoring systems in healthcare environments were included. The experiments indicated that an AR approach has the potential to improve the effectiveness of tele-mentoring systems in both clinician training and medical practice. In training, AR technology enabled trainees, as mentees in a tele-mentorship, to immerse themselves in a real practice environment. The real-time feedback functionality available via the two-way communication capabilities of the AR systems may improve trainees’ educational experience through their greater involvement. This approach is consistent with modern learning theories advocating active participation of learners with immediate application of knowledge (Dreyfus & Dreyfus, 2004). In medicine, local less- experienced health practitioners, in the role of the mentees, could consider applying AR technology to receive and follow instructions verbally and visually from remote experts. The practitioners could also see annotations that are overlaid on the visual field of the local user in real time. Ponce, Jennings et al. (2014) found that practitioners using AR tele-mentoring systems perceived great comfort under the supervision of remote instructors located outside the practice field, as they experienced a feeling of autonomy while still having appropriate oversight. For remote instructors, the analysis of their perspectives showed the improvement in their instruction. The systems can convey instructors’ hand gestures or annotations, and this was thought to be more natural and precise than verbal instruction only. In clinical environments, remote experts in the role of the mentors could use AR technology to guide local less-experienced health practitioners, as the mentees, in performing medical procedures. They could use AR tele-mentoring systems to improve the mentees’ confidence, task completion time and task performance, as well as to reduce their task errors and focus shifts. Technically, the connection was, more often than not, established under a local network. Therefore, the experimental systems were enabled, and local network security officers mandated the use of authentication. Security concerns and issues, such as that of privacy, may be raised when connecting the AR system to the Internet using a public network. In this case, the devices are vulnerable due to the risk of unauthorised access to personal information. The devices without login entry password-protected capability also may be recognised as unsecured devices and removed from the network by the network security officers. Other technical problems existed in the design and installation of the AR systems. HMDs could be improved in areas such as increased processor power to provide higher definition video image quality; better sound quality; lightweight; long battery life; larger field of view; and camera alignment with the human line of sight. Moreover, improvements are necessary to minimise the delays or loss of network connection that Australasian Journal of Educational Technology, 2021, 37(4). 77 may result in longer completion time or higher error rates or even leave the local clinician without expert assistance (Anvari et al., 2005; Geelhoed et al., 2009). Systems accessing the wireless network also need to deal with poor wireless connectivity attributed to the absence of the wireless signal coverage, the weakness of the signal, or the incompatibility between the wireless system and the AR system. In one experiment, Andersen et al. (2019) suggested that a future step in visualisation could be a backup mechanism in case of poor network connectivity. For example, before the experiment, the steps of the procedure or performance task could be audio-recorded, and then during the experiment, users could switch between the live instruction provided by the mentor in real time and the instruction recorded previously and stored in the device. AR devices could be improved technically in areas such as weight, field of view, ergonomic design, stability of the network connection and battery life. Although the benefits of AR tele-mentoring systems in practice were demonstrated in all 25 experiments, challenges were identified in experimental design. The sample sizes of the studies were limited. This could result in potential biases in findings and prevent the generalisation to other settings. Next, while the use of non-virtual interactive technology to access expertise in real-time has helped reduce mortality and costs (Marttos et al., 2013; Wilcox & Adhikari, 2012), the AR experiments included in this review showed a lack of measurable evidence on how a virtual interactive presence impacted patient care outcomes and cost benefits. Additionally, many experiments were not conducted under entirely real circumstances or uncontrolled conditions. Because participants had been drawn from a healthy population, it was difficult to claim that the technique was valid without examining its efficacy on patients who were currently the recipients of medical treatment. The majority of experiments were performed and assessed in laboratory settings, and not in routine clinical practice environments, that is, hospitals or clinics. Further AR-related experiments are necessary to better explore AR tele-mentoring systems’ utility across a range of healthcare environments with larger samples drawn from real patient populations, including cost-benefit analysis and assessment of the impacts on short- and long-term patient outcomes. Limitations Lack of statistical power associated with small sample sizes, indeterminant effect size and other methodological shortcomings of the experiments selected limit the generalisations that could be made from this review. It is also limited by excluding any non-English studies on the basis that this allowed a consensus to be reached on the articles to include in the analysis. As a result, some relevant papers may have been excluded. The benefits of AR applications in healthcare tele-mentoring have yet to be fully determined. Their effectiveness compared to non-AR or other virtual reality–based devices requires further investigation. However, AR represents an emerging technology in health environments. Those reported in these experiments were broadly based on experimental design and thus are able to point to possible causal relationships. Conclusion This paper reports on a systematic literature review that aimed to identify how tele-mentoring systems that incorporate AR technology are being used in healthcare environments. In total, 25 experiments consisting of randomised controlled trials and non-randomised were analysed by using the concept-centric approach. AR tele-mentoring system was shown to benefit various outcomes in terms of task performance and system performance, including reduction in skill errors and focus shifts, improvement in task completion time and in task accuracy, as well as positive feedback from relevant users. There was a lack of evidence on the effect of AR tele-mentoring systems on patient outcomes and the retention of professional skills, indicating that further evaluations are required. Acknowledgements The authors acknowledge the support received from the University of Tasmania and the Commonwealth Government Department of Health Rural Health Multidisciplinary Training program. Australasian Journal of Educational Technology, 2021, 37(4). 78 References Aarnio, P., Rudenberg, H., Ellonen, M., & Jaatinen, P. (2000). User satisfaction with teleconsultations for surgery. Journal of Telemedicine and Telecare, 6(4), 237–241. https://doi.org/10.1258/1357633001935301 Agarwal, R., Levinson, A. W., Allaf, M., Makarov, D. V., Nason, A., & Su, L.-M. (2007). The Robo Consultant: Tele-mentoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. Adult Urology, 70(5), 970–974. https://doi.org/10.1016/j.urology.2007.09.053 Andersen, D., Cabrera, M. E., Rojas-Munoz, E. J., Popescu, V. S., Gonzalez, G. T., Mullis, B., Marley, S., Zarzaur, B. L., & Wachs, J. P. (2019). Augmented reality future step visualization for robust surgical telementoring. Simulation in Healthcare, 14(1), 59–66. https://doi.org/10.1097/sih.0000000000000334 Andersen, D., Popescu, V., Cabrera, M. E., Shanghavi, A., Gomez, G., Marley, S., Mullis, B., & Wachs, J. (2016). Virtual annotations of the surgical field through an augmented reality transparent display. Visual Computer, 32(11), 1481–1498. https://doi.org/10.1007/s00371-015-1135-6 Andersen, D., Popescu, V., Cabrera, M. E., Shanghavi, A., Mullis, B., Marley, S., Gomez, G., & Wachs, J. P. (2017). An Augmented Reality-based approach for surgical telementoring in austere environments. Military Medicine, 182(S1), 310–315. https://doi.org/10.7205/milmed-d-16-00051 Antoniou, S. A., Antoniou, G. A., Franzen, J., Bollmann, S., Koch, O. O., Pointner, R., & Granderath, F. A. (2012). A comprehensive review of telementoring applications in laparoscopic general surgery. Surgical Endoscopy, 26(8), 2111–2116. https://doi.org/10.1007/s00464-012-2175-x Anvari, M., Broderick, T., Stein, H., Chapman, T., Ghodoussi, M., Birch, D. W., Mckinley, C., Trudeau, P., Dutta, S., & Goldsmith, C. H. (2005). The impact of latency on surgical precision and task completion during robotic-assisted remote telepresence surgery. Computer Aided Surgery, 10(2), 93– 99. https://doi.org/10.3109/10929080500228654 Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators & Virtual Environments, 6(4), 355–385. https://doi.org/10.1162/pres.1997.6.4.355 Barsom, E., Graafland, M., & Schijven, M. P. (2016). Systematic review on the effectiveness of augmented reality applications in medical training. Surgical Endoscopy, 30(10), 4174–4183. https://doi.org/10.1007/s00464-016-4800-6 Bogen, E. M., Augestad, K. M., Patel, H. R., & Lindsetmo, R.-O. (2014). Telementoring in education of laparoscopic surgeons: An emerging technology. World Journal of Gastrointestinal Endoscopy, 6(5), 148–155. https://doi.org/10.4253/wjge.v6.i5.148 Budrionis, A., Augestad, K. M., Patel, H. R., & Bellika, J. G. (2013). An evaluation framework for defining the contributions of telestration in surgical telementoring. Interactives Journal of Medical Research, 2(2), Article e14. https://doi.org/10.2196/ijmr.2611 Carbone, M., Ferrari, V., Marconi, M., Piazza, R., Del Corso, A., Adami, D., Lucchesi, Q., Pagni, V., & Berchiolli, R. (2018). A tele-ultrasonographic platform to collect specialist second opinion in less specialized hospitals. Updates in Surgery, 70(3), 407–413. https://doi.org/10.1007/s13304-018-0582-9 Chinthammit, W., Merritt, T., Pedersen, S., Williams, A., Visentin, D., Rowe, R., & Furness, T. (2014). Ghostman: Augmented reality application for telerehabilitation and remote instruction of a novel motor skill. Biomed Research International, 2014, Article 646347. https://doi.org/10.1155/2014/646347 Davis, M. C., Can, D. D., Pindrik, J., Rocque, B. G., & Johnston, J. M. (2016). Virtual Interactive Presence in global surgical education: International collaboration through augmented reality. World Neurosurgery, 86, 103–111. https://doi.org/10.1016/j.wneu.2015.08.053 Dey, A., Billinghurst, M., Lindeman, R. W., & Swan, J. E. (2018). A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5(37). https://doi.org/10.3389/frobt.2018.00037 Doze, S., Simpson, J., Hailey, D., & Jacobs, P. (1999). Evaluation of a telepsychiatry pilot project. Journal of Telemedicine and Telecare, 5(1), 38–46. https://doi.org/10.1258/1357633991932379 Dreyfus, H. L., & Dreyfus, S. E. (2004). The ethical Implications of the five-stage skill-acquisition model. Bulletin of Science, Technology & Society, 24(3), 251–264. https://doi.org/10.1177/0270467604265023 Erridge, S., Yeung, D. K. T., Patel, H. R. H., & Purkayastha, S. (2019). Telementoring of surgeons: A systematic review. Surgical Innovation, 26(1), 95–111. https://doi.org/10.1177/1553350618813250 https://doi.org/10.1258/1357633001935301 https://doi.org/10.1016/j.urology.2007.09.053 https://doi.org/10.1097/sih.0000000000000334 https://doi.org/10.1007/s00371-015-1135-6 https://doi.org/10.7205/milmed-d-16-00051 https://doi.org/10.1007/s00464-012-2175-x https://doi.org/10.3109/10929080500228654 https://doi.org/10.1162/pres.1997.6.4.355 https://doi.org/10.1007/s00464-016-4800-6 https://doi.org/10.4253/wjge.v6.i5.148 https://doi.org/10.2196/ijmr.2611 https://doi.org/10.1007/s13304-018-0582-9 https://doi.org/10.1155/2014/646347 https://doi.org/10.1016/j.wneu.2015.08.053 https://doi.org/10.3389/frobt.2018.00037 https://doi.org/10.1258/1357633991932379 https://doi.org/10.1177/0270467604265023 https://doi.org/10.1177/1553350618813250 Australasian Journal of Educational Technology, 2021, 37(4). 79 Geelhoed, E., Parker, A., Williams, D. J., & Groen, M. (2009). Effects of latency on telepresence (Report No. HPL-2009-120). Hewlett-Packard Development Company, L.P. https://www.hpl.hp.com/techreports/2009/HPL-2009-120.html Google. (n.d.). Discover Glass enterprise edition: A hands-free for smarter and faster hands-on work. https://www.google.com/glass/start/ Hanna, M. G., Ahmed, I., Nine, J., Prajapati, S., & Pantanowitz, L. (2018). Augmented reality technology using Microsoft HoloLens in anatomic pathology. Archives of Pathology & Laboratory Medicine, 142(5), 638–644. https://doi.org/10.5858/arpa.2017-0189-OA Hong, Q. N., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M. P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M. C., & Vedel, I. (2018). The mixed methods appraisal tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34(4), 285–291. https://doi.org/10.3233/EFI-180221 Issenberg, S. B., & Scalese, R. J. (2008). Simulation in health care education. Perspectives in Biology and Medicine, 51(1), 31–46. https://doi.org/10.1353/pbm.2008.0004 Kotwal, R. S., Howard, J. T., Orman, J. A., Tarpey, B. W., Bailey, J. A., Champion, H. R., Mabry, R. L., Holcomb, J. B., & Gross, K. R. (2016). The effect of a golden hour policy on the morbidity and mortality of combat casualties. JAMA Surgery, 151(1), 15–24. https://doi.org/10.1001/jamasurg.2015.3104 Lin, C., Andersen, D., Popescu, V., Rojas-Muñoz, E., Cabrera, M. E., Mullis, B., Zarzaur, B., Anderson, K., Marley, S., & Wachs, J. (2018). A first-person mentee second-person mentor AR interface for surgical tele-mentoring. In U. Eck & O. Hilliges (Eds.), ISMAR-Adjunct: Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality (pp. 3–8). IEEE. https://doi.org/10.1109/ISMAR-Adjunct.2018.00021 Marttos, A., Kelly, E., Graygo, J., Rothenberg, P., Alonso, G., Kuchkarian, F. M., Gibson, S., Augenstein, J., & Schulman, C. I. (2013). Usability of telepresence in a level 1 trauma center. Telemedicine Journal and E-Health, 19(4), 248–251. https://doi.org/10.1089/tmj.2012.0102 Mather, C., Barnett, T., Broucek, V., Saunders, A., Grattidge, D., & Huang, W. (2017). Helping Hands: Using augmented reality to provide remote guidance to health professionals. Studies in Health Technology and Informatics, 241, 57–62. https://doi.org/10.3233/978-1-61499-794-8-57 Microsoft. (2019). Microsoft HoloLens. https://docs.microsoft.com/en-us/hololens/ Mitsuno, D., Hirota, Y., Akamatsu, J., Kino, H., Okamoto, T., & Ueda, K. (2019). Telementoring demonstration in craniofacial surgery with HoloLens, Skype, and three-layer facial models. Journal of Craniofacial Surgery, 30(1), 28–32. https://doi.org/10.1097/scs.0000000000004899 Moehr, J. R., Schaafsma, J., Anglin, C., Pantazi, S. V., Grimm, N. A., & Anglin, S. (2006). Success factors for telehealth: A case study. International Journal of Medical Informatics, 75(10-11), 755– 763. https://doi.org/10.1016/j.ijmedinf.2005.11.001 Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), Article e1000097. https://doi.org/10.1371/journal.pmed.1000097 Oyama, E., Watanabe, N., Mikado, H., Araoka, H., Uchida, J., Omori, T., Shinoda, K., Noda, I., Shiroma, N., Agah, A., Hamada, K., Yonemura, T., Ando, H., Kondo, D., & Maeda, T. (2010). A study on wearable behavior navigation system - Development of simple parasitic humanoid system. In R. Mahony & E. Oyama (Eds.), Proceedings of the 2010 IEEE International Conference on Robotics and Automation (pp. 5315–5321). IEEE. https://doi.org/10.1109/ROBOT.2010.5509141 Panait, L., Rafiq, A., Tomulescu, V., Boanca, C., Popescu, I., Carbonell, A., & Merrell, R. C. (2006). Telementoring versus on-site mentoring in virtual reality-based surgical training. Surgical Endoscopy, 20(1), 113–118. https://doi.org/10.1007/s00464-005-0113-x Ponce, B. A., Jennings, J. K., Clay, T. B., May, M. B., Huisingh, C., & Sheppard, E. D. (2014). Tele- mentoring: Use of Augmented Reality in orthopaedic education: AAOS exhibit selection. Journal of Bone and Joint Surgery, 96(10), Article e84. https://doi.org/10.2106/JBJS.M.00928 Ponce, B. A., Menendez, M. E., Oladeji, Fryberger, C. T., & Dantuluri, P. K. (2014). Emerging technology in surgical education: Combining real-time augmented reality and wearable computing devices. Orthopedics, 37(11), 751–757. https://doi.org/10.3928/01477447-20141023-05 Rio, M. D., Meloni, V., Frexia, F., Cabras, F., Tumbarello, R., Montis, S., Marini, A., & Zanetti, G. (2018). Augmented reality for supporting real time tele-mentoring: An exploratory study applied to ultrasonography. In T. Sakurai & C. Chang (Eds.), Proceedings of the 2nd International Conference on Medical and Health Informatics (pp. 218–222). ACM. https://doi.org/10.1145/3239438.3239444 https://www.hpl.hp.com/techreports/2009/HPL-2009-120.html https://www.google.com/glass/start/ https://doi.org/10.5858/arpa.2017-0189-OA https://doi.org/10.3233/EFI-180221 https://doi.org/10.1353/pbm.2008.0004 https://doi.org/10.1001/jamasurg.2015.3104 https://doi.org/10.1109/ISMAR-Adjunct.2018.00021 https://doi.org/10.1089/tmj.2012.0102 https://doi.org/10.3233/978-1-61499-794-8-57 https://docs.microsoft.com/en-us/hololens/ https://doi.org/10.1097/scs.0000000000004899 https://doi.org/10.1016/j.ijmedinf.2005.11.001 https://doi.org/10.1371/journal.pmed.1000097 https://doi.org/10.1109/ROBOT.2010.5509141 https://doi.org/10.1007/s00464-005-0113-x https://doi.org/10.2106/JBJS.M.00928 https://doi.org/10.3928/01477447-20141023-05 https://doi.org/10.1145/3239438.3239444 Australasian Journal of Educational Technology, 2021, 37(4). 80 Rojas-Munoz, E., Cabrera, M. E., Andersen, D., Popescu, V., Marley, S., Mullis, B., Zarzaur, B., & Wachs, J. (2018). Surgical telementoring without encumbrance: A comparative study of see-through augmented reality-based approaches. Annals of Surgery, 270(2), 384–389. https://doi.org/10.1097/sla.0000000000002764 Schlachta, C. M., Lefebvre, K. L., Sorsdahl, A. K., & Jayaraman, S. (2010). Mentoring and telementoring leads to effective incorporation of laparoscopic colon surgery. Surgical Endoscopy, 24(4), 841–844. https://doi.org/10.1007/s00464-009-0674-1 Sebajang, H., Trudeau, P., Dougall, A., Hegge, S., McKinley, C., & Anvari, M. (2006). The role of telementoring and telerobotic assistance in the provision of laparoscopic colorectal surgery in rural areas. Surgical Endoscopy, 20(9), 1389–1393. https://doi.org/10.1007/s00464-005-0260-0 Shenai, M. B., Dillavou, M., Shum, C., Ross, D., Tubbs, R. S., Shih, A., & Guthrie, B. L. (2011). Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance. Neurosurgery, 68(1). https://doi.org/10.1227/NEU.0b013e3182077efd Sielhorst, T., Obst, T., Burgkart, R., Riener, R., & Navab, N. (2004). An augmented reality delivery simulator for medical training. In M. Berger & N. Navab (Eds.), Proceedings of AMI-ARCS 2004: International Workshop on Augmented Environments for Medical Imaging Conference and Computer-aided Surgery (pp. 11–20). https://ami2004.loria.fr/PAPERS/26obetoebiel.pdf Sommerauer, P., & Müller, O. (2018). Augmented reality for teaching and learning - A literature review on theoretical and empirical foundations. Research Papers, 31. https://aisel.aisnet.org/ecis2018_rp/31/ Van Zuuren, E., & Fedorowicz, Z. (2016). Moose on the loose: Checklist for meta-analyses of observational studies. The British Journal of Dermatology, 175(5), 853–854. https://doi.org/10.1111/bjd.15038 Vera, A. M., Russo, M., Mohsin, A., & Tsuda, S. (2014). Augmented reality tele-mentoring (ART) platform: A randomized controlled trial to assess the efficacy of a new surgical education technology. Surgical Endoscopy, 28(12), 3467–3472. https://doi.org/10.1007/s00464-014-3625-4 Wagner, A., Millesi, W., Watzinger, F., Truppe, M., Rasse, M., Enislidis, G., Kerner, C., & Ewers, R. (1999). Clinical experience with interactive teleconsultation and teleassistance in craniomaxillofacial surgical procedures. Journal of Oral and Maxillofacial Surgery, 57(12), 1413–1418. https://doi.org/10.1016/s0278-2391(99)90722-x Wang, S., Parsons, M., Stone-McLean, J., Rogers, P., Boyd, S., Hoover, K., Meruvia-Pastor, O., Gong, M., & Smith, A. (2017). Augmented reality as a telemedicine platform for remote procedural training. Sensors, 17(10). https://doi.org/10.3390/s17102294 Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2). https://www.jstor.org/stable/4132319 Wilcox, M. E., & Adhikari, N. K. (2012). The effect of telemedicine in critically ill patients: Systematic review and meta-analysis. Critical Care, 16(4), Article R127. https://doi.org/10.1186/cc11429 Yoshinaga, T., Arita, D., Miyazaki, W., & Masuda, K. (2011). Development of tele-echography interface with AR/VR visualization system of internal organs. In G. Cauwenberghs & A. Mason (Eds.), Proceedings of the 2011 IEEE Biomedical Circuits and Systems Conference (pp. 90–93). IEEE. https://doi.org/10.1109/BioCAS.2011.6107734 Zhu, E., Hadadgar, A., Masiello, I., & Zary, N. (2014). Augmented Reality in healthcare education: An integrative review. PeerJ, 2, Article e469. https://doi.org/10.7717/peerj.469 Corresponding author: Dung Trung Bui, dungtrung.bui@utas.edu.au Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Bui, D. T., Barnett, T., Hoang, H., & Chinthammit, W. (2021). Tele-mentoring using augmented reality technology in healthcare: A systematic review. Australasian Journal of Educational Technology, 37(4), 68-88. https://doi.org/10.14742/ajet.6243 https://doi.org/10.1097/sla.0000000000002764 https://doi.org/10.1007/s00464-009-0674-1 https://doi.org/10.1007/s00464-005-0260-0 https://doi.org/10.1227/NEU.0b013e3182077efd https://ami2004.loria.fr/PAPERS/26obetoebiel.pdf https://aisel.aisnet.org/ecis2018_rp/31/ https://doi.org/10.1111/bjd.15038 https://doi.org/10.1007/s00464-014-3625-4 https://doi.org/10.1016/s0278-2391(99)90722-x https://doi.org/10.3390/s17102294 https://www.jstor.org/stable/4132319 https://doi.org/10.1186/cc11429 https://doi.org/10.1109/BioCAS.2011.6107734 https://doi.org/10.7717/peerj.469 mailto:dungtrung.bui@utas.edu.au https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.6243 Australasian Journal of Educational Technology, 2021, 37(4). 81 Appendix Characteristics of the included studies Reference System & location Aim Design, sample & intervention Findings (R1) Andersen et al., 2017 System for tele- mentoring with AR (STAR), USA. Compare STAR with telestration where annotations are provided on a monitor outside the local surgeon’s field of view. Randomised control trial (RCT) 10 premedical and medical students used STAR, while 10 others used telestration system. Participants performed two simulated surgical tasks under tele-mentored guidance. - Participants who used STAR completed the task with fewer focus shifts and with greater accuracy. (R2.1) Carbone et al., 2018 Tele- ultrasono- graphic platform, Italy. Assess quality of ultrasound video on healthy patients and evaluate the usability and trustworthiness of the platform Quasi-experimental design (post-test only). 5 clinicians executed ultrasound evaluation on healthy volunteers under the guidance of a remote expert. - Confirmed the robustness of the platform and the trust of clinicians in its potential. - Allowed some aspects of the workflow to be assesses. (R2.2) Carbone et al., 2018 Tele- ultrasono- graphic platform, Italy. Explore the utility of the system from both local and remote clinician perspectives. Quasi-experimental design (post-test only). Clinicians performed Doppler ultrasound on 12 patients under the guide of a remote expert. - The platform and the telemedicine paradigm can reduce the costs related to transporting critical patients when there is a need for a specialist second opinion. - Having an expert guide and comment on the diagnostic examination allowed the local clinicians to grow in competencies over time. (R3) Chinthammit et al., 2014 Tele- rehabilitatio n system (Ghostman), Australia. Evaluate the efficacy of the Ghostman prototype as a tool for remote teaching of a novel motor skill using chopsticks. RCT design (pre- and post- tests). 12 participants were randomised to two groups receiving either Ghostman or face-to-face instructions by an instructor to use a pair of chopsticks and transfer all the blocks one at a time to the target bowl. Ghostman is as effective for motor learning, in terms of reduction in skill errors and improvements in task completion time, as the current best practice face-to-face training. (R4) Davis et al., 2016 Virtual interactive presence and AR (VIPAR), USA and Vietnam. Evaluate system performance in an international telecollaboration and continuing education between two groups of surgeons using the VIPAR system. Quasi-experimental design (post-test only). A paediatric neurosurgeon in the USA was contacted with a neurosurgeon in Vietnam via VIPAR during a neurosurgery. - VIPAR allowed both surgeons to engage in complex visual and verbal communication during the procedure. - Video delay of 237 milliseconds relative to the audio signal. Excellent image resolution allowed the remote neurosurgeon to visualise all critical anatomy. - The remote neurosurgeon could gesture to structures with no detectable difference in accuracy between stations, allowing for sub-millimetre precision. Australasian Journal of Educational Technology, 2021, 37(4). 82 Reference System & location Aim Design, sample & intervention Findings (R5) Hanna et al., 2018 Microsoft HoloLens, USA. Test the HoloLens for clinical and nonclinical applications in pathology. Quasi-experimental design (post-test only). Pathology assistants performed specimen radiograph co-registration in either Conventional or AR workflows. - The HoloLens is a novel AR tool with multiple clinical and nonclinical applications in pathology. - The device was comfortable to wear, easy to use, provided sufficient computing power and supported high-resolution imaging. - It was useful and usable technology for radiograph specimen co-registration. (R6) Masuda et al., 2011 Tele- echography system, Japan. Confirm effectiveness of non-robotic system in a domestic or emergency, respectively. Quasi-experimental design. 4 university students handled the probe to obtain an echocardiogram skill under following 2 conditions: (1) The operator hears only voice from the remote technician. (2) The operator refers to AR interface that represents the probe operation instructed by the remote doctor with voice. - The operability was improved by using the graphical interface. - Instructing the probe operation contributed to shorten time to obtain echogram. It was confirmed to be able to support tele- echography by using AR interface. - Tele-echography has an aspect of effectiveness but it is adapted for the situation. (R7.1) Mather et al., 2017 AR audio- visual guidance system (Helping Hands), Australia. A feasibility trial to ensure that clinical skill outcomes were not being compromised by using the innovative AR approach. RCT design (pre- and post- tests). 15 nursing students were randomly allocated to one of two groups: (1) The control group received the usual instruction; (2) The intervention group received the Helping Hands instruction. All student participants were instructed to wash their hands after applied Glitter bug, and then their hands were photographed under UV light. - Clinical skill outcomes are not compromised by using the Helping Hands technology. - Participants were similarly confident in their technique regardless of the method of instruction they received. - The experimental (Helping Hands) group demonstrated a greater improvement in confidence than the control group. - A tele-guidance system such as Helping Hands is readily acceptable by end users. (R7.2) Mather et al., 2017 AR audio- visual guidance system (Helping Hands), Australia. Examine how well the equipment worked in different clinical locations and with different end users. Quasi-experimental design (post-test only). 4 nursing students and 13 paramedic students were instructed in the handwashing technique using Helping Hands - The system was easy to learn and a useful way to receive guidance. - Where Wi-Fi, Internet and/or the 3G network is available, the Helping Hands technology can augment current student training and potentially provide a sophisticated backup for trained medical personnel while making significant savings in time, resources and the utilisation of expertise. Australasian Journal of Educational Technology, 2021, 37(4). 83 Reference System & location Aim Design, sample & intervention Findings (R8) Mitsuno et al., 2019 Microsoft HoloLens with a built- in Skype, Japan. Reveal the advantages and problems of the system through a series of demonstrations. Quasi-experimental design. A resident surgeon performed surgery procedures on a 3-layer model of complex facial fractures and a 2-layer model of cleft lip under remote guidance by a mentor surgeon 4 times on different dates via HoloLens. - There was no delay in voice communication and a delay of < 0.5 seconds in the video. - The resident was able to confirm the main landmarks of the surgical field and to grasp the situation without problems. - The mentor could send appropriate instructions by voice, could point out a specific part by telestration function, and could draw lines on the 2-dimentional images pasted on the operator’s field of vision. (R9) Oyama et al., 2010 Wearable behaviour navigation system (WBNS), Japan. Evaluate the effectiveness of the system. Quasi-experimental design (post-test only). - Preliminary experiment: 8 participants saw the goal of the treatment as shown in a picture and then made an arm sling for a person with injured forearm by using a triangular bandage. - Experiment: 8 participants wore the system and made an arm sling under the instruction of a remote expert. - With the WBNS, all the subjects successfully completed the treatment. Without the WBNS, most subjects finished the treatment much faster than the subjects with the WBNS. - 87.5% of the subjects could not make an arm sling appropriately. - 37.5% of the subjects felt that the head mounted display was heavy. 25% of the subjects wished for an extension of the field of view. The other 25% of the subjects wished for the natural correspondence between the displayed and the real views. (R10) Ponce et al., 2014 Virtual interactive presence (VIP), USA. Evaluate the efficiency and performance of a VIP system implemented in an operating room setting, determine the potential utility of the system for guidance of surgical procedures, and assess the safety of the system. Quasi-experimental design (post-test only). 6 surgical residents used the VIP system to remotely proctor one or more resident surgeons as portions of the arthroscopic shoulder surgical procedure were performed. - The VIP technology was efficient, safe and effective as a teaching tool. - Training was enhanced, and this occurred without increasing operative times. - This technology improved teaching effectiveness. Australasian Journal of Educational Technology, 2021, 37(4). 84 Reference System & location Aim Design, sample & intervention Findings (R11) Ponce et al., 2014 Google Glass (GG) and VIPAAR, USA. Analyse the feasibility of AR technology with a mobile video conferencing platform in a surgical setting. Quasi-experimental design (pre- and post-tests). A surgeon wore GG during the performance of a shoulder surgery and interacted with and streamed live video to a remote expert who used the VIPAAR - GG can be mounted and worn in the operating room and that the system can host facilitative technology applications such as VIPAAR. - Its use as a real-time surgically optimised AR interface is not yet practical. Improvements to GG such as longer battery life, increased processor power providing higher-definition video image quality, an operating room optimised matching of surgeon and camera view lines, and better sound quality will enhance its performance in the operating room. (R12) Rio et al., 2018 Real time tele- mentoring system (CRS4- TELEMED, Italy. Evaluate the impact of AR in remotely assisted ultrasound exams. Quasi-experimental design (post-test only). 4 doctors were introduced to the system and then performed a basic cardiac ultrasound exam on a volunteer under the guidance by a remote specialist using CRS4- TELEMED. - Participants judged the system positively, considering AR tele-pointers useful for improving remote assistance. - The following issues emerged: optical alignment is suboptimal, some training is required to understand how smart glasses and AR work, smart glasses result uncomfortable, especially if the user wears glasses. (R13) Rojas-Munoz et al., 2018 Surgical tele- mentoring system based on ARHMD, USA. Investigate the benefits of the ARHMD. RCT design (Post-test only). 20 medical students were randomly assigned to one of two tele-mentoring conditions ARHMD or telestrator and then performed 2 tele-mentored tasks (an anatomical marker placement and a mock abdominal incision) on a patient simulator. - The ARHMD condition yielded smaller placement errors, fewer focus shifts and longer completion times. - The ARHMD avoided potential tablet collisions. (R14) Shenai et al., 2011 VIPAR, USA. Demonstrate the feasibility of the system. Quasi-experimental design. A resident neurosurgeon receiving the instruction from a remote neurosurgeon performed a carotid endarterectomy (CEA) and pterional craniotomy on a human cadaver sample under the VIPAR system. - The VIPAR system allowed for real-time, virtual interaction between a local resident and remote surgeon. - In both carotid and pterional dissections, major anatomic structures were visualised and identified. - Virtual interaction permitted remote instruction for the local surgeon, and MRI augmentation provided spatial guidance to both surgeons. - Camera resolution, colour contrast, time lag and depth perception were identified as technical issues requiring further optimisation. Australasian Journal of Educational Technology, 2021, 37(4). 85 Reference System & location Aim Design, sample & intervention Findings (R15) Suenaga et al., 2001 AR based tele- instruction system, Japan. Evaluate the utility of the proposed system and the accuracy of the transmitted instructions. Quasi-experimental design. Postgraduate students handled the ultrasound diagnostic device to get the long axis view of the chest and four chamber view from the left side with/ without using the system. Three methods of instruction were used: (1) Instruction by voice only, (2) instruction by voice and the proposed system without the immersive display, and (3) instruction by voice and the proposed system with the immersive display. - The proposed telemedicine system realised smooth communication and an intuitive interface. - Some problems occurred in the first trial when acquiring images from the side, because the web-mark was distorted due to the non-flat display surface, i.e., the patient’s body. - The experimental results confirmed that support information effectively resolved this problem. - Although a definite difference was not found, the immersive display seemed to be more useful for tele-instruction. (R16) Vera et al., 2014 AR ele- mentoring platform (ART), USA. Compare the effectiveness of this new teaching modality to traditional methods in novices performing an intra-corporeal suturing task. RCT design (post-test only). 19 medical students were randomised into traditional mentoring and AR group. Students received either traditional mentoring or ART for 1 hour on the validated fundamentals of laparoscopic surgery intracorporal suturing task - The ART platform may be a more effective training technique in teaching laparoscopic skills to novices compared to traditional methods. - ART conferred a shorter learning curve, which was more pronounced in the first 4 trials. - ART reduced the number of failed attempts and resulted in faster suture times by the end of the training session. (R17) Wagner et al., 1999 Virtual patient system (VPS), Austria and 20 global locations. Evaluate the clinical value and feasibility of the VPS for tele- navigation and interactive tele- assistance with composite reality environments Quasi-experimental design (Post-test only). Surgical teams used VPS for computer-aided tele- navigation and interactive tele-assistance between clinics in various cases of the cranio-maxillofacial surgeries - The principles of computer- aided tele-navigation were applied successfully. - Technical problems in 6 cases did not cause a breakdown of overall system performance. - Tele-consultation with remote experts was a useful tool, although some shortcomings exist. - The financial and personal effort involved is considerable. Australasian Journal of Educational Technology, 2021, 37(4). 86 Reference System & location Aim Design, sample & intervention Findings (R18) Wang et al., 2017 HoloLens and Leap Motion, Canada. Explore the feasibility and user experiences of novice practitioners and a mentor using AR to enhance remote Point of Care Ultrasound (PoCUS) training and compare the performance to a standard remote training platform RCT design (post-test only). 24 paramedics and undergraduate medical students were assigned into either HoloLens group or telemedicine group under the guide of a mentor. Each student completed a right upper quadrant focused assessment using sonography in trauma ultrasound examination on a healthy volunteer. - Compared to available tele- mentoring applications that mostly include visual and auditory instructions, the system was more immersive as it presented a controlled hand model with an attached ultrasound transducer. - Compared to other gesture- based AR systems, the system was easier to set up and run. - Every participant reported different levels of physical discomfort during the study, and an assistant must ensure that the device is properly worn. - The completion time for the HoloLens application is longer than the other setup. - The single mentor reported that the task became harder when using the HoloLens. (19.1) Yoshinaga et al., 2011 Tele- echography system, Japan. Evaluate the effectiveness of visualisation of internal organs for remote instructor RCT design. 20 engineering students acquired a remote environment imaging experiment in two situations: (1) Showing only computer graphic body mark to acquire long- axis view; (2) Showing only computer graphic of left ventricle (10 students for each situation). - In both situations, there was no major difference seen in the average time required for the acquisition of long axis view. - At the same time, when comparing the results of short axis view taken under the same circumstances, a reduction was confirmed in average time. - Both sides had a verified hazard ratio of less than 1% and statistical significance was recognised in the time needed for imaging instruction depending on whether the left ventricle is made visible (19.2) Yoshinaga et al., 2011 Tele- echography system, Japan. Verify the effectiveness of the instruction using the system RCT design. A technician instructed 60 students the way to acquire echogram of long axis view of heart under two different instruction methods: (1) Voice only; (2) Voice and computer graphics representing the probe operation performed by the technician with. - Instructing the probe operation contributed to shorten time to obtain echogram. - It was confirmed to be able to support tele-echography by using this interface. Australasian Journal of Educational Technology, 2021, 37(4). 87 Reference System & location Aim Design, sample & intervention Findings (R20) Andersen et al., 2016 AR transparent display, USA. Compare the usability of the AR system with using a conventional system for tele- mentoring based on displaying mentor feedback on a nearby monitor RCT design. 22 students were randomly divided into two groups using either the AR or the conventional conditions to identify regions of the neck area of a patient, which usually is a necessary condition to conduct a cricothyrotomy. - AR condition: The participant saw mentor's hand and the sticker through the transparent display and would guide the sticker to coincide with the virtual annotation. - Conventional condition: The participant looked at the LCD and then back at the patient simulator for guidance as to where to place each sticker. - The placement error was considerably smaller when using the AR system than when using a separate screen. - Focus shifts were greatly reduced when using the tablet system as opposed to the conventional system. - The task completion time was slightly longer for the AR condition than it was for the conventional condition. (R21) Andersen et al., 2019 STAR, USA. Evaluate the effectiveness of a tele-mentoring system with visualisation of future steps in the context of a non-robust network connection RCT design (post-test only). 20 participants were randomly assigned to one of two groups using conventional telestrator or STAR to perform a simulated cricothyroidotomy on a patient simulator under tele-mentored guidance with an unstable connection. - Telestrator condition: a screen was positioned in front of the trainee, which showed visual guidance from the remote mentor to the trainee. - STAR condition: participants viewed the patient simulator by looking through a tablet preloaded with the future step visualisation of a cricothyroidotomy procedure. - The results indicate that surgical tele-mentoring with future step instruction excelled when compared with conventional telestrator-based tele-mentoring. - Participants using future step visualisation completed the operation with proportionally less idle time, with less recall error and with improved task performance, compared with participants using conventional tele-mentoring. Australasian Journal of Educational Technology, 2021, 37(4). 88 Reference System & location Aim Design, sample & intervention Findings (R22) Lin et al., 2018 AR surgical tele- mentoring system, USA. Test the system in the context of a fasciotomy tele-mentoring RCT design (post-test only). 14 surgery residents and 6 medical students were randomly assigned to one of two groups: - Control group (CG): participants received instruction on how to perform the fasciotomy from an illustrated brochure - Experiment group (EG): received real-time guidance with the tele-mentoring system - EG participants received a median IPS which was 16% higher than for CG participants. - EG participants reported a higher preference on the self- reported usability than CG participants - EG participants reported a statistically significant improvement in all four confidence categories - The CG participants were more confident than the EG participants in their knowledge of the procedure before the task, but EG participants were more confident after the task - EG participants completed the procedure marginally faster than CG participants Introduction Methods Search strategy Search outcomes Data extraction Methodological quality assessment Findings Overview AR tele-mentoring system performance Overall system Video or images or audio Health professional task performance Task performance Tele-mentoring Discussion Limitations Conclusion Acknowledgements References Appendix