Australasian Journal of Educational Technology, 2021, 37(3). 119 Using head mounted display virtual reality simulations in large engineering classes: Operating vs observing Andrew Valentine, Tom Van der Veen, Patrick Kenworthy, Ghulam Mubashar Hassan, Andrew Guzzomi, R. Nazim Khan, Sally A. Male The University of Western Australia A barrier to using head mounted display (HMD) virtual reality (VR) in education is access to hardware for large classes. This paper compares students’ learning when engaging with an HMD VR simulation as the operator and as the observer, to evaluate whether benefits of HMD VR can be achieved without requiring all students to operate the equipment. Postgraduate engineering students (N = 117) completed a safety hazard identification exercise in a workshop. The performance of students who operated and observed was compared. Results showed that students performed similarly in the exercise that followed the simulation whether they operated HMD VR (n = 33) or observed (n = 84). The finding suggests that educators may be able to use HMD VR simulations in classes with a large enrolment, by reducing the need for investment and management of a large number of sets of HMD VR equipment. Implications for practice or policy:  Engineering educators can use HMD VR simulations to teach students about safety in design.  Engineering students are able to identify safety hazards in a HMD VR simulation effectively whether they are operating the equipment or observing another student in their group operating the VR equipment.  One HMD VR set per student group is sufficient.  HMD VR simulations can be used inclusively, even when some students are unable or unwilling to wear the headset. Keywords: virtual reality, engineering education, safety in design, work-integrated learning, active learning Introduction Learning is facilitated most effectively when: students have necessary prior knowledge, students are able to elaborate on their knowledge, and the situation in which something is learned closely resembles the situation in which it will later be applied (Schmidt, 1983). Educators can provide an authentic environment for learners by using virtual reality (VR), which has seen widespread and increased usage since 1991 (Slater & Sanchez-Vives, 2016). Benefits of VR in education One of the primary advantages of VR is that it can enable users to experience situations where it may otherwise not be feasible to do so in reality, such as dangerous workplaces (Freina & Ott, 2015; Slater & Sanchez-Vives, 2016). Additionally, VR can enable a user to role-play the experience of another person more easily, leading to greater empathy and understanding of the other’s perspective (Herrera et al., 2018). Within educational contexts, using VR systems can help students’ engagement and motivation and help them to understand concepts (Huang et al., 2010). Use of VR (especially head mounted display; HMD) can improve knowledge retention compared with traditional educational methods (Chittaro & Buttussi, 2015) and has been shown to lead to better memory recall than situations that do not use VR (Krokos et al., 2019). In a systematic literature review covering the use of VR in education, Kavanagh et al. (2017) found that expectations of increased immersion, increased motivation and deeper learning were the most common motivating factors for educators in deciding to use VR in educational settings. Research has shown that in many cases, educators’ intentions of using VR to improve learning outcomes in educational settings are Australasian Journal of Educational Technology, 2021, 37(3). 120 met. For example, several meta-analyses have concluded that VR and computer simulations can have a positive influence on learning outcomes when compared with traditional classroom instructional approaches (Merchant et al., 2014; Smetana & Bell, 2012; Vogel et al., 2006). More recently, other studies have also reported various educational benefits of VR including technical skill acquisition (Gunn et al., 2018), enabling users to repeatedly practise skills and learn from their mistakes including through informative feedback (Papanikolaou et al., 2019) and the development of skills in training (driving a wheelchair) while avoiding the associated risks which are present in the real world (John et al., 2018). VR, immersion and presence In considering the educational benefits of using VR, it is important to discern the different types of VR, which can have different benefits or limitations. These include HMD VR systems, desktop display systems which may include one or more screens (Carlozzi et al., 2013) or cave automatic virtual environments (CAVE) where several walls are covered with display screens (Jensen & Konradsen, 2018). Immersion describes the degree of involvement a user experiences in a simulation (Cheng et al., 2015, p. 233), while presence refers to the “sense of being in the virtual environment” (Slater & Wilbur, 1997, p. 603). A meta-analysis of 115 effect sizes from 83 studies found that immersion has a medium effect size on presence (Cummings & Bailenson, 2016). Additionally, increased levels of user presence have been linked to increased performance (Makransky et al., 2017; Stevens & Kincaid, 2015) and reduced levels of cybersickness (Weech et al., 2019). Studies have reported that HMD VR can have advantages over both desktop VR systems and CAVE systems. Using an HMD was reported to enhance users’ presence compared with using an LCD screen (Kim et al.,, 2014; Stevens & Kincaid, 2015). Compared with desktop VR systems, HMD VR systems have also been found to be rated as more intuitive and natural by users (Santos et al., 2009), provide superior spatial awareness (Krokos et al., 2019), lead to better memory recall (Krokos et al., 2019), have a positive effect on engagement (Buttussi & Chittaro, 2018) and have a positive effect on self-efficacy (Buttussi & Chittaro, 2018), among other outcomes. Within the context of engineering education, Alhalabi (2016) demonstrated that HMD VR was superior to a system known as Corner CAVE System (a version of CAVE where display screens are attached to walls in a room) and consistently led to better results on tests. These findings highlight that there are benefits of HMD VR over desktop VR systems and CAVE VR systems, which can make HMD VR more applicable for certain learning activities. HMD VR systems are especially useful for situations where the user needs to be able to understand and remember information or knowledge which is either spatial or visual in nature (Jensen & Konradsen, 2018). Potential issues with using HMD VR There can be issues using HMD VR. More immersive VR "may overload and distract the learner” (i.e., cognitive overload) (Makransky et al., 2017, p. 225). In addition, using a higher immersive VR technology can lead to lower student performance (Makransky et al., 2017; Richards & Taylor, 2015), although other studies report that using higher immersive VR can lead to increased performance (Alhalabi, 2016; Webster, 2016). Moreover, although HMD VR systems are more immersive than desktop VR systems and can provide a more realistic environment than desktop VR systems, they can be expensive and cumbersome, introducing challenges when student numbers in a cohort are high (Taxen & Naeve, 2002). This is critically important in the context of engineering education. A significant challenge when using VR is that users may suffer from cybersickness. The possibility of cybersickness in VR environments is particularly an issue where a person uses a HMD, and women are more likely to be affected than men (Munafo et al., 2017). For example, the view that is presented to the hands-on operator through the HMD may not match the operator’s real physical actions, and this can lead to the onset of cybersickness (McGill et al., 2017). Moreover, this is not necessarily limited to operators who wear the HMD but also observers who watch a live stream of the operator’s actions on a nearby screen. The operator or observer may be presented with a situation where they seem to be moving through the virtual environment, but they will actually be stationary (LaViola, 2000), which can cause sensory conflict. Videos (or live streams) can also be shaky (Kasahara et al., 2015) or have fast movement (Tran et al.,, 2017), which can induce cybersickness. Indeed, Wanick et al. (2018, p. 9) contended that “in scenarios in which the user looks at immersive omnidirectional videos (ODV) from a first-person perspective, it is Australasian Journal of Educational Technology, 2021, 37(3). 121 possible that users might experience cybersickness since the video was recorded from another person’s [point of view] and then translated to a larger screen”. Learning by observing versus learning by doing Research has investigated the differences in learning which can occur when a person learns by observing someone taking a hands-on role, compared with taking on the hands-on role themselves. One of the fundamental areas where students learn (at least in part) by observation is healthcare education. O’Regan et al. (2016) conducted a systematic review of nine studies (published between 1980 and 2014) within the context of health professional education, which investigated whether healthcare students tended to learn as effectively when observing someone completing a hands-on simulation task (using mannequins), compared with completing the hands-on task themselves. Overall, they found that in five of the nine studies the learning outcomes for observers were similar or better than for students who took on the hands-on role in the simulation. Observers were provided with the important benefit of having an overarching view of the simulation as it took place, although some students disliked the observer role, which they considered to be passive or boring compared with taking on the hands-on role in the simulation. Also in the area of healthcare education, a study that investigated self-reported learning outcomes of observers and hands-on participants during a simulation training (in emergency medicine) found that observers and hands-on participants had comparable results in three of the six learning outcomes (Reime et al., 2017). These studies demonstrate that in certain contexts students can learn effectively either by observing a participant complete a hands-on task or completing the hands-on task themselves. However, it is important to note that these studies were in the area of healthcare education and did not involve the use of VR technology. In a situation where HMD VR is used, observers can only interact with the simulation by observing what the VR operator is doing in the simulation on a nearby screen, unlike a simulation within a medical emergency room where observers can physically see what is happening anywhere in the simulation environment at any time. Challenges of using HMD VR in large classes Learning is facilitated in activities when there are conditions that allow for elaboration of knowledge. Although there are several ways this can be fulfilled in education, one way in which students can do this is by “discussing subject matter to be learned with other students” (Rudduck, 1978, cited by Schmidt, 1983, p. 12). A barrier to using VR in education is access to hardware for large classes. When using HMD VR simulations, educators can design the learning activity so that all members of a group complete the VR simulation independently, then consolidate their learning by discussing the experience after all group members have completed the simulation. However, this creates a problem of logistics and resourcing. Although large-scale laboratories equipped with over 25 headsets have been implemented (Ijaz, Marks, Hartley, Gibbens, & Thomas, 2017), such facilities require significant resources and are not feasible in all institutions. Furthermore, large spaces would be required for VR simulations in which students’ movement is more extensive than rotation. Within Australia, there are a large number of engineering students attending tertiary institutions, and the number of enrolments has steadily increased since 2001. In 2001, there were 46,897 students enrolled in bachelor degrees in the field of engineering and related technologies. By 2010, this number had increased to 61,518, and by 2017, there were 75,767 students enrolled (Kaspura, 2019, p. 13). In addition, the number of students enrolled in master’s degrees by coursework in the field of engineering and related technologies increased from 3799 in 2001 to 9266 during 2010 and 21605 during 2017 (Kaspura, 2019, p. 13). Many institutions now operate units that have over 200 students enrolled (Cash et al., 2017). Larger classes create a requirement for more resources, which can create problems for educators (Cash et al., 2017). Similar problems are also present in the area of healthcare education; Reime et al. (2017) reported that the limited time available to faculty coupled with the occurrence of large student groups have led educators to ask whether students can learn effectively by observing medical simulations instead of taking on an active role in the simulation themselves. Australasian Journal of Educational Technology, 2021, 37(3). 122 Coordinating courses with large numbers of enrolled students creates a challenge where the benefits of VR systems and potential of using them to enhance students’ learning are confounded by the logistics of being able to resource such large classes and providing students the opportunity to operate and use HMD VR systems. To overcome logistical and resourcing issues, this study adapted learning activities in which one group member operated the HMD VR simulation while other group members observed the operator and what the operator was viewing on a nearby screen. Students were able to elaborate on their knowledge and learning through discussion of material with other students in a group discussion toward the end of the workshop. This created an important question – whether, on a personal level, students’ learning outcomes are negatively influenced by observing rather than operating the HMD VR simulation. Contribution to the literature No previous studies have substantially compared the experience of operating an HMD VR simulation with observing an HMD VR simulation. This study investigated the feasibility of teaching using HMD VR technology in large classes, by evaluating whether students are able to effectively meet specified learning outcomes when either operating the HMD VR simulation or observing both another group member operating the simulation and also the operator’s headset view. Research question The following question was addressed: How do learning outcomes from observing another student operating an HMD VR simulation and observing the operator’s first-person perspective on a nearby monitor compare with wearing the headset and operating the simulation? Research method To answer the research question, a quasi-experimental design was adopted, and qualitative and quantitative data analysis techniques were used. Below is an outline of the method:  A workshop was held where participants either operated an HMD VR simulation or observed another operating the simulation along with the operator’s headset view. Participants volunteered to be one of a limited number of operators of the HMD VR. The study was conducted within a class that was part of a credit-bearing unit at a tertiary institution. The study provided an accurate representation of how the simulation would be used in the classroom and in engineering practice.  At specified stages of the workshop, participants completed worksheet templates relating to the learning outcomes for the workshop. The worksheets were collected at the end of the class.  The content of the participants’ written work on the templates was analysed qualitatively. This identified the breadth and depth of information that participants had written relating to the learning outcomes. Quantitative data analysis was subsequently used to compare the breadth and depth of information written by participants who operated HMD VR equipment with that written by participants who observed another using the equipment. Ethics approval for the study was granted by The University of Western Australia Human Research Ethics Committee. Description of the workshop Participants and setting The workshop took place in Semester 1 of 2018. The setting for the study was a 2-hour workshop class in an engineering design project unit. Participants of the study (Table 1) were postgraduate engineering students (predominantly 5th-year students) from the discipline of mechanical engineering. Ten classes completed the workshop. A total of 22 participants did not fully complete the demographic questionnaire and were subsequently excluded from the analysis presented in this study, leaving 117 participants. Australasian Journal of Educational Technology, 2021, 37(3). 123 Table 1 Participant demographic information Demographic characteristic Value N % Participants - 117 100.0 Gender Female 17 14.5 Male 100 85.5 Speak English at home Yes 92 78.6 No 25 21.4 At the start of the workshop, students elected to voluntarily participate in the research project by completing a consent form, returned to the researchers at the end of the class. Students were able to complete the workshop without participating in the research project. The intended learning outcome was that on completion of the workshop students would be able to consider safety systematically and authentically as in engineering practice. Safety in design is a critical topic for students to learn in engineering education (Foley et al., 2016). To provide access to an authentic engineering practice workspace, a VR simulation was used. As discussed above, VR is especially applicable in contexts such as this where access to real engineering work sites may not be feasible or may be hazardous, and VR can effectively emulate the setting through immersive simulations (Freina & Ott, 2015; Garrett & McMahon, 2013; Slater & Sanchez-Vives, 2016). The design of the workshop conformed to conditions that facilitate learning, as reported by Schmidt (1983). Students used a tool (described in the Workshop procedure section below) that is widely used in engineering practice to systematically consider safety and applied it under conditions that resembled an application in engineering practice, through use of a VR simulation. Students were also able to elaborate on their knowledge, in this case through conditions that allowed for subject matter to be learned with other students through group discussion. Safety in design scenario During 2010, Worksafe Queensland issued a safety alert regarding operation of vehicle loading cranes (VLCs; Worksafe Queensland, 2010). This followed workplace incidents during 2009 and 2010 which lead to two fatalities. Reasoning for the incidents was attributed to incorrect operation of the VLC, based on manufacturer instructions. Several design factors were highlighted as being contributing factors and of significant safety concern. One of the major issues was that the position of the control panel located on the side of the vehicle made it difficult to safely operate the crane arm, which can be behind the operator. Therefore, this scenario was simulated in VR. Spatial awareness was highly important for the scenario. HMD was therefore used, because HMD VR systems provide superior spatial awareness over desktop VR systems (Krokos et al., 2019). Hardware Several HMD sets designed for VR simulations have recently become available on the commercial market (John et al., 2018). This study used the HTC Vive headset and controllers, which enabled full interactivity and position tracking with the VR simulation. Each of the HTC Vive sets was connected to a computer with a high-power graphical processing unit. Located near each set of HMD VR equipment was a large screen, which duplicated the view seen by the operator of the HMD VR equipment in their headset. While one group member used the HMD VR equipment, other members of the group observed both the student operating the simulation and also the operator’s headset view in real time. Australasian Journal of Educational Technology, 2021, 37(3). 124 Software The VR simulation presented the user with an implementation of the situation described in the safety alert provided by Worksafe Queensland (2010). Simulated was a VLC with a lever control system that was poorly designed and reflected the authentic safety hazards described in the safety alert (Figures 1–4). The simulation was designed for students to obtain an understanding of the crane’s design and to identify hazards. The HMD VR simulation described here has also been used in previous research studies (Male et al., 2018). Figure 1. View of VR simulation depicting the scenario where the crane operator is controlling the crane arm Figure 2. View of VR simulation depicting location surrounding the vehicle loading crane Australasian Journal of Educational Technology, 2021, 37(3). 125 Figure 3. First-person perspective of HMD VR equipment – operator looking at the loading crane controls Figure 4. First-person perspective of HMD VR equipment – operator looking towards the right The simulation was built using Unity, a game engine software, which has been used in the development of previous simulations (Ibanez & Delgado-Kloos, 2018). Unity supports many VR hardware options, including desktop platforms and HMDs (John et al., 2018), making it suitable for this application. Tasks to be completed in the VR simulation by the operator In the VR simulation, the participants using the HMD took on the role of a VLC operator. In the scenario, the operator stood to the side of a VLC (Figure 1). Behind the operator high off the ground were powerlines, which were a hazard that had to be avoided (Figure 2). The operator had a first-person view of the simulation, in front of them was a control panel attached to the left side of the VLC (Figure 3). The five levers on the control panel were used to extend, retract and rotate each section of the crane arm. The operator was able to look freely around the simulation in all directions (Figure 4 shows the operator looking right). When the operator interacted with any of the five control panel levers using the HTC Vive controller, that lever was highlighted yellow in the simulation to indicate to the operator that they had connected with the lever. By holding down the button on the back of the HTC Vive controller when the lever was highlighted yellow, the operator could move the lever up or down. This caused one section of the crane arm to extend, retract or rotate. The levers were intentionally not labelled, because this was an authentic safety hazard that Australasian Journal of Educational Technology, 2021, 37(3). 126 was present in real VLCs and which students were expected to identify. The operator was therefore likely to be unsure what each lever did, and this reflected information in the safety alert by Worksafe Queensland (2010). The operator was required to carry out two tasks to complete the simulation, after which the simulation would reset. First, the operator was required to use the crane arm to pick up the generator located on the back of the vehicle (Figure 4). After picking up the generator, the operator then needed to carefully place the generator in the blue square which appeared on the ground behind them (partially completed in Figure 2 – blue square not shown) without dropping the generator or hitting other objects or themselves. Once complete, the blue square would disappear to indicate to the operator that they had completed this section of the simulation. The operator was then required to retract all the crane arm sections and stow the crane arm away behind the vehicle cabin in its typical transport position. A message appeared on-screen informing the operator that they had completed the simulation, and the simulation reset. Because the operator was carrying out a careful, controlled task while standing at a fixed point near the controller, there was no quick movement in the simulation. This reflects real-world usage of a VLC. Additionally, a message appeared on-screen if the operator caused a safety problem, such as dropping the generator with the crane arm, hitting the power lines with the crane arm or hitting themselves with the crane arm. The simulation would then reset. Workshop procedure Prior to the workshop Construction Hazard Analysis Implementation Review (CHAIR) is a hazard identification and analysis process widely used in engineering industry (Australian Safety and Compensation Council, 2006, pp. 28– 29) and was selected as the authentic industry process for students to use to assist them to identify safety hazards. Before the workshop class, students were provided with an assessed homework task which focused on learning how to apply the CHAIR method. CHAIR involves using generic guidewords as prompts to identify and analyse hazards. The homework did not form part of the research study. Students were aware that VR equipment would be used during the workshop class. Workshop overview As noted above, the design of the workshop conformed to conditions that facilitate learning, aligning with the framework set out by Schmidt (1983). The phases of the workshop are described in detail below and outlined in Figure 5. Students first individually identified hazards in Phase 1 and Phase 3 (described below). Students then elaborated on their knowledge through discussion with peers (as suggested by Schmidt, 1983) in Phase 4. Figure 5. Workshop procedure Australasian Journal of Educational Technology, 2021, 37(3). 127 Group allocation and working area Classes of approximately 15 students completed each workshop, and three sets of HMD VR equipment were available for each class. Students were allocated into groups of approximately 4–6 members each. Each group had an area of 3 x 3 metres marked out where the VR simulation operator was able to move. This space was necessary for providing the spatial awareness that was important for the activity. Introduction phase Participants were provided with several resources: a one-page introduction to the CHAIR process, which included an example of how to apply the technique; an A3-size template which students were able to use when completing the CHAIR process; and a red, blue and black pen used for recording identified hazards on the CHAIR template at different phases of the workshop. The workshop facilitators explained each column of the CHAIR template (guideword, risk issue, causes, consequences, strategies, actions) using the example on the one-page introduction. During subsequent phases of the workshop, as described below, participants used the CHAIR process to identify safety hazards. Participants were instructed to write down the hazards they identified during each phase using different coloured pens. Red was used during the first phase, blue during the second and third phases and black during the fourth phase. The colours enabled researchers to categorise students’ responses by workshop phase. This enabled analysis to determine whether participants identified additional hazards during different phases of the workshop (i.e., performed more effectively). Phase 1 Students were provided with pictures of the scenario (screenshots from the VR simulation). Parts of the VLC were labelled, and an enlarged image of the control panel was positioned on the vehicle. While presented with the still images, participants were requested to use the CHAIR process to individually identify safety hazards and to write them on the CHAIR template using a red pen. Participants did not interact with each other during this phase of the workshop. Phase 2 During Phase 2, one student from each group operated the HMD VR simulation, while the other group members observed the student operator and the HMD view on the screen. Groups decided among themselves which member operated the HMD VR simulation. In some groups where time permitted, more than one student operated the HMD VR simulation, meaning some participants both used the VR equipment and observed others use the equipment. The study provided an accurate representation of how the simulation would be used in a class, where students would be likely to identify the user among themselves. The total number of participants who used each method is shown in Table 2. Participants were able to interact with each other during Phase 2 of the workshop, although they were not specifically requested to do so. A requirement of the ethics approval for the study was that all students were provided with the opportunity to operate the simulation, should they choose to do so. Students who did not have a chance during the formal workshop were able to operate the simulation afterwards. Table 2 Methods of participation during the workshop Participation method N % Observed use of HMD VR equipment 84 71.8 Operated HMD VR equipment 33 28.2 Phase 3 After observing a student operating the HMD VR equipment or operating the HMD VR themselves, students considered what they had observed, identified new hazards and wrote them on the CHAIR template Australasian Journal of Educational Technology, 2021, 37(3). 128 using a blue pen. Participants did not interact with each other during Phase 3 of the workshop and recorded identified hazards on the template independently. Phase 4 During Phase 4 of the workshop, students engaged in group discussion using the CHAIR technique, to try to identify further safety hazards. Participants wrote down identified hazards on the CHAIR template using a black pen. Data analysis Identification of themes in student worksheets using content analysis Data gathered during Phases 1 and 3 (Figure 5) were analysed to address the research question. The completed worksheet templates were analysed to understand the safety hazards that participants had identified in Phases 1 and 3. Through content analysis, a set of 10 themes were identified among the safety hazards recorded by participants:  The range of motion of the crane arm [causes collision with people and objects in the environment, or with the vehicle itself].  Dangerous positioning of the control panel [places operator at risk of collision with crane arm].  Dangerous positioning of the control panel [makes it difficult to simultaneously operate the controls and view the crane].  Insufficient labelling, sensitivity, protection or poor design of the control panel [causes undesired crane arm movement].  Overloading of the crane [causes damage or failure of the crane arm].  Overloading of the crane [causes the vehicle to become unbalanced].  Inadequate attachment of the load or swinging of the load [causes damage to truck, environment, or operator, or causes load to fall].  The student’s response showed a lack of understanding of the exercise or was not a valid risk.  The student’s response could not be interpreted or was too vague.  The student identified valid risks that could be interpreted but they were not themes 1–7. The first seven of these themes refer to hazards that students identified from the case study. Themes 1 to 4 were related to the design of the crane (position and design of control panel, range of motion of the crane arm), and themes 5 to 7 were associated with how the crane is used (overloading, poor attachment of load). Each of these hazards was broken down into a cause and a consequence [in brackets]. Themes 8 to 10 enabled the categorisation of entries that did not identify any of the hazards of themes 1 to 7. As such, every entry could be allocated to one particular theme. Theme 9 included responses that were indecipherable due to poor handwriting or broken English (e.g., “One the many load, then chances the failure of controller”) as well as entries that were too broad to be an acceptable explanation of the risk (e.g., “A heavy load”). There were some valid hazards that were identified by only one or two students, such as ergonomic issues. Such entries were combined under theme 10. Evaluation of the student responses Each entry on the worksheet was individually analysed and assigned to one of the ten themes. The phase in which each entry had been written was also recorded based upon the colour which had been used (Figure 5). Entries which fell under categories 8, 9 or 10 were excluded from further analysis. Entries were then analysed to see whether the student had demonstrated limited understanding or high understanding of the safety hazard. This assessment was made by breaking down the hazards of themes 1 to 7 into their causes and consequences. An entry that clearly referenced both the cause and the consequence of the hazard was considered to demonstrate a comprehensive understanding of the theme. Australasian Journal of Educational Technology, 2021, 37(3). 129 Table 3 Samples of student responses demonstrating limited and comprehensive understanding of themes 1–7 Theme no. Sample response demonstrating limited understanding Sample response demonstrating comprehensive understanding 1 Risk issue: Interference with overheads Causes: Improper training Consequences: [no response] Risk issue: Range of motion of crane not limited Causes: [no response] Consequences: Collision of arm with power lines, etc. 2 Risk issue: Control panel within area of crane operation Causes: [no response] Consequences: [no response] Risk issue: Operator might be struck by the beam Causes: the control panel is very close to the crane when moving Consequences: Crane can strike the operator 3 Risk issue: Operator looking at crane and control panel Causes: Control panel being low down, crane being higher Consequences: Muscle strain Risk Issue: Operator cannot see crane movement whilst pushing buttons leads to unwanted positioning Causes: Control panel position makes it difficult for operator to view crane Consequences: Crane may come in contact with someone or something 4 Risk issue: Poor labelling of control panels Causes: [no response] Consequences: [no response] Risk issue: Wrong or unwanted crane movement Causes: Poor control panel labelling Consequences: [no response] 5 Risk issue: Overload mechanical Causes: Overload Consequences: [no response] Risk issue: The load of the object could cause stress of failure of instrument Causes: Load greater than rated causing mechanical failure Consequences: [no response] 6 Risk issue: Small stabiliser legs, also not supporting centre of gravity if telescoping arms extended Causes: [no response] Consequences: Damage Risk issue: Crane toppling over Causes: Load lifted is heavier than crane’s capability Consequences: Crane uprooted & toppled over, possibly damaged 7 Risk issue: Dropped loads Causes: If winch cable snaps, a heavy load could drop Consequences: Crush injuries Risk issue: A large item may have momentum in swinging fatiguing or bending hook joint Causes:[no response] Consequences: Injury, damage or fatality Entries that vaguely alluded to the cause and consequence or those that identified a correct cause but did not attribute a correct consequence – or vice versa – were deemed as demonstrating a limited understanding of the theme. For themes with multiple causes or consequences (such as themes 4 and 7), it was sufficient for an entry to clearly identify just one of these causes or consequences to be classified as showing a comprehensive understanding. Table 3 lists samples of both classes of response for each of themes 1 to 7. Identification of accumulated understanding during the workshop The distinction between limited and comprehensive understanding of themes 1 to 7 provided a means of quantifying the change in student understanding of these themes throughout the phases of the workshop. For example, if a student did not identify theme 1 during the viewing of the still picture, they would be allocated a score of 0 for that theme at Phase 1 of the workshop. If an entry written after the VR simulation then demonstrated a limited understanding of theme 1, the student would receive a score of 1 for that theme Australasian Journal of Educational Technology, 2021, 37(3). 130 at this phase of the workshop. As such, the student’s understanding of this theme could be tracked at each phase of the workshop. If a student was evaluated to highly understand a specific theme right after seeing the photo, this demonstrated that the student would also be likely to highly comprehend that specific theme after the VR simulation. As a result, when a student identified a theme at a certain phase of the workshop, the given evaluation score was carried through to subsequent phases of the workshop (if it was higher). For example, if a participant highly comprehended a specific theme after viewing the still photo but did not write this again during the group discussion, the participant’s response was also evaluated as demonstrating high comprehension of the theme during the VR simulation. If a participant had shown only limited understanding of the concept theme (i.e., categorised as a 1), this entry was excluded from further analysis. Only where a student had shown high comprehension of a theme (i.e., categorised as a 2) was the entry included in the further analysis. Using the accumulative method of scoring described above, it was established for each participant how many themes they had highly comprehended at each relevant phase of the workshop (after the still photo and after the VR simulation). As there were seven themes that were identified, this resulted in a score out of 7 for each of these phases of the workshop. From this, the performance of the cohort at each phase of workshop was established. Results Data were analysed using the R statistical environment (R Core Team, 2019). The data contain the following variables:  Demographic information o Gender of the student (male of female) o Whether the student spoke English at home (yes or no)  Use of VR simulation – whether the student operated the VR simulation (yes) or was an observer (no). Statistical analysis was conducted to determine whether any of the above three variables affected the mean number of hazards highly comprehended (NHHC) at Phases 1 and 3, that is, before and after the VR simulation. Table 4 shows some summary statistics for the two phases, namely the overall mean NHHC, as well as that for each group of students. The mean NHHC at each phase was similar for students who had observed and those who had operated the VR simulation, although at both phases, the mean and standard deviation were lower for the group that operated the simulation. Table 4 Number of hazards highly comprehended by whether students operated the HMD VR equipment or observed another using HMD VR equipment (out of 7) Number of safety hazards highly comprehended Group No. Phase 1. After still image (before VR simulation) Phase 3. After VR simulation M SD M SD Observers 84 1.79 1.13 2.75 1.46 Operators 33 1.67 0.92 2.63 1.19 Overall 117 1.75 1.07 2.72 1.39 Australasian Journal of Educational Technology, 2021, 37(3). 131 Because the dependent variable is count data, a log-linear Poisson regression model was fitted to the data. A feature of the Poisson distribution is that its mean and variance are equal. The maximum counts of 4 and 6 for Phases 1 and 3 respectively cause the data to be underdispersed; that is, the variance is smaller than the mean. To adjust for this, quasi-Poisson log-linear models were fitted to the data (Zeileis et al., 2009). Two quasi-Poisson log-linear models were fitted, one each for the counts before and after the simulation. In both models, there was no significant difference in the mean NHHC identified between the students who observed and those who operated the VR simulation (p > 0.05). Additionally, there was no significant difference in the mean NHHCH between male and female students. In contrast, compared with those who did not speak English at home, students who did speak English at home identified on average 1.4 times the NHHC before the VR simulation and 1.5 times the NHHC after the VR simulation. A one sample t test showed that the mean difference between the NHHC before and after the VR simulation was significantly greater than zero (p < 0.001), and the average increase was 0.97 hazards highly comprehended. Discussion Benefits of using HMD VR This study focused on evaluating whether students achieved the same learning outcomes (the ability to identify safety hazards in this case) when they either operated or observed the use of a HMD VR simulation. Although it was not the area of focus in this study, it was nonetheless important to consider how the use of VR technology in itself may have helped students during the learning activity and compare this to the existing literature. In other words, did incorporation of the VR simulation add any benefits to the learning activity compared with if the VR simulation had not been used? It is possible that an alternative method of providing students with virtual access to the worksite (such as a video) may have been as effective. However, using a video would not be authentic. The students were learning about a process to identify hazards during design. Videos can be created only after construction or at least simulation. Whether video of a simulation or another form of active learning would be equally effective is outside the scope of this study and a suitable topic for further research. Out of the seven applicable types of safety hazards which were identified in the analysis, participants identified an average of only 2.72 highly comprehended safety hazards. This may seem low. However, the task was intentionally challenging. The findings of this study support meta-analyses which have concluded that the use of VR and computer simulations can have a positive influence on learning outcomes (Merchant et al., 2014; Smetana & Bell, 2012; Vogel et al., 2006). Reflecting upon the results shown in Table 4, it was found that the average number of hazards which had been identified significantly increased between before the VR simulation (and after using the still photo) (Phase 1) and after the VR simulation (Phase 3). The VR simulation significantly improved the NHHC as indicated by the t test reported in the Results. Overall, the findings are consistent with those in which the use of VR simulations has been found to be beneficial in educational contexts. Key findings and implications for practice Students’ ability to address the learning outcome and identify safety hazards during the workshop was not significantly different based upon whether students operated or observed the HMD VR simulation (Table 4). The baseline performance of students who observed and those who operated was similar before the VR simulation (1.78 to 1.67 hazards identified on average), and the performance of both groups increased to similar levels after the VR simulation (2.75 to 2.63 hazards identified on average). The difference between the performances of students who had observed and operated the VR simulation was statistically insignificant before the VR simulation and remained statistically insignificant after the VR simulation. Although the difference between the groups was not statistically significant, it is possible that the HMD operators may have suffered from cognitive overload (Makransky et al., 2017), which may have reduced their performance in the task of recording identified hazards. Australasian Journal of Educational Technology, 2021, 37(3). 132 Addressing the question “How do learning outcomes from observing another student operating an HMD VR simulation and observing the operator’s first-person perspective on a nearby monitor, compare with wearing the headset and operating the simulation?”, the similarity of performance suggests that students benefited from the VR simulation similarly regardless of whether they operated the HMD VR equipment or observed the interactions of another student who operated the equipment. Although previous research has compared various aspects about different types of VR simulations, such as desktop VR simulations and HMD VR simulations (e.g., Buttussi & Chittaro, 2018; Kim et al., 2014; Krokos et al., 2019; Li & Giudice 2013; Santos et al., 2009) or CAVE VR simulations and HMD VR simulations (e.g., Alhalabi, 2016; Kim et al., 2014), this appears to be the first study to evaluate the differences between operating and observing HMD VR simulations. The result supports the conclusion that the benefits of HMD VR simulations in certain educational contexts are not limited to students who operate the equipment and are also applicable to others who can observe the actions of the operator within the real world and interactions of the operator with the VR simulation using a nearby monitor. The result of this study indicates that students are not necessarily disadvantaged by observing rather than operating the simulation. The finding is important for educators seeking to design learning activities that are equitable and inclusive. It is possible to provide equitable learning opportunities when using VR simulations, even in student groups that include students who do not wish to or are unable to operate a simulation. This study has important implications for educators who are seeking to use the potential benefits of HMD VR simulation technology (such as providing access to worksites which would otherwise be too dangerous) with a large number of students but face the challenge of resource limitations. It is not necessary to engage every student individually in completing the HMD VR simulation to obtain relevant benefits, because similar outcomes can be achieved by using group situations where one group member uses the equipment and other group members observe the operator’s interaction with the HMD VR simulation. The finding of the study is similar to those of studies from the area of healthcare education which concluded that observers can often learn just as well in (non-VR) simulations, if not in some cases better, than participants who take on the hands-on role of completing the simulation (O’Regan et al., 2016; Reime et al., 2017). Consistent with the findings of these studies, this study found that students can learn effectively either by observing a participant complete a hands-on task or completing the hands-on task themselves. Interestingly, unlike other studies which reported that observers may suffer from motion sickness due to observing a first-person point-of-view video (Kasahara et al., 2015; LaViola, 2000; Tran et al., 2017; Wanick et al., 2018), students did not report this problem during the workshop. The screen was only 24 inches in size, suggesting that it is unlikely that the simulation would have been very immersive or induced a large sense of presence (Cummings & Bailenson, 2016). Because presence is negatively related to cybersickness (Weech et al., 2019), the experience in this study was surprising. A possible explanation for the lack of cybersickness reported by participants was that (as described above) the operator was carrying out a careful, controlled task which reflects the real-world usage of a VLC. The result of this was that there was no quick movement in the simulation, which can cause the onset of cybersickness (Tran et al., 2017). Limitations and future work Two potential research design limitations are considered. First, due to the voluntary nature of using the HMD (to provide an accurate representation which reflects how it would work in the real classroom, as described above), it is possible that participants who used the headset had fewer reservations about using new technology or were more comfortable using technology than the students who observed the simulation. This could be seen as a limitation. However, the workshop design, and any bias, is authentic to how the workshop can be used in class. The opportunity to enable students to self-select to observe rather than operate the simulation without an expectation that they will be disadvantaged by this choice is a possible benefit of the finding of the study. Second, participants’ spatial understanding was not evaluated prior to using the VR simulation. It is possible that spatial understanding was important for being able to identify safety hazards and that there may be bias in the sample of those participants who used the VR headset. Future research could seek to evaluate whether Australasian Journal of Educational Technology, 2021, 37(3). 133 participants’ spatial understanding is important for identifying hazards both when observing or operating a VR simulation. There are possibilities for enhancing the workshop. Despite the apparent benefit that the VR simulation provided to participants, the number of hazards identified by the participants appeared to be relatively low. The task was intentionally challenging. Although the learning activity was beneficial to students, further enhancement of learning may be possible, and this possibility will be considered in future studies. The reasons for different results between students who did and did not speak English at home are unknown. It is possible that students who did not speak English at home had difficulty articulating their understanding of hazards. Therefore, it is difficult to determine whether there was a difference in comprehension of the hazards between students who did and did not speak English at home. Finally, this study was conducted in a master’s engineering course at a research-intensive university in Australia. Generalisability to other cohorts is likely but cannot be assumed. It is expected that the result would be likely to generalise across year levels and geographic location but less likely to generalise across disciplines of study. The simulation and implementation involved a physical engineering system and spatial visualisation, the nature of which engineering students are relatively familiar. Conclusion A barrier to using HMD VR in education in large classes is access to hardware. Finding ways to reduce this barrier is essential for enhancing educators’ options regarding adoption of HMD VR in tertiary education. The purpose of this study was to establish how students’ changes when operating a HMD VR simulation compared with observing the simulation interaction on a nearby screen. Using a quasi-experimental research design, a workshop was created that engaged students in a safety hazard identification activity. Students either operated the HMD VR simulation or observed. The performance of operators and observers was compared before and after the VR simulation was completed. It was found that engineering students’ performance during the safety hazard identification activity was similar and was not negatively affected by observing the interaction of the HMD VR operator and simulation on a nearby screen rather than operating the VR simulation. The result suggests that educators could use HMD VR simulations in classes with a large enrolment, by reducing the need for investment and management of a large number of sets of HMD VR equipment by requiring only one student to operate the simulation per group of approximately five students. Acknowledgements The authors gratefully acknowledge the students who agreed to participate in the research. The authors acknowledge Safe Work Australia and the regulatory bodies that assisted with identification of the design to be simulated. Development, embedding, and testing of the simulation was supported by The University of Western Australia Education Enhancement Unit. Support for the overarching project Virtual Work Integrated Learning for Engineering Students has been provided by the Australian Government Department of Education, The University of Western Australia, CingleVue International, Curtin University, Engineers Australia, the Australian Council of Engineering Deans, The University of Queensland and Murdoch University. The views in this paper do not necessarily reflect the views of the Australian Government Department of Education or project partners. References Alhalabi, W. (2016). Virtual reality systems enhance students’ achievements in engineering education. Behaviour & Information Technology, 35(11), 919–925. https://doi.org/10.1080/0144929X.2016.1212931 Australian Safety and Compensation Council. (2006). Safe design for engineering students: An educational resource for undergraduate engineering students. Commonwealth of Australia. Buttussi, F., & Chittaro, L. (2018). Effects of different types of virtual reality display on presence and learning in a safety training scenario. IEEE Transactions on Visualization and Computer Graphics, 24(2), 1063–1076. https://doi.org/10.1109/TVCG.2017.2653117 https://doi.org/10.1080/0144929X.2016.1212931 https://doi.org/10.1109/TVCG.2017.2653117 Australasian Journal of Educational Technology, 2021, 37(3). 134 Carlozzi, N. E., Gade, V., Rizzo, A. S., & Tulsky, D. S. (2013). Using virtual reality driving simulators in persons with spinal cord injury: Three screen display versus head mounted display. Disability and Rehabilitation: Assistive Technology, 8(2), 176–180. https://doi.org/10.3109/17483107.2012.699990 Cash, C. B., Letargo, J., Graether, S. P., & Jacobs, S. R. (2017). An analysis of the perceptions and resources of large university classes. CBE – Life Sciences Education, 16(2), 1–12. https://doi.org/10.1187/cbe.16-01-0004 Cheng, M. T., She, H. C., & Annetta, L. A. (2015). Game immersion experience: its hierarchical structure and impact on game‐based science learning. Journal of Computer Assisted Learning, 31(3), 232–253. https://doi.org/10.1111/jcal.12066 Chittaro, L., & Buttussi, F. (2015). Assessing knowledge retention of an immersive serious game vs. a traditional education method in aviation safety. IEEE Transactions on Visualization and Computer Graphics, 21(4), 529–538. https://doi.org/10.1109/TVCG.2015.2391853 Cummings, J. J., & Bailenson, J. N. (2016). How immersive is enough? A meta-analysis of the effect of immersive technology on user presence. Media Psychology, 19(2), 272–309. https://doi.org/10.1080/15213269.2015.1015740 Foley, B., Howard, P., Toft, Y., & Hurd, M. (2016). Increasing safe design practice within the engineering curriculum. In Proceedings of the 27th Annual Conference of the Australasian Association for Engineering Education (pp. 259–265). Australasian Association for Engineering Education. https://aaee.net.au/wp-content/uploads/2018/10/AAEE2016-Foley_Howard_Toft_et_al- Increase_safe_design_practice.pdf Freina, L., & Ott, M. (2015). A literature review on immersive virtual reality in education: State of the art and perspectives. Rethinking Education by Leveraging the Elearning Pillar of the Digital Agenda for Europe! Vol. I(1), 133–141. https://doi.org/10.12753/2066-026x-15-020 Garrett, M., & McMahon, M. (2013). Indirect measures of learning transfer between real and virtual environments. Australasian Journal of Educational Technology, 29(6), 806–822. https://doi.org/10.14742/ajet.445 Gunn, T., Jones, L., Bridge, P., Rowntree, P., & Nissen, L. (2018). The use of virtual reality simulation to improve technical skill in the undergraduate medical imaging student. Interactive Learning Environments, 26(5), 613–620. https://doi.org/10.1080/10494820.2017.1374981 Herrera, F., Bailenson, J., Weisz, E., Ogle, E., & Zaki, J. (2018). Building long-term empathy: A large- scale comparison of traditional and virtual reality perspective-taking. PloS one, 13(10), Article e0204494. https://doi.org/10.1371/journal.pone.0204494 Huang, H.-M., Rauch, U., & Liaw, S.-S. (2010). Investigating learners’ attitudes toward virtual reality learning environments: Based on a constructivist approach. Computers & Education, 55(3), 1171– 1182. https://doi.org/10.1016/j.compedu.2010.05.014 Ibanez, M. B., & Delgado-Kloos, C. (2018). Augmented reality for STEM learning: A systematic review. Computers & Education, 123, 109–123. https://doi.org/10.1016/j.compedu.2018.05.002 Ijaz, K., Marks, B., Hartley, T., Gibbens, P., & Thomas, J. (2017). The immersive learning laboratory: Employing virtual reality technology in teaching. In N. Huda, D. Inglis, N. Tse, & G. Town (Eds.), Proceedings of the 28th Annual Conference of the Australasian Association for Engineering Education (pp. 974–981). Australasian Association for Engineering Education. https://aaee.net.au/wp- content/uploads/2018/09/AAEE2017-Thomas_Ijaz_Marks_et_al- Employing_virtual_reality_technology_in_teaching.pdf Jensen, L., & Konradsen, F. (2018). A review of the use of virtual reality head-mounted displays in education and training. Education and Information Technologies, 23(4), 1515–1529. https://doi.org/10.1007/s10639-017-9676-0 John, N. W., Pop, S. R., Day, T. W., Ritsos, P. D., & Headleand, C. J. (2018). The implementation and validation of a virtual environment for training powered wheelchair manoeuvres. IEEE Transactions on Visualization and Computer Graphics, 24(5), 1867–1878. https://doi.org/10.1109/TVCG.2017.2700273 Kasahara, S., Nagai, S., & Rekimoto, J. (2015). First person omnidirectional video: System design and implications for immersive experience. In D. Greets (Ed.), Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video (pp. 33–42). Association for Computing Machinery. https://doi.org/10.1145/2745197 Kaspura, A. (2019). Australia’s next generation of engineers: University statistics for engineering. Engineers Australia. https://bit.ly/2GeU0mg https://doi.org/10.3109/17483107.2012.699990 https://doi.org/10.1187/cbe.16-01-0004 https://doi.org/10.1111/jcal.12066 https://doi.org/10.1109/TVCG.2015.2391853 https://doi.org/10.1080/15213269.2015.1015740 https://aaee.net.au/wp-content/uploads/2018/10/AAEE2016-Foley_Howard_Toft_et_al-Increase_safe_design_practice.pdf https://aaee.net.au/wp-content/uploads/2018/10/AAEE2016-Foley_Howard_Toft_et_al-Increase_safe_design_practice.pdf https://doi.org/10.12753/2066-026x-15-020 https://doi.org/10.14742/ajet.445 https://doi.org/10.1080/10494820.2017.1374981 https://doi.org/10.1371/journal.pone.0204494 https://doi.org/10.1016/j.compedu.2010.05.014 https://doi.org/10.1016/j.compedu.2018.05.002 https://aaee.net.au/wp-content/uploads/2018/09/AAEE2017-Thomas_Ijaz_Marks_et_al-Employing_virtual_reality_technology_in_teaching.pdf https://aaee.net.au/wp-content/uploads/2018/09/AAEE2017-Thomas_Ijaz_Marks_et_al-Employing_virtual_reality_technology_in_teaching.pdf https://aaee.net.au/wp-content/uploads/2018/09/AAEE2017-Thomas_Ijaz_Marks_et_al-Employing_virtual_reality_technology_in_teaching.pdf https://doi.org/10.1007/s10639-017-9676-0 https://doi.org/10.1109/TVCG.2017.2700273 https://doi.org/10.1145/2745197 https://bit.ly/2GeU0mg Australasian Journal of Educational Technology, 2021, 37(3). 135 Kavanagh, S., Luxton-Reilly, A., Wuensche, B., & Plimmer, B. (2017). A systematic review of virtual reality in education. Themes in Science and Technology Education, 10(2), 85–119. http://earthlab.uoi.gr/theste/index.php/theste/article/view/241 Kim, K., Rosenthal, M. Z., Zielinski, D. J., & Brady, R. (2014). Effects of virtual environment platforms on emotional responses. Computer Methods and Programs in Biomedicine, 113(3), 882–893. https://doi.org/10.1016/j.cmpb.2013.12.024 Krokos, E., Plaisant, C., & Varshney, A. (2019). Virtual memory palaces: immersion aids recall. Virtual Reality, 23(1), 1–15. https://doi.org/10.1007/s10055-018-0346-3 LaViola, J. J. (2000). A discussion of cybersickness in virtual environments. ACM SIGCHI Bulletin, 32(1), 47–56. https://doi.org/10.1145/333329.333344 Li, H., & Giudice, N. A. (2013). The effects of immersion and body-based rotation on learning multi- level indoor virtual environments. In M Tomko, S. Bell, & K. Li (Eds.), Proceedings of the Fifth ACM SIGSPATIAL International Workshop on Indoor Spatial Awareness (pp. 8–15). Association for Computing Machinery. https://doi.org/10.1145/2533810 Makransky, G., Terkildsen, T. S., & Mayer, R. E. (2017). Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learning and Instruction, 60, 225–236. https://doi.org/10.1016/j.learninstruc.2017.12.007 Male, S. A., Kenworthy, P., Hassan, G. M., Guzzomi, A., Van der Veen, T., & French, T. (2018). Teaching safety in design in large classes using VR. In Proceedings of the 29th Australasian Association for Engineering Education Conference 2018 (pp. 459–465). Australasian Association for Engineering Education. McGill, M., Ng, A., & Brewster, S. (2017). I am the passenger: how visual motion cues can influence sickness for in-car VR. In G. Mark (Ed.), Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 5655–5668). Association for Computing Machinery. https://doi.org/10.1145/3025453 Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta- analysis. Computers & Education, 70, 29–40. https://doi.org/10.1016/j.compedu.2013.07.033 Munafo, J., Diedrick, M., & Stoffregen, T. A. (2017). The virtual reality head-mounted display Oculus Rift induces motion sickness and is sexist in its effects. Experimental Brain Research, 235(3), 889– 901. https://doi.org/10.1007/s00221-016-4846-7 O’Regan, S., Molloy, E., Watterson, L., & Nestel, D. (2016). Observer roles that optimise learning in healthcare simulation education: a systematic review. Advances in Simulation, 1(1), 1–10. https://doi.org/10.1186/s41077-015-0004-8 Papanikolaou, I. G., Haidopoulos, D., Paschopoulos, M., Chatzipapas, I., Loutradis, D., & Vlahos, N. F. (2019). Changing the way we train surgeons in the 21th century: A narrative comparative review focused on box trainers and virtual reality simulators. European Journal of Obstetrics & Gynecology and Reproductive Biology, 235, 13–18. https://doi.org/10.1016/j.ejogrb.2019.01.016 Reime, M. H., Johnsgaard, T., Kvam, F. I., Aarflot, M., Engeberg, J. M., Breivik, M., & Brattebø, G. (2017). Learning by viewing versus learning by doing: A comparative study of observer and participant experiences during an interprofessional simulation training. Journal of Interprofessional Care, 31(1), 51–58. https://doi.org/10.1080/13561820.2016.1233390 Richards, D., & Taylor, M. (2015). A comparison of learning gains when using a 2D simulation tool versus a 3D virtual world: An experiment to find the right representation involving the marginal value theorem. Computers & Education, 86, 157–171. https://doi.org/10.1016/j.compedu.2015.03.009 R Core Team. (2019). The R Project for Statistical Computing. https://www.R-project.org/ Santos, B. S., Dias, P., Pimentel, A., Baggerman, J.-W., Ferreira, C., Silva, S., & Madeira, J. (2009). Head-mounted display versus desktop for 3D navigation in virtual reality: a user study. Multimedia Tools and Applications, 41(1), Article 161. https://doi.org/10.1007/s11042-008-0223-2 Schmidt, H. G. (1983). Problem-based learning: rationale and description. Medical Education, 17(1), 11– 16. https://doi.org/10.1111/j.1365-2923.1983.tb01086.x Slater, M., & Sanchez-Vives, M. V. (2016, December 19). Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 3(74). https://doi.org/10.3389/frobt.2016.00074 Slater, M., & Wilbur, S. (1997). A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoperators & Virtual Environments, 6(6), 603–616. https://doi.org/10.1162/pres.1997.6.6.603 http://earthlab.uoi.gr/theste/index.php/theste/article/view/241 https://doi.org/10.1016/j.cmpb.2013.12.024 https://doi.org/10.1007/s10055-018-0346-3 https://doi.org/10.1145/333329.333344 https://doi.org/10.1145/2533810 https://doi.org/10.1016/j.learninstruc.2017.12.007 https://doi.org/10.1145/3025453 https://doi.org/10.1016/j.compedu.2013.07.033 https://doi.org/10.1007/s00221-016-4846-7 https://doi.org/10.1186/s41077-015-0004-8 https://doi.org/10.1016/j.ejogrb.2019.01.016 https://doi.org/10.1080/13561820.2016.1233390 https://doi.org/10.1016/j.compedu.2015.03.009 https://www.r-project.org/ https://doi.org/10.1007/s11042-008-0223-2 https://doi.org/10.1111/j.1365-2923.1983.tb01086.x https://doi.org/10.3389/frobt.2016.00074 https://doi.org/10.1162/pres.1997.6.6.603 Australasian Journal of Educational Technology, 2021, 37(3). 136 Smetana, L. K., & Bell, R. L. (2012). Computer simulations to support science instruction and learning: A critical review of the literature. International Journal of Science Education, 34(9), 1337–1370. https://doi.org/10.1080/09500693.2011.605182 Stevens, J. A., & Kincaid, J. P. (2015). The relationship between presence and performance in virtual simulation training. Open Journal of Modelling and Simulation, 3(2), 41–48. https://doi.org/10.4236/ojmsi.2015.32005 Taxen, G., & Naeve, A. (2002). A system for exploring open issues in VR-based education. Computers & Graphics, 26(4), 593–598. https://doi.org/10.1016/S0097-8493(02)00112-7 Tran, H. T. T., Ngoc, N. P., Pham, C. T., Jung, Y. J., & Thang, T. C. (2017). A subjective study on QoE of 360 video for VR communication. In Proceedings of the 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP) (pp. 1–6). IEEE. https://doi.org/10.1109/MMSP.2017.8122249 Vogel, J. J., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006). Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research, 34(3), 229–243. https://doi.org/10.2190/FLHV-K4WA-WPVQ-H0YM Wanick, V., Xavier, G., & Ekmekcioglu, E. (2018). Virtual transcendence experiences: Exploring technical and design challenges in multi-sensory environments. In W. Ooi (Ed.), Proceedings of the 10th International Workshop on Immersive Mixed and Virtual Environment Systems (pp. 7–13). Association for Computing Machinery. https://doi.org/10.1145/3210438 Webster, R. (2016). Declarative knowledge acquisition in immersive virtual learning environments. Interactive Learning Environments, 24(6), 1319–1333. https://doi.org/10.1080/10494820.2014.994533 Weech, S., Kenny, S., & Barnett-Cowan, M. (2019). Presence and cybersickness in virtual reality are negatively related: A review. Frontiers in Psychology, 10, 1–19. https://doi.org/10.3389/fpsyg.2019.00158 Worksafe Queensland. (2010, February 4). Vehicle loading cranes. https://www.worksafe.qld.gov.au/injury-prevention-safety/alerts/whsq/2010/vehicle-loading-cranes Zeileis, A., Kleiber, C., & Jackman, S. (2008). Regression models for count data. Journal of Statistical Software, 27(8), 1–25. https://doi.org/10.18637/jss.v027.i08 Corresponding author: Sally A. Male, sally.male@uwa.edu.au Copyright: Articles published in the Australasian Journal of Educational Technology (AJET) are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant AJET right of first publication under CC BY-NC-ND 4.0. Please cite as: Valentine, A., Van der Veen, T., Kenworthy, P., Hassan, G. M., Guzzomi, A., Khan, N., & Male, S. A. (2021). Using head mounted display virtual reality simulations in large engineering classes: Operating vs observing. Australasian Journal of Educational Technology, 37(3), 119–136. https://doi.org/10.14742/ajet.5487 https://doi.org/10.4236/ojmsi.2015.32005 https://doi.org/10.1016/S0097-8493(02)00112-7 https://doi.org/10.1109/MMSP.2017.8122249 https://doi.org/10.2190/FLHV-K4WA-WPVQ-H0YM https://doi.org/10.1145/3210438 https://doi.org/10.1080/10494820.2014.994533 https://doi.org/10.3389/fpsyg.2019.00158 https://www.worksafe.qld.gov.au/injury-prevention-safety/alerts/whsq/2010/vehicle-loading-cranes https://doi.org/10.18637/jss.v027.i08 mailto:sally.male@uwa.edu.au https://creativecommons.org/licenses/by-nc-nd/4.0/ https://doi.org/10.14742/ajet.5487