Using Virtual and Augmented Reality to Create Realistic Training Events Using Virtual and Augmented Reality to Create Realistic Training Events Liz Bacon and Lachlan MacKinnon University of Greenwich, School of Computing & Mathematical Sciences Introduction The importance of realistic training cannot be overemphasised in certain areas of training, for example where quick decisions are required that may impact on life and death situations. Research shows that learning under stress, for example studying for exams or meeting deadlines for the submission of coursework, can help focus the mind and can result in faster processing, storage and recall of information. While there is often a debate about the retention of that information and the knowledge thereby gained, there can be no doubt that this model is frequently purposely self-imposed by students. Research also shows that the performance of individuals under stress can vary tremendously, with some performing worse and others performing better. It is therefore important to bear this in mind when developing training environments that are intended to be realistic and reflect the real-world situations that a crisis manager might encounter. In addition, because people respond to stress in different ways, it is important that the stress in a training environment is managed and controlled at an individual level, and appropriate checks and balances are provided to protect trainees from harm. Strategic level Gold Commanders are trained to manage crisis situations. Typical examples of crises they might have to deal with range from train or plane crash, floods, widespread fire, through to terrorist activities. They will typically always involve a multi-agency response. Gold Commanders are typically trained in one of two ways. The first is through the use of table-top exercises, this is where a group of trainees sit around a table, and are presented with a paper description of a scenario describing a crisis situation. The group of trainees, from each of the key crisis management agencies, e.g. Police, Fire, Health Authority, work together to propose a solution to the problem. The group are guided by a trainer, who has the challenge of making the crisis appear realistic and engaging, and enthusing the trainees to the extent that they suspend disbelief and engage at an emotional level in the crisis situation they are trying to solve. The second typical training mode is through the use of a live exercise designed to simulate a real life crisis situation. These are extremely effective in training and emotionally engaging for the participants in the scenario. They are however extremely time consuming and expensive to set up, and can only ever simulate a small part of a potential scenario leading to a limited number of possible outcomes. At the ISCM symposium in Athens (ISCM 2011), a Greek ship-owner made a plea to those developing training environments for crisis management to ensure that the trainees are made to experience the stress of a real crisis situation. He gave an example of one of his ships that had been lost, attributing this to failures in the crisis management of the situation. Apparently, those responsible for the situation had abrogated responsibility, deferring decisions to insurers and politicians resulting in the breakup of a ship followed by an ecological disaster. The crisis managers, under the stress of the actual situation, had felt unable to take Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 6, 2012 responsibly and make decisions that could have averted a disaster. In recent years, the availability of computer-based multimedia systems, and computer games technologies, has led to a growth in the development of more interactive and media inclusive training environments for crisis management, as evidence by the wealth of projects presented at ISCM 2012 (ISCM 2012). Some of these have begun to address the issue of realistic levels of stress for trainees (e.g. ETC 2012), but to date there has been no systematic approach to the creation and management of stress within individual trainees engaged in strategic level crisis management training. In the rest of this paper we describe the Pandora project, which is seeking to develop just such an approach. Overview of the Pandora Project The EU FP7 Project, Pandora (Bacon et al., 2011 and 2012, MacKinnon and Bacon 2012), has developed a rich multimedia training environment, and associated toolbox, which uses augmented reality to create an emotionally affective ambience, in order to impact decision-making and enhance learning. The application, developed to the level of a robust prototype, is designed to enhance and expand training exercises for Gold Commanders in crisis management. The role of a Gold Commander is to develop strategic plans, and, in the event of a crisis, their key objectives (from the Civil Contingencies act 2004) are as follows (these are not in any order as priorities will vary depending on the nature of the crisis): ●● Relieve suffering ●● Contain the emergency ●● Save and protect life ●● Protect property ●● Maintain / restore critical services ●● Protect the health and safety of staff ●● Provide the public with information ●● Safeguard the environment ●● Maintain normal services ●● Promote and facilitate self-help ●● Facilitate the investigation / inquiry ●● Facilitate community recovery ●● Evaluate and identify lessons learned Training events for Gold Commanders seek to address these key objectives, and are specifically designed to: ●● Determine the strategic planning ability, decision-making capability, flexibility and capability under pressure of the trainees ●● Develop the collaborative skills of the trainees in formulating strategic responses across a number of organisations ●● Push trainees to move away from their pre-prepared plans and think of innovative and creative solutions to solve challenging problems ●● Help the trainees to understand the impact and implications of their decisions on other services ●● Develop the responsive skills of trainees in formulating alternative strategies and remediating actions in the event of the failure of a strategic response ●● Help trainees to recognise and challenge their assumptions as these will inevitably be the focus of any post-crisis analysis, regardless of the outcome ●● To monitor the risk taking behaviour of the group, given research evidence suggests that risk taking behaviour increases in a group situation (Clark 1974) ●● Develop skills to deal with the media, which are inevitably required in the event of a crisis 62 Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 6, 2012 As stated above, there are significant drawbacks with the current approaches to training crisis managers. When a crisis occurs, human behaviour and preparedness are critical to a successful outcome and therefore training events must be as realistic as possible. In order to simulate the pressure of a real crisis situation, a system must be able to simulate the information overload and related stress, together with the requirement for making decisions under pressure that have to be negotiated between the representatives of the different agencies, who may hold opposing views. Pandora therefore aims to bridge the gap between the low cost, table-top exercises and expensive real world simulations by providing a training environment, utilising virtual and augmented reality components, in which the group and the trainer can participate. This environment provides realistic, dynamically changing, time sensitive, immersive crisis simulation exercises that allow trainees to practice their decision-making and negotiation skills within realistic, stress-controlled scenarios, in a safe environment. The Pandora system must provide an environment that engages the trainees at an emotional level, with the aim that the trainees should experience emotions of a similar nature and intensity to those that they might experience when dealing with a real emergency. In order to deliver this requirement, the emotional state of each trainee needs to be monitored and manipulated during the training in a similar way to that which could occur during a real crisis. This aspect of Pandora is based on research into models of emotion, decision-making and learning which are discussed in section 5 of this paper. Serious games and rich multimedia technologies The popularity in the use of games technologies and gaming approaches for areas other than entertainment, typically referred to as “serious games”, has exploded in recent years (Adobe 2007). There has been a proliferation of games in e-Learning, eGovernance, eHealth, Process Visualisation, Industrial Animation, Complexity Modelling, and almost any kind of professional training that can be imagined, including crisis management. The rationale for this rapid growth is linked to the perceived benefits of playing games such as their immersive and engaging environments which can result in the addiction of players, who will play games for hours, trying to solve problems, frequently failing and restarting, not noticing time passing, in order to progress through a game and reach their goal. If this enthusiasm and determination to succeed could be applied to games that support learning then considerable benefits could be realised. This has been documented in many reports, for example the Federation of American Scientists released a report in 2006 saying that students could learn some skills better from playing games than from conventional training. This included the ability to make fast decisions in critical, high-stress situations, and this view was reported from a number of other industries (Chan 2007). Two particular approaches are of relevance here, in reflecting on what might be used to create realistic training scenarios for strategic level crisis management: ●● Creating a rich multimedia training environment, where games and multimedia technologies can be used to enhance existing physical training media and processes (AR: Augmented Reality) ●● Creating a training game where the tasks are crisis management activities, including the planning and realisation phases, and the dynamic feedback and decision modelling (VR: Virtual Reality) In considering the VR approach, three key issues currently have impact on the design: ●● Graven & MacKinnon (2007) report on the importance of fully integrating the learning content with the game play in existing pre-authored games. Overlaying learning content without fully embedding it as part of the game can destroy game play. The research showed that the development of games for a specific purpose is more effective, and that without effective game-play the benefits of immersion and engagement are lost. Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 6, 2012 ●● Flynn et al., (2011) report on the use of games technology to train social workers in scenarios they might not otherwise be able to encounter during their training. The system uses virtual reality immersion within a blended learning environment. The game presents an unfolding scenario to the player, periodically presenting them with decision points.. The game then continues according to the decision made by the student and provides a safe, virtual reality, immersive and engaging environment in which students can explore different outcomes through repeated plays of the game. ●● The age demographic of Gold Commanders is such that the majority of them will have had limited exposure to virtual reality environments and games technologies. Feedback from users indicated that use of such environments may have a negative effect on their perceptions of the technology, and result in a reluctance to engage and immerse fully. It is very likely that over the next 10-20 years we will see a far greater use and acceptance of virtual technologies to train and prepare for actual physical situations, as the current “digital native” population move into more senior positions, and the technologies prove their worth. Whilst there is a debate over how the younger generation learns with technology, it is acknowledged that those born with widespread technology do view it differently to those who weren’t (Becker 2012). However for Pandora it was decided that the use of virtual reality would not be appropriate as the main mode of delivery. So for Pandora, it was felt that the best approach would be to focus on an augmented reality approach but include a virtual reality capability, with the expectation of greater take-up in the future. The advantages of the augmented reality approach are in its inherent flexibility to use variations of digital media etc. in combinations that best fit the profile and makeup of a particular trainee group, which lends itself to an adaptive blended learning approach. Design of Pandora In the Pandora system, training takes place in a crisis training room which has two interfaces: a client that can be downloaded and installed, and a 3D virtual room delivered through a web browser in which trainees are represented by avatars (Liu et al. 2011), see Figure 1. Both of these can be used to deliver training in the following ways: i) At a training site typically used for training Gold Commanders where the trainer and trainees are co- located in the same room where trainees can work at their own computer using the Pandora system to communicate, but meet when needed. Several large screens may be used to display multimedia information to all the trainees such as sound/pictures/maps/animations/videos, to simulate receiving information about the crisis e.g. news broadcasts. ii) Same as i) however delivered at a non-specialised site such as the offices of one of the Gold Commanders (portable mode). iii) The trainer and the trainees are geographically dispersed and communicate only through the Pandora system (distributed mode). 64 Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 6, 2012 Figure 1 VR representation of crisis room, with Slides, Streaming Video and Map Applications A crisis scenario is described in terms of a sequence of events that occur as the crisis unfolds, e.g. event 1 = plane crashes into dam, event 2 = flood in local town, event 3 = town loses electricity supply etc. All of these events are represented on a timeline within an event network, and can be delivered to the trainees through a variety of media, e.g. via text message, through a news broadcast, or through a multimedia clip. The crisis room is managed through the Trainer Support Framework, which provides a number facilities to setup and configure the system both prior to a training session and during a training session. For example, scenario loading, interject new events into a scenario or dynamically, as the scenario is executing, rollback a scenario to a previous decision point, or jump forward to a future point, speed up or slow down the execution of the scenario whilst running. The trainer is also able to configure the scenario to set up non- playing characters (NPC), e.g. to role play an emergency service not represented within the group of trainees (some scenarios require that certain roles are filled e.g. Chief of Police, if no physical trainee is available to undertake that role); subject matter experts; higher control (HICON), such as Government ministers; and lower control (LOCON) - representing the lower levels of command within the crisis team. The system can be configured to play the roles automatically, but the trainer can also take over and simulate responses by these NPCs at any time during the execution of the scenario if they so desire. A record of every event, trainee decision, action, e-mail, use of the chat facility etc. is logged and can be accessed at the end of the training session to support an objective post-training analysis, reflection and feedback by both the trainer and individual trainees. Emotional modelling and use of augmented reality to create affect In order to impact the emotions of trainees and provide personalisation of a training plan, a decision was taken to assess various psychological traits that have been shown to impact decision making under stress. Examples of these psychological traits are background experience (known as a static trait, as this cannot change) and stress and anxiety (which are dynamic traits, as they may vary during a training session). These traits are ascertained through a pre-training questionnaire, and are used in conjunction with goals set by the trainer for each individual trainee (e.g. trainee profile requires a confidence building exercise) to adapt the training as the scenario unfolds. Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 6, 2012 During scenario execution, the system runs through a continuous cycle of planning, execution and re- planning. Throughout the training, the stress level of individual trainees is measured through a number of means such as self-reporting, physiological sensors, and trainer observation. If the trainer decides that one or more trainees appears too relaxed or too stressed and that their stress levels should change up or down, to better reflect a real world crisis. This can be done through a variety of manual interventions such as compressing timelines thus forcing decisions to be taken faster or by interjecting new events into the system. More importantly, it can be addressed in an automated mode by the Pandora system. Within the Pandora training environment, the relationship between affective state and learning is key, and in order to optimise that the project has to have techniques to determine the affective state of the trainee and make appropriate changes to the environment and scenario to address that state. Pandora uses a W3C standard on emotional modelling, to which the project has contributed and which is still under development, called EmotionML (MacKinnon et al., 2012, W3C 2011), to develop the Pandora affective model and mark-up affective content. Having determined the emotional and behavioural state of each trainee, the calculation of the desired state may be pre-defined within the training scenario, dynamically generated by the system, or as a result of direct trainer input. If the desired state differs from the existing state, the system has to plan suitable events to try and move a trainee towards the target state. The key argument for the use of affect in the Pandora scenario is in the creation of affective ambience, by which we mean the use of multimedia assets and information management/manipulation to engender requisite levels of emotional impact on the trainees at decision making points. The key affective representation is associated with the creation of ambience or atmosphere. A number of techniques from film and TV production have been taken, combined with a number of known issues in information distribution and management, to create a set of rules by which multimedia mashups can be created and made available to the trainees. This provides the potential to induce stress in the trainees through the use of vocative inflections, video and image representations of crisis situations, and textual updates from those situations. An example of this might be to show a picture of a hospital that is flooded, and overlay an appropriate piece of audio either saying that the water is receding or that the water is rising and people are about to drown to increase or decrease stress. However, the system can also induce stress through the corruption of media channels, incomplete information, missing information, noisy channels, and a variety of other techniques to diminish the certainty of information available to trainees at those points in the scenario where they are required to make decisions. Using this kind of induced stress, the system can impact on trainees in terms of their self- efficacy, leadership capability, and existing personality traits, to determine their capability to make effective and appropriate decisions under stress. Existing crisis management training using human media personnel as actors to achieve these effects has had considerable success (National Crisis Leadership Centre 2012), and existing research in affective computing has shown this can be successfully achieved systemically (Marsella et al., 2010). Results of user trials A robust prototype of the PATE was used in three days of trials at the Emergency Planning College in the UK. A total of 13 existing Gold Commanders (from novice to very experienced) were split into three groups, each receiving one day of training. The total number of trainees per group were 6 on day 1 (experienced Gold Commanders), 5 on day 2 (Novice Gold Commanders) and 6 on day 3 (immediate experienced Gold Commanders). Some individuals participated in more than one training exercise to ensure a sufficiently full 66 Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 6, 2012 group and, for these trainees, their evaluation was captured after their first training experience. All trainees found Pandora to be engaging and all of the groups were immersed in Pandora for the entire time, trying to resolve the crisis situation they had been given, which was a severe weather scenario. At several points during each training session, the trainees were asked to fill out a questionnaire and graded their reaction on a scale from 0 to 6 (where 6 is high), in the following areas: 1. Learning climate 2. Self-perception of learning 3. Security and privacy 4. Technology Acceptance 5. Oral instructions 6. Tutorial 7. User Interface (overall reaction, system rate, screen features, terminology, learning) Areas 1-4 received no scores below 3 and overall the vast majority of the scores were in the 3-6 range. With only 13 users considerable qualitative feedback was also obtained. Comments focussed around the realism of the tool, the potential, how intuitive it was and how useful it was to be put under the stress of a real crisis situation. The focus of the criticisms were typically around the user interface and the request for enhanced features. However, without exception, the trainees described the training experience as far more realistic than any other training models they had experienced. Some quotes from the trainees included: “Real potential to understand your own ability and potential” “I like the flexibility of the system – it enables real decision making” “Really good system overall, simple to use, I was very impressed, easy to manipulate and do what you want.” “It enables real decision making”, “Really good, definitely as true to life as you can get in a training environment”, “Overall very intuitive, encourages people to respond who might otherwise be less comfortable speaking out”, “Emotional learning to put you under stress, is good, we assume people can do what they can do because they have the job but it’s not necessarily the case so really good.” Similarly positive feedback was received from the trainers. Particular areas highlighted were the learning experiences gained from letting people see the consequences of their mistakes and the importance of the chronological record of events for post-crisis training. Some of the trainers had initially been slightly sceptical about the use of heart-rate sensors until the actual trials when they realised that provided invaluable insight into how each trainee was really feeling, given that several managed to hide any visible signs of stress and also noticing the wide reactions to the same event by different individuals as the scenario unfolded. Some quotes from the trainers were: Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 6, 2012 “I can deliver training to Senior Directors across the country simultaneously using this system” “Being able to let people see the consequences of their mistakes in advance is valuable experiential learning”. “Easy to use” “Flexible system – and truly dynamic” “Total classroom control” “My Executive Level customers would enjoy and welcome the opportunity to train using this system” “When can we have it and how much will it cost?!” Overall the user trials were extremely successful, they were however undertaken on a robust prototype and they were also limited to one day which did not provide trainees with the full training experience which would normally be considerably longer. Whilst the training environment was slightly artificial as the trainees understood the context in which they were being trained, they did nevertheless engage with the training as if it was for real. The outcome of the evaluation validated the approach taken by the project in using immersive multimedia and augmented reality to create realistic training events, and offers considerable potential for future development and use of the approach. Conclusions and future work The growth in the use of digital technologies, and in particular visualisation and simulation technologies, has already had an impact on the training of crisis personnel at operational and tactical levels. However, the provision of training for those involved in the development of high level tactical and strategy plans and responses has yet to take advantage, in any real sense, of these technologies. There are good reasons for this, both in the engagement of high level personnel with modern VR and games technology and, more pertinently, in the need to develop an appropriate, dedicated training environment that addresses their needs. The Pandora project has taken the best of the existing models, in terms of the use of AR approaches, the benefits of table-top and physical simulation, and the impact of real world factors, representing by public media and emotional artefacts, to create just such a dedicated training environment. Unlike existing systems, the Pandora system has been developed both as an AR and a VR system with a view to offering both blended learning and a fully virtual experience, where this is appropriate and desired. In keeping with the needs of an expanding requirement to train more personnel in crisis management and to deal with ever more complex scenarios, the system has the potential to encompass crisis situations of all types and perspectives and to support strategic level training in all of them. The Pandora box, being the set of tools and the methodological approach to configure the system and environment, have intentionally been developed to be generic, rather than specific. The next stage of development for the Pandora system, which is currently in prototype and will be commercialised in 2012, is the development of a robust general-purpose scenario-authoring tool and the redeployment of the environment as a command and control centre. The Pandora system has been developed as a generic event-based system which could be used for training people in a number of fields. The criteria for applicability would be where trainees need to engage with a scenario that has a series of unfolding events which they need to respond to, for example: 68 Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 6, 2012 transport planning, military strategic planning, health service delivery planning, organisational strategic planning, local government service delivery planning and stock portfolio planning. The use of Pandora in these contexts would require that the trainers develop the scenario and corresponding multimedia assets for the system to play out; however, a variety of scenario planning tools are available in the market to assist with this process. Acknowledgement The authors wish to acknowledge the support of the European Commission in funding this work under the IST & Security Call of the 7th Framework - FP7-ICT-SEC-2007-1, Grant Agreement No. 225387. We also wish to acknowledge and thank our partners in the Pandora project for their contributions to this work. References Adobe. 2007. Serious Games: Online Games for learning, Available at: http://www.adobe.com/products/ director/pdfs/serious_games_wp_1107.pdf [Accessed 28 October 2012]. Bacon, L., Windall, G., and MacKinnon, L. 2011. The development of a rich multimedia training environment for crisis management: using emotional affect to enhance learning. Proceedings of the 18th association for learning technology conference (ALT-C 2011), September 6-8, at the University of Leeds, UK. Paper 0151. [Online]. Available at: 02/rlt.v19s1/7780.Full proceedings at available at: http://repository. alt.ac.uk/id/eprint/2159. Bacon, L., MacKinnon, L. Cesta, A. and Cortellessa, G. 2012. Developing a Smart Environment for Crisis Management Training. Journal of Ambient Intelligence and Humanized Computing, 3(2). Becker, K., Fleming, J., and Keijsers, W. 2012. E-learning: ageing workforce versus technology-savvy generation, Education + Training, 54(5), pp.385 – 400. Chan, W-H. 2007. Serious Games=Serious Training, Published in FCW.com as a cover story May 7, 2007. Clark Russell D., III. 1974 .Risk Taking in Groups: A Social Psychological Analysis, The Journal of Risk and Insurance. 41(1), 75-92. eCentre Research Group, University of Greenwich. [Online]. Available at: http://www2.gre.ac.uk/research/ centres/ecentre [Accessed 14 May 2012]. ETC Simulation Training Systems. 2012. [Online]. Available at: http://www.trainingfordisastermanagement. com/about/etc-simulation/ Flynn,R., MacKinnon, L., Bacon, E. and Webb, J. 2011. Maritime City: “Using Games Technology to train Social Workers – some initial results”. Proceedings of ICEC 2011, 10th International Conference on Entertainment Computing, 5-8 October 2011 Vancouver, Canada. Graven, O. and MacKinnon, L. 2010. A Summary of Investigation into Developing Games for Learning Using Different Techniques. Proceedings of the World Conference on E-Learning in Corporate, Government, Healthcare and Higher Education (ELEARN), Orlando, Florida, October 2010, AACE: EdITLib, pp. 1724-1731. http://www.adobe.com/products/ http://repository/ http://www2.gre.ac.uk/research/ Compass: The Journal of Learning and Teaching at the University of Greenwich, Issue 6, 2012 ISCM. 2011. International Symposium on Crisis Management, Vougliameni, Athens, Greece, June 2011. [Online]. Available at: http://www.iscm2011.eu/ [Accessed 19 December 2012]. ISCM 2012. Second International Symposium on Crisis Management, London, UK, March 2012. [Online]. Available at: http://www.pandoraproject.eu/. Liu, H., Arafa, Y., Boldyreff, C. and Dastbaz, M. 2011. Cost-effective Virtual World Development for Serious Games, IGIC 2011. Proceedings of the 3rd IEEE International Games Innovation, pp 48 – 51. MacKinnon, L., Bacon, E. and Windall, G. 2010.Tracking and Influencing Trainee Emotions in a Crisis- Planning Scenario. W3C Workshop on Emotion Markup Language. Paris, 5-6 October 2010. MacKinnon L. and Bacon L. 2012. Developing Realistic Crisis Management Training. Proceedings of the 9th International Conference on Information Systems for Crisis Response and Management (ISCRAM) 22- 25 April 2012, Vancouver, Canada. National Crisis Leadership Centre – Kent County Council Learning & Development. [Online]. Available at: http://www.kent.gov.uk/business/council_business_services/kent_learning development/leadership management/leadership_development/national_crisis_leadership.aspx, [Accessed 14 May 2012]. Marsella, S., Gratch, J. and Petta. P., 2010 Computational Models of Emotion. In Scherer, K.R., Bänziger, T., & Roesch, E. (Eds.) A blueprint for an affectively competent agent: Cross-fertilization between Emotion Psychology, Affective Neuroscience, and Affective Computing. Oxford: Oxford University Press. W3C. 2011. Emotion Markup Language (EmotionML) 1.0 Working Draft 7 April 2011. [Online] Available at: http://www.w3.org/TR/emotionml/ Author Biographies Professor Liz Bacon PhD, CEng, CSci, FBCS, CITP, FHEA, is Dean of the School of Computing & Mathematical Sciences at the University of Greenwich, Chair of the BCS Academy of Computing and a BCS (Chartered Institute for IT) Vice President. She is Co-Director of the eCentre research group and is an experienced systems designer and developer. Lachlan MacKinnon is professor of Computing Science (Strategic Development), Head of two Departments in the School of CMS, and co-Director of the eCentre, at the University of Greenwich, U.K. His research interests are in computing policy, information and knowledge engineering, smart systems, games and creative technologies, eHealth and e-Learning, and computer security. 70 http://www.iscm2011.eu/ http://www.pandoraproject.eu/ http://www.kent.gov.uk/business/council_business_services/kent_learning http://www.w3.org/TR/emotionml/ Liz Bacon and Lachlan MacKinnon Overview of the Pandora Project Serious games and rich multimedia technologies Design of Pandora Emotional modelling and use of augmented reality to create affect Results of user trials Conclusions and future work Acknowledgement References Author Biographies