e-Review of Tourism Research (eRTR), Vol. 17, No. 2, 2019 http://ertr.tamu.edu 253 Janine Videva, janine.videva@usi.ch USI - Università della Svizzera italiana (Lugano, Switzerland) Elena Marchiori, elena.marchiori@usi.ch USI - Università della Svizzera italiana (Lugano, Switzerland) Lorenzo Cantoni, lorenzo.cantoni@usi.ch USI - Università della Svizzera italiana (Lugano, Switzerland) Assessing usability and user experience of immersive web VR platforms for tourism destinations Until recently, Virtual Reality (VR) was considered as a niche technology due to slow advancement and high costs. However, thanks to new industry improvements, VR has started to approach the mainstream audience especially through the use of web VR, therefore becoming a new tool for communication. In this sense, due to its content richness and emotional density, the tourism industry reveals as a proper one to exploit the new opportunities coming from web VR. As a result, it becomes imperative to research usability and user experience of web VR platforms for tourism destinations. Despite this importance, standards and guidelines on the topic are still missing. This paper focuses on exploring usability and user experience issues of two web VR experiences devoted to tourism destinations, namely Petra, the ancient city in Jordan, and Bilbao in Spain. The methodology used for this research is based on two sets of VR heuristics applied to the MiLE+ usability evaluation method. Merging these two methodologies has allowed to explore both technical, application independent issues, as well as user experience-related, application dependent issues. Finally, guidelines on usability and user experience factors is proposed, as a first step towards the preparation of a complete list of guiding principles on this topic. Key words: web VR, Usability and User Experience, Tourism Destination Janine Videva is graduate student from the Faculty of Communication Science at USI - where she earned MSc in Corporate Communication. Elena Marchiori is a fellow researcher and lecturer at USI. Her research areas are online tourism communication, reputation in online media, usability and digital analytics, maturity of destinations and web adoption, augmented and virtual reality in tourism. Lorenzo Cantoni is full professor at USI, Faculty of Communication Sciences, where he is director of the Institute of Digital Technologies for Communication. http://ertr.tamu.edu/ mailto:janine.videva@usi.ch mailto:elena.marchiori@usi.ch mailto:lorenzo.cantoni@usi.ch e-Review of Tourism Research (eRTR), Vol. 17, No. 2, 2019 http://ertr.tamu.edu 254 Introduction It hasn’t been long since virtual reality (VR) was serving only limited users in few industries, mainly in military trainings and medical environments (Murtza, Monroe, & Youmans, 2017). In the last years, this scenario has changed due to novel improvements like the web VR, allowing users to access a VR experience simply through the internet browser either with or without a VR headset. The easy access to VR through the web opens the stage to a new tool for communication, which is also highly engaging from an emotional point of view. In this sense, one of the fields where web VR is making its first steps is the tourism one, being a pioneer in the application of information and communication technologies (ICTs) (Jung and Dieck, 2018; Tussyadiah, Wang, Jung, and Dieck, 2018). In fact, this is not a casual choice of application, since the nature of tourism is being content rich and emotionally dense and therefore it is a proper area where to apply VR experiences. Considering both VR as a new communication tool, and the tourism field as a plausible application for such innovation, it becomes imperative to research usability and user experience of web VR for tourism destinations. The easiness of use for non-experienced users becomes a necessary requirement in order to achieve mainstream adoption. Nevertheless, challenges in terms of evaluation methods for usability and user experience are arising. Despite their importance, standards and guidelines on the topic are still missing. In the last years, some researchers have worked on defining specialized sets of heuristics for immersive environments. However, the literature lacks studies on this topic applied to web VR environments for tourism destinations. As a consequence, the present research focuses on the need for better understanding usability and user experience of current web VR tourism applications. Research design The following research aims at identifying some common issues connected to the usability and the user experience of two web VR platforms. To do this, the Lugano Milano Evaluation Method (MiLE+) (Triacca and Bolchini, 2004; Triacca, Inversini and Bolchini, 2005) has been used, together with selected VR heuristics from Sutcliffe and Gault (2004) and Rusu et al. (2011). The methodology of the present research is based on three main parts – Technical Inspection, User Experience Inspection (UEI), and User Testing. In order to answer the research questions, the following procedure was executed. Step 1: Development of Usability Kits. Creation of U-KITs for both technical and user experience inspections, as well as http://ertr.tamu.edu/ e-Review of Tourism Research (eRTR), Vol. 17, No. 2, 2019 http://ertr.tamu.edu 255 interview questions for user testing. The purpose of this step is to prepare the set of tools needed during the experiment, including user profiles, goals and tasks, heuristics, UEIs, and evaluation matrixes. Step 2: Conduction of usability and user experience inspection and user testing. Four experts participated to the usability testing performing technical inspections on the selected Web VR, covering the destinations Petra and Bilbao. The expert’s background was select as follows: i) a full-time university professor of usability and online communication; ii) a lecturer and ICT researcher; iii) a lecturer and expert in user experience and virtual reality; and iv) a PhD candidate specialized in online communication. For the sake of simplification, user experience inspections were executed assigning two experts to Petra’s platform and two to Bilbao’s website. Finally, an end-user testing was performed with the help of other four participants recruited from the university, allocating two of them to one platform and two to the other one. These candidates were selected according to the user personas created for this research. Therefore, the four participants were chosen according to their different level of VR experience and technology expertise. This was done with the aim to represent a wider span of real-life users. Regarding the headset of use, the one selected for this study was the Google Cardboard, which is the most accessible one on the market in terms of price and easiness of use. Step 3: Data analysis and comparison of the results. After the end of the assessments with experts and end-users, the data gathered through the evaluation matrixes, observation, note-taking and interviews was carefully transcribed. Therefore, the main issues were extensively described, answering in this way to the first research question (RQ1: Which are the main issues that threaten the usability of each web VR 3016 experience?). Next, the results were compared in order to find common issues between the two platforms and answer to the second research question (RQ2: Which are the common patterns of issues present in the web VR 360 experiences?). Afterwards, the results of the overall subjective satisfaction of final users were reported and described, giving an answer to the last research question (RQ3: Are the end-users satisfied with the web VR experiences?). Case studies and sampling For this study, two destination-related websites were chosen. One dedicated to Petra, the ancient city in Jordan (petravr.withgoogle.com), and one dedicated to Bilbao, the industrial port city of Spain (bilbao360walk.com). http://ertr.tamu.edu/ e-Review of Tourism Research (eRTR), Vol. 17, No. 2, 2019 http://ertr.tamu.edu 256 Figure 1 Petra's web VR platform Figure 2 Bilbao's web VR platform The main criteria of selection were the good technical functionality of the website using the Google Cardboard headset, the website’s topic related to a specific touristic destination, as well as the language of the platform. Petra’s full 360-degree experience for Cardboard was created by Google in 2017, whereas a first version of Bilbao’s 360 walk was online already in 2015. Usability Testing Kits (U-KITs) preparation Prior to research execution, the Usability Testing Kits for the three tests were prepared. Concretely, the U-KIT for the technical inspection comprises the technical inspection heuristics, and technical inspection matrix. Usability heuristics are general principles that help with the evaluation of navigation problems mostly related to user interface design, whereas the technical inspection matrix represents a table comprising the selected heuristics serving as a guideline of the expert executing the evaluation. The U-KIT for the user http://ertr.tamu.edu/ e-Review of Tourism Research (eRTR), Vol. 17, No. 2, 2019 http://ertr.tamu.edu 257 experience inspection, on the other hand, includes user profile and scenario, user experience indicators (UEIs), and evaluation matrix for user experience inspection. Similarly to the technical inspection, the user experience indicators (UEIs) represent a selected list of relevant heuristics focused, this time, on experience-related topics. Finally, the scenario-based user testing contains the user profile and scenario, and the questions for users’ interview. Technical Inspection Heuristics and UEIs for web VR: the technical inspection heuristics and the user experience indicators for this research were created by integrating and merging some dimensions of the two VR heuristic sets from Sutcliffe and Gault (2004) and Rusu et al. (2011) and the usability heuristic framework used in MiLE+. Precisely, for the technical inspection heuristics the following items from Rusu et al. (2011) were chosen: Low memory load, Flexibility and efficiency of use, Orientation and navigation, VR interaction, VR rules, Error prevention, Recovering from errors. Additionally, Clear entry and exit point, and Faithful viewpoints were taken from Sutcliffe and Gault (2004). At the end, Text Accuracy, Text Errors, Coverage, Information Overload, Clarity, Simplicity from MiLE+ were included. For the creation of UEIs, the heuristics were combined as follows: Completeness of content, Richness of content, Satisfaction of content, Predictability of content, Understandability of content, Naturalness, Effectiveness, Engagement, Satisfaction of the experience from MiLE+, and Sense of presence from Sutcliffe and Gault (2004). Another practical tool used during the testing is the evaluation matrix. As previously mentioned, it represents a table of the heuristics of reference, plus score and comment boxes where evaluators are asked to fill-in their observations. As specified in the literature, the score scale can be decided by the evaluator themselves and for this reason it was preferred to follow the example of Triacca (2005) applying a score of 3 for poor performance, 6 for average, and 9 for a good one. Continuing further, two user profiles and scenarios were developed for the user experience inspection. Their main objective is to help inspectors impersonate real users while interacting with the platform. The method applied for the creation of the scenarios is the so-called visioning technique, in which the main target group of end-users, their goals and tasks are imagined by the researcher. Following the guidelines for user profiles creation from Inversini and Cantoni (2014), dimensions such as demographic data, technical ability and knowledge, motivation, and behavior were considered. Finally, a semi-structured interview with the users was performed by the researcher during the User Testing. The interview questions were based on the same dimensions of the UEIs (Content Experience, Navigation & Cognitive Experience, Interaction Flow Experience) in order to allow comparison with inspector’s results at a later stage. Moreover, at the end of the interview two questions, aimed http://ertr.tamu.edu/ e-Review of Tourism Research (eRTR), Vol. 17, No. 2, 2019 http://ertr.tamu.edu 258 at assessing the overall subjective satisfaction of users, are asked. These questions aim to answer the third research question and are based on the model from Nielsen & Pernice (2010). The model helps the understanding of the connection between task-completion success and happiness with the experience. Results During the testing, data was collected through the compilation of the inspection matrixes for technical inspection and user experience inspection, careful observation, note-taking, and interviews with end-users. At the end, content analysis was performed. RQ1: Main Issues related to Petra’s web VR experience: the technical inspection for Petra’s VR has shown an average usability performance (6.38 out of 9) with main issues connected to navigation. Concretely, the forward and backward feature was rated as poor and the information regarding avatar’s position was not perceived as clear. Other issues that emerged were missing abbreviations and shortcuts and the impossibility to reach a point of interest without returning back to settings. RQ 1: Main Issues related to Bilbao’s web VR experience: Differently from Petra, the Bilbao’s platform is quite poor of content with no audio guide nor text or labels available. As a consequence, the lowest score from the inspection testing goes to the content dimension, with an overall performance between poor and average (4.68). It is interesting to notice how, even in an immersive environment, the importance of the content is crucial, and it affects all other dimensions. For example, the navigation was highly criticized for not providing indications about how to reach a desired destination. According to the users, there should be text labels on the menu to help guide the navigation or other text indications that would give an understanding of what is being seen at the moment and what is still there to discover. RQ2: Common patterns of issues present in both platforms: the following list presents the patterns of issues emerged for Petra’s and Bilbao’s platforms: - Usability issues: clear understanding of avatar’s position at any point; more in-depth content for both platforms is needed; the faithful viewpoints weren’t satisfactory; need for balancing the information load of the interface design. - User Experience issues: better distribution and architecture of the content required; clear purpose of the platforms is missing; inefficiency in terms of tasks completion. - User testing issues: Content distribution is not sufficiently thought through; images were not vivid enough; poor interactivity with objects leads to less natural perception. http://ertr.tamu.edu/ e-Review of Tourism Research (eRTR), Vol. 17, No. 2, 2019 http://ertr.tamu.edu 259 RQ3: End-users satisfaction with the web VR experiences: the results from the end-user satisfaction show that the immersive experiences are “pretty satisfying”, despite the difficulty to complete a task. In this sense, it can be assumed that immersive experiences have an emotional effect on participants, which differs greatly from navigating a website with the only aim of accomplishing a precise task, like for example booking a ticket. This positive emotional component, indeed, makes users overlook eventual lacks in performance, while still enjoying the navigation. Conclusion Results of the study revealed common factors of improvement which were grouped in the following nine points. This list could serve as a first step toward future research in the field of usability and user experience for touristic immersive experience in web VR and the preparation of a complete list of guidelines on the topic: 1) having a clear map of avatar/user’s position at any point easily accessible on the desktop; 2) being able to instantly access from one point to another directly from the menu; 3) having clear indications of what can be found in any different scene through text labels or images; 4) having clear indications of interactive objects with the help of arrows, buttons, signs, and color; 5) having the forward and backward navigation feature always easily accessible on the screen; 6) being able to get additional in-depth information on specific points of interest; 7) having a balanced distribution of information without overloading with heterogenous tools like text and audio all at once; 8) provide a way to mark the already visited scenes in the menu or on the map; 9) being able to explore a scene from more than one perspective through different angles or additional images. The main limitation of this study is connected to the limited sample and the empirical research methods used to gather data from the experiments. For this purpose, an eye tracking device that can be used with Web AR is suggested for future research. Furthermore, different and more advanced headset should be tested, and the sample should be enlarged in order to ensure variability in the results. References Inversini, A., & Cantoni, L. (2014). Nuovi media nella comunicazione turistica. Roma: Società Editrice. Jung, T., & Dieck, M. (2018). Augmented Reality and Virtual Reality. Springer. http://ertr.tamu.edu/ e-Review of Tourism Research (eRTR), Vol. 17, No. 2, 2019 http://ertr.tamu.edu 260 Murtza, R., Monroe, S., & Youmans, R. (2017). Heuristic Evaluation for Virtual Reality Systems. Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting. Nielsen, J., & Pernice, K. (2010). Eyetracking web usability. New Riders. Rusu, C., Roncagliolo, S., Rusu, V. (2011). A Methodology to establish usability heuristics. ACHI 2011 : The Fourth International Conference on Advances in Computer-Human Interactions. ISBN: 978-1-61208-117-5 Sutcliffe, A., & Gault, B. (2004). Heuristic evaluation of virtual reality applications. Interacting with computers, 16(4), 831-849. Sutcliffe, A., & Kaur, K. (2000). Evaluating the usability of virtual reality user interfaces. Behaviour & Information Technoloogy 19(6), 415-426. Triacca, L., & Bolchini, D. (2004). MiLE+: Milano-Lugano Evaluation method, A systematic approach to usability. Retrieved from USI TEC-lab: http://hoc.elet.polimi.it/garzotto/pdf/MiLE+_article_dec04.pdf Triacca, L., Inversini, A., & Bolchini, D. (2005). Evaluating web usability with MiLE+. Proceedings ofthe 2005 Seventh IEEE Internatioonal Symposium of Web Sitet Evolution. Tussyadiah, I., Wang, D., Jung, T., & Dieck, M. (2018). Virtual reality, presence, and attitude change: Empirical evidence from tourism. Tourism Management, 66, 140-154. http://ertr.tamu.edu/ Technical Inspection Heuristics and UEIs for web VR: the technical inspection heuristics and the user experience indicators for this research were created by integrating and merging some dimensions of the two VR heuristic sets from Sutcliffe and Gaul... Results