Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism Immersive Virtual Reality in Improving Communication Skills in Children with Autism https://doi.org/10.3991/ijim.v11i2.6555 O. Halabi Qatar University, Doha, Qatar ohalabi@qu.edu.qa S. A. El-Seoud The British University in Egypt, Cairo, Egypt Samir.Elseoud@bue.edu.eg J. M. Alja'am Qatar University, Doha, Qatar jaam@qu.edu.qa H. Alpona Qatar University, Doha, Qatar ha1106514@student.qu.edu.qa M. Al-Hemadi Qatar University, Doha, Qatar ma1001810@student.qu.edu.qa D. Al-Hassan Qatar University, Doha, Qatar da1003669@student.qu.edu.qa Abstract—Individuals in the Autism Spectrum often encounter situations where they have to respond to questions and situations that they do not know how to respond to, such as, questions asked by strangers including ones related to daily-life activities. A variety of research has been done to improve social and communication impairments in children with autism using technology. Immersive virtual reality is a relatively recent technology with a potential to bring an effective solution and used as a therapeutic tool to develop different skills. This paper presents a virtual reality solution to reduce the gap experi- enced by autistic children due to their inability to establish a communication. An interactive scenario-based system that uses role-play and turn-taking tech- nique was implemented to evaluate and verify the effectiveness of immersive environment on the social performance of an autistic child. Preliminary testing of the system demonstrated the feasibility of VR-based system as a tool for im- proving the communication skill in Autism Spectrum Disorder (ASD) children. The results of the comparative usability study show the effectiveness of immer- sive VR in motivating and satisfying the autistic. Keywords—Autism spectrum disorder, communication skill, immersion, social performance, virtual reality. 146 http://www.i-jim.org Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism 1 Introduction A person diagnosed with autism faces disabilities such as social, physical or emo- tional, within a spectrum known as Autism Spectrum Disorder (ASD). ASD is a neu- rodevelopmental disorder that is characterized by deficits in social interaction and communication that is accompanied by restricted patterns of interest and behavior [1]. The rise in the number of children worldwide diagnosed with autism from 1 in 150 (in the year 2000) to 1 in 68 (in the year 2010) indicates the prevalence of autism around the world in recent years [2]. The three main areas of difficulty which all people with autism share are some- times known as The ‘triad of impairments’: (1) social impairment (difficulties in relat- ing to other people), (2) communication impairment (difficulties in recognizing verbal and non-verbal communication channels), and (3) rigidness in thinking, language and behavior [3]. Impairments in communication remains as one of the most prominent limitations of children with autism [4]. Children in the autism spectrum commonly demonstrate difficulties in initiating as well as responding to social interactions. These difficulties include processing the situation and dialogue surrounding them, along with the inabil- ity of understanding non-verbal aspects of communication, lack of human engage- ment, and the inability to generalize between environments [3][5]. Traditional ways to cope with these challenges that autistic children face include one-on-one repetitive interactions, sustained attention activities, and engagement in reinforcement strate- gies. Many studies have investigated applications of technology as assistive therapeutic tools for autism, namely, computer technology [6], Virtual Reality (VR) [4], and ro- botics systems [7]. We chose VR because it provides a simplified but explorative training environment with no complex direct human-to-human interaction. VR sys- tems are suitable to provide maximum feeling of presence in a computerized virtual setting [4]. Hence, VR-based human-computer interaction tasks have been proposed as a reasonable medium to provide ASD intervention [5]. With the recent advance- ments in computer-aided learning, the individualized intervention approach is consid- ered a key method particularly for young children [3]. In recent years, after the realization of the significance of using technology in the education process of autistic children [3], a variety of research using virtual reality technology has been initiated to positively impact communication skills in children with autism. The novel techniques of using virtual reality is recognized to prove effec- tive in achieving enhanced communication or dialogue in autistic children [5] [8] [9]. Due to the fact that the behaviorally defined syndrome of autism is a spectrum dis- order it does not have a universally accepted treatment. Despite this fact, creating generalized situations as part of an intervention program that could potentially act as an individualized virtual environment solution to enhancing communication skills of the targeted children at their own pace. The nature of VR environments is such that they maintain the ability to induce controlled stimuli (verbal or non-verbal) and also allow to monitor the behavior of the child within the virtual environment (VE) [3]. Since autistic children have the inability to establish human to human interaction, iJIM ‒ Vol. 11, No. 2, 2017 147 Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism with the use of explanatory environments, the VE replicates as a real-life scenario in the form of human to avatar interaction [8]. Therefore, VR based systems can prove to be effective as a medium of educating children with ASD in controlled interactive environments [3][8]. Using techniques of Role-play and Turn-taking are approaches that are efficient in establishing conversations that are similar to the virtual scenarios in reality. In a con- trolled environment, VEs provide the advantage of designing specific social situations where users can mimic avatar(s) by receiving instructions from them. The study presented in this paper is an initial prototype of a VR-based system in- tended to explore the effectiveness of VR solution targeting communication skills in high-functioning children with autism. Therefore, the objective of this study has two- fold: (1) the development and design of immersive VR-based system with verbal/non- verbal input interaction that is based on the time required for the user to respond in the VE task, (2) study the impact of immersion level on the performance of high- functioning autistic users. The virtual reality experience presented to the subject in study using three different installations, four walls projection room CAVE (Computer Augmented Virtual Environment) that provides high-level of immersion and interac- tivity, Head Mounted Display (HMD), and the third is non-VR normal desktop screen. Through our system was not designed as an intervention platform, the prelim- inary feasibility study was designed as a proof-of-concept application. 2 System Design The VR-based system is comprised of two main modules: 1) system architecture framework; 2) social communication task module. 2.1 System architecture The system architecture comprises many modules integrated together towards VR- based interactive simulation three subsystems: display and trackers, auto-navigation, speech recognition, and gesture recognition Figure 1 shows an overview of the system architecture with all modules integrated. Display and trackers: An immersive surround-screen display system comprising four rear-projected stereoscopic screens, three as front, left and right wall, and the fourth one as floor. Polarized glasses were worn on the user’s head with two markers on the rim of the glasses to be tracked by PPT-E tracking cameras from Worldviz [10] to calculate the head position. The second display used is the Oculus Rift head mounted display. The third type is normal 20 inches LCD display. These are used in the automatic navigation module to guide the user though the virtual environment based on the user’s view. There are also parts of the display module to present differ- ent level of immersion of the virtual environment. Speech recognition: Since this project is centered on improving primary social communication skills in children with autism, the system uses this module to effi- ciently recognize the speech coming from the user as response to the virtual charac- 148 http://www.i-jim.org Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism ter(s) in the VE scenario. A wearable microphone is used to allow the system to re- ceive user’s voice. Microsoft speech SDK was used to detect the recommended word within the scenario. The only problem we faced is that the engine can only deal with English language. We need to have support for Arabic language, as most of the pa- tients in the gulf region may not be able to communicate in English Gesture recognition: We included this module in the system to be able to detect hand gestures, such as waving hello etc. from the user. This contributes toward the non-verbal communication that user exhibits instead of verbal, to be able to track any kind of (verbal/non-verbal) response from the user. LEAP Motion device is used to obtain the position of the hand and its skeleton information for mapping the user’s hand into the system. Virtual environment and avatar: In this work, we developed the VR scenario in order to achieve verbal and/or non-verbal response from user with ASD using Vizard from WorldViz [10]. The Software comes with limited resources such as avatars, virtual objects and scenes. So we had to design customed avatars and virtual scenes to create specific social communication environment for our study. The virtual world and the character were designed and rendered in 3Ds Max [10]. Each of the avatars was built with a purpose that contributes to the scenario as can be seen in; and were generated using Autodesk’s Character Generator while they were rigged and animated using Autodesk’s 3Ds Max and Motion Builder software. The environment selected to be a class room at school which is common place and should be familiar with most of the subjects. Figure 2 shows the school virtual envi- ronment and the avatars created for the simulation Auto-navigation module: This module uses the Tracking Sensors as input and starts the automatic navigation through the virtual environment based on the user’s view. All the above modules manipulate the Virtual Environment and Characters of the system which affects the view and the sound output as per the input change. User experiences changes based on these modules and the input provided on the output devices. 2.2 Social Communication Task Module To be able to improve communication skills in high-functioning children with au- tism, it is essential to carefully make sure the environment we develop is efficient in triggering expected response from the user. We developed a scenario of ’greeting’ within a classroom (school) setting which involves communication between a virtual character and the user. Our communication task module is comprised of (1) the ‘greeting’ scenario, and (2) the 3D environment and characters to implement the scenario. Design the scenario: We developed the ‘greeting’ scenario in consultation with experts and practitioners in the local center of education for the autistic who handle autistic children on a daily basis. The scenario was also inspired from the novel tech- niques of ‘Social Stories’ being used by a number of autism-related VR solutions that further justify our approach [11]. The scenario of ‘greeting’ is a general situation that iJIM ‒ Vol. 11, No. 2, 2017 149 Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism Fig. 1. High Level System Architecture & Mockups. (a) (b) (c) Fig. 2. Snapshots of the virtual world and the avatars. (a) the school; (b) Sarah the student; (b) the teacher avatar. an average individual faces multiple times every day; which can help us track the user’s level of social interaction with multiple usages of the VR system. In the scenario, we start by virtually auto-navigating from outside a school envi- ronment going into the school and reaching the classroom through the corridor. The classroom environment is the main setting where the conversation development task takes place. A virtual teacher avatar within the classroom is standing with two other virtual student avatars in front of the chalkboard. The teacher initiates conversation and instructs all the students including the subject to perform the greeting task, see Figure 3 to Figure 8. 150 http://www.i-jim.org Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism The virtual avatar of the teacher was designed to look similar to what we found the instructors/therapists wear within the school’s premises during their sessions with autistic students. The virtual avatar of the students within the virtual environment as well as the virtual environment (school and classroom environment) itself try to fol- low a similar conduct of presentation to provide high amount of relatability to the participant. Keeping in mind the constraints related to the curriculum being followed at these traditionally-practicing institutions, and the inherent limitations of high- functioning autistic children in general, the scenario has been developed to cope with the barrier of initiating or responding to a conversation with a stranger. Role-play: Since our scenario should be shaped around getting a response from the child, all the tasks and conversations within the scenario build to rise the potentiality of the user responding to the task given to him/her by the virtual teacher (i.e. to greet her back) by showing the user (through the other virtual students playing their charac- ter in the scenario) how to reply to a greeting. User response is triggered/instigated (but not forced) by using the technique of role-playing the student avatars in VE. Turn-taking:In order to ensure structured form within the task, each virtual student character is asked to perform the assigned task of ’greeting’ turn-by-turn. After the teacher has instructed the students of the task, each student is greeted and asked to reply to the greeting. The user is assigned the same task in the end of the scenario. This teaches the user to respond when spoken to and wait for his/her turn when in groups. The Scenario in the virtual environment: We designed the virtual world to display and integrate the above mentioned scenario in the following order: 1. Auto-Navigation: The user automatically navigates through the virtual environ- ment (outside the school to inside the classroom through the corridor) Figure 3. 2. Welcome and Introduction: Virtual teacher character introduces self and virtual student avatars Figure 4. 3. Virtual teacher greets one student avatar in the VE first, and receives the response Figure 5. 4. The teacher then greets the next virtual student in the VE and gets similar verbal and non-verbal response (i.e. “waving animation”) Figure 6. 5. The user gets the final turn where the virtual teacher avatar greets the user (custom- ized using the name of the user within the teacher’s speech); and the teacher ex- pects a response from the user. 6. Voice and Action Monitoring: The user’s voice and physical motion is continually tracked to record a response and its corresponding time to respond Figure 7. 7. Task Completion: The virtual environment displays text labels that show the amount of time the user has taken to respond to the task assigned (i.e. the user has greeted back) as shown in Figure 8. iJIM ‒ Vol. 11, No. 2, 2017 151 Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism ! Fig. 3. Auto-Navigation from outside the school to classroom ! Fig. 4. Welcome and task introduction ! Fig. 5. The teacher greets one student and recevies the response, verbaly and non-verbaly througn waving hands animation ! Fig. 6. The teacher greets the next student and gets similar response ! Fig. 7. It is the turn of the subject to do the same. The teacher greets the user calling his name and wait for a response. 152 http://www.i-jim.org Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism ! Fig. 8. Task completed. The response time shows on the display. ! 3 Experiments and Methods 3.1 Experimental setup The investigation experiment was developed using the VR environment described in previous section. The participants’ view orientation was tracked by Positional Tracker in Oculus Rift HMD and/or PPT Tracking Camera System for CAVE display experiments. The scenario is presented on three interactive displays with different levels of im- mersion, namely: (1) Desktop computer, with no immersion factor, (2) Head-Mounted Display - Oculus Rift, and (3) CAVE immersive display. Start time and the participant’s response time is recorded and displayed at the end of successful completion of the assigned task. The response time is logged when the system detects user’s speech recognized by the voice input device. The task progress can be observed during the sessions by therapists and for tracking non-verbal re- sponse. The system also has the ability to receive input from a monitor to record re- sponse time in cases of non-verbal response, such as waving etc. Tasks and procedures: We designed a usability study as proof-of-concept to in- vestigate the implications of the designed system with the three different interactive displays. The participants were required to commit to a total of two sessions for the study (Sessions 1 and 2) on separate days for a duration of approximately 20 minutes per session plus their feedback. We first conducted a learnability session where the team briefed the participants and their caregivers about the session and had the partic- ipants try the immersive display setup(s) (CAVE and HMD) to get used to the tech- nology for genuine results. The participants were also told that they could choose to withdraw at any point during the session for reasons such as feeling uncomfortable with the system, or dizziness due to wearing HMD, etc. Participants were introduced to each system one by one, i.e. the session started with the scenario being presented to iJIM ‒ Vol. 11, No. 2, 2017 153 Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism the participant on the desktop computer first, where the participant was equipped with only a microphone (for voice input) which was used for all three interaction sessions. Second, the participant was presented the same scenario and assigned the same task on the Head-Mounted Display (Oculus Rift) and the response time was recorded for it. Finally, the scenario was replayed on the CAVE with the trackers and 3D glasses and the response time for each session was recorded similarly. The next day of the tests, Session 2 was performed on the participants where they went through the same procedures as Session 1 but without any guidance or pre-testing. Each participant was asked to provide feedback for the questionnaire after every session. The questionnaire included multiple-choice questions and statements of the follow- ing themes: (1) Usability Satisfaction of the system on each interactive display, and (2) Impact of the level of immersion on the response of the participant. We extracted information about the satisfaction and impact of immersion level on the user (partici- pant) based on their feedback. 4 Results and Comparative Study According to our goals and objectives, the system was to be tested on high- functioning autistic children of ages 9-12. Although, due to the unavailability of tar- geted participants, due to the lack of administrative and logistical support at local autism-related institutions, we had to perform our tests on typically developing chil- dren (not in the autism spectrum) with equivalent ages. We found that, typically de- veloping children of ages between 3-6 are closest in mental and cognitive develop- ment to that of high-functioning autistic children aged 9-12 [12]. System usability and acceptability: In our usability study, we investigated the ef- fectiveness of our VR-based interactive system by understanding the ‘Usability Satis- faction’ feedback from the participants on every interactive device per session. Using our participants’ exit survey feedback at the end of the experiment, it was revealed that it was a general trend among our participants to favor the CAVE environment while being satisfied with all the different interactive displays and virtual environ- ments. They had a certain (acceptable) level of understanding of the scenario present- ed to them in the virtual environment. They were also able to recognize the system’s virtual classroom, school, classmates and teacher. They faced very minimal problems trying to feel at-ease within the virtual environment, according to our observation. Moreover, they kept focus on the VR-system, in an attentive manner. We found the learning curve to stabilize around an average response time of 20-30 seconds for the first training session the participants had to get acquainted with our VR-based interactive system in the CAVE. As a conclusion, the system proved to be an easy-to-use platform that our participants would benefit interacting with. In the present study the questions in the exit survey were subjective. Thus, from our finding from exit survey, we can infer that our system has a potential to be accepted by the target population. 154 http://www.i-jim.org Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism 4.1 Quantitative analysis of participant’s’ performance in multi-session interactions User satisfaction: Our findings indicated that users were, in general, satisfied with the virtual environment in our VR-based system as per their feedback, as well as their decreased response time. A decreased response time shows the users’ understanding of the scenario and how many sessions they take to understand a virtual scenario, to be able to respond promptly and correctly to the system. In our comparative study, the three aforementioned immersive interfaces displayed the same scenario to every participant per session. The user satisfaction for each level of immersion for each session is shown in Figure 9. Impact of Immersiveness on Participants’ Performance: The scenario was test- ed on each user, with the three different displays per session to determine the impact of the level of immersion of the VR system on the users’ performance. From Figure 10 we find that according to the response time of the participants in every individual interactive display device, the general impact of the CAVE is much higher and effec- tive in obtaining the desired response from the user. Thus, we find an improvement in performance of our participants when exposed to immersive VR interface. Since CAVE provides the highest level of immersion in VR, this proves the general effec- tiveness of an immersive interface in improving communication skills in children with autism using a realistic scenario. Fig. 9. User satisfaction with the system for session 1 and session 2. iJIM ‒ Vol. 11, No. 2, 2017 155 Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism Fig. 10. Impact of immersive display on participants’ performance. 5 Conclusion In the present work, we developed a VR-based (verbal/non-verbal) interactive sys- tem for developing communication skills in children with autism based on a prede- fined and planned greeting scenario. We have presented the system’s development with results and comparisons from the usability study to test the efficacy of the developed system as an effective tool for the development of communication skills for autistic children. The results of the comparative usability study discussed above shows the effective- ness of the system contributing to improvement of the autistic user’s communication skills. Although, the study has shown positive impact in almost all the participants, there are certain limitations to considering these results in general for all high- functioning children with ASD. This is due to the fact that the participants could be real autistic users to provide us with more accurate results in order to understand how our findings will relate to real autistic participants, instead of substitutes. This is also essential so as to verify how the impact of our VR-system can be generalized to all high-functioning population of individuals with the disorder. In this research, the system is tested to help overcome communication impairment challenges that can result in development of targeted skills in targeted autistic children thereby helping them proceed toward a more independent life. 156 http://www.i-jim.org Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism 6 Acknowledgment We would like to thank Mr. Bryan Boyle from MADA for his advices regarding autism, Mr. Abdulla Ityani from Shafallah Center for allowing us to interact with the children and feedback, and Dr. Hatem Alkhamra from Qatar University, College of Education his assistant in improving the communication scenario. 7 References [1] A. P. Association, Diagnostic and statistical manual of mental disorders. Washington, DC, 2013. https://doi.org/10.1176/appi.books.9780890425596 [2] C. for D. C. and P. (CDC), “Autism Spectrum Disorder (ASD), Data & Statistics.” [Online]. Available: http://www.cdc.gov/ncbddd/autism/data.html. [Accessed: 01-Jan- 2016]. [3] U. Lahiri, K. C. Welch, Z. Warren, and N. Sarkar, “Understanding psychophysiological response to a Virtual Reality-based social communication system for children with ASD,” in 2011 International Conference on Virtual Rehabilitation, ICVR 2011, 2011. https://doi.org/10.1109/icvr.2011.5971841 [4] U. Lahiri, E. Bekele, E. Dohrmann, Z. Warren, and N. Sarkar, “Design of a virtual reality based adaptive response technology for children with autism.,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 21, no. 1, pp. 55–64, 2013. https://doi.org/10.1109/TNSRE. 2012.2218618 [5] S. Kuriakose, S. Kunche, B. Narendranath, P. Jain, S. Sonker, and U. Lahiri, “A step towards virtual reality based social communication for children with Autism,” in CARE 2013 - 2013 IEEE International Conference on Control, Automation, Robotics and Embedded Systems, Proceedings, 2013. https://doi.org/10.1109/care.2013.6733744 [6] R. W. Picard, “Future affective technology for autism and emotion communication.,” Philos. Trans. R. Soc. Lond. B. Biol. Sci., vol. 364, no. 1535, pp. 3575–3584, 2009. https://doi.org/10.1098/rstb.2009.0143 [7] G. Pioggia, M. L. Sica, M. Ferro, R. Igliozzi, F. Muratori, A. Ahluwalia, and D. De Rossi, “Human-robot interaction in autism: FACE, an android-based social therapy,” in Proceedings - IEEE International Workshop on Robot and Human Interactive Communication, 2007, pp. 605–612. https://doi.org/10.1109/roman.2007.4415156 [8] N. Parés, P. Masri, G. van Wolferen, and C. Creed, “Achieving dialogue with children with severe autism in an adaptive multisensory interaction: the ‘MEDIAte’ project.,” IEEE Trans. Vis. Comput. Graph., vol. 11, no. 6, pp. 734–43, 2005. https://doi.org/10.1109/ TVCG.2005.88 [9] E. Ahlsén and V. Geroimenko, “Virtual reality as a communication aid for persons with aphasia,” in Proceedings of the 2nd European Conference on Disability, Virtual Reality and Associated Technologies, 1998, pp. 229–235. [10] Worldviz, “Vizard Virtual Reality Software Toolkit,” 2016. [Online]. Available: http://www.worldviz.com/vizard-virtual-reality-software/. [Accessed: 01-May-2016]. [11] C. Volioti, T. Tsiatsos, S. Mavropoulou, and C. Karagiannidis, “VLEs, social stories and children with autism: A prototype implementation and evaluation,” Education and Information Technologies, Springer New York LLC, 2015. iJIM ‒ Vol. 11, No. 2, 2017 157 Paper—Immersive Virtual Reality in Improving Communication Skills in Children with Autism [12] F. G. . Happé, “The role of age and verbal ability in the theory of mind task performance of subjects with autism.,” Child Dev., vol. 66, no. 3, pp. 843–855, 1995. https://doi.org/10.2307/1131954 8 Authors O. Halabi is with Department of Computer Science and Engineering, Qatar Uni- versity, P.O.Box 2713, Doha, Qatar, (ohalabi@qu.edu.qa). S. A. Elseoud, is with the Faculty of Informatics and Computer Science, The British University in Egypt, El Sherouk City, Cairo, Egypt, (Samir.Elseoud@bue.edu.eg). J. M. Alja'am is with is with Department of Computer Science and Engineering, Qatar University, P.O.Box 2713, Doha, Qatar, (jaam@qu.edu.qa). H. Alpona is with is with Department of Computer Science and Engineering, Qa- tar University, P.O.Box 2713, Doha, Qatar, (ha1106514@student.qu.edu.qa). M. Al-Hemadi is with is with Department of Computer Science and Engineering, Qatar University, P.O.Box 2713, Doha, Qatar, (ma1001810@student.qu.edu.qa). D. Al-Hassan is with is with Department of Computer Science and Engineering, Qatar University, P.O.Box 2713, Doha, Qatar, (da1003669@student.qu.edu.qa). This article is a revised version of a paper presented at the BUE International Conference on Sustainable Vital Technologies in Engineering and Informatics, held Nov 07, 2016 - Nov 09, 2016 , in Cairo, Egypt. Article submitted 19 December 2016. Published as resubmitted by the authors 20 February 2017. 158 http://www.i-jim.org iJIM – Vol. 11, No. 2, 2017 Immersive Virtual Reality in Improving Communication Skills in Children with Autism