PAPER A PICTORIAL MOBILE APPLICATION FOR IMPROVING COMMUNICATION SKILLS IN NON-VERBAL AUTISM. A Pictorial Mobile Application for Improving Communication Skills in Non-Verbal Autism. http://dx.doi.org/10.3991/ijim.v9i4.4699 M. Samir Abou El-Seoud1, AbdelGhani Karkar2, Jihad M. Al Ja'am2, Omar H. Karam1 1The British University in Egypt (BUE), Cairo, Egypt 2Qatar University, Doha, Qatar Abstract—It is estimated that as many as 25 percent of indi- viduals living with autism spectrum disorders are non- verbal. That is, they cannot functionally communicate with others using their voice. Despite that substantial fraction, we still know very little about these individuals, their abilities, and their needs. "We still know very little about the cogni- tive capabilities of nonverbal people with autism, and how best to help them learn to communicate," said Geri Dawson, Ph.D., Autism Speaks chief science officer. Non-verbal people with autism are usually unable to com- municate normally using natural languages. They can, however, learn to communicate through specific symbols and images. Special education instructors have adopted this method of communication to teach non-verbal people with autism. They introduce the symbols and images to them through different methodologies. This learning pro- cess appeared to be effective but it is very long. The process is carried out manually and requires a lots of times, dedica- tion, and resources. The instructors should find the materi- als in different formats and circumstances. They should repeat the lessons several times and normally in a face- to-face framework. We propose in this paper a mobile- based application that allows non- verbal people with autism to learn and communicate with their surroundings using a smart device. They can then be taught to use specific symbols and images through the smart mobile phones. They can form simple words and sentences to express their feelings and needs. The application is flexible and allows the addition of new contents very easily. To assess the progress of the users, different exercises and puzzles are proposed. These allow the users to improve their skills and to continue learning outside the classrooms. Index Terms—components: Learning, Autism, Mobile-based Application, Smart Mobile phones, Non-verbal. I. INTRODUCTION Around 10% of the world’s populations have been identified with some kind of disability. This percentage is estimated to rise to 32% by the year 2026. People with impaired mobility, the visually impaired, those with speech difficulties, or who have a hearing impairment, can become totally isolated from the digital-based society as obtaining information from their surroundings becomes more and more difficult and problematic. These people have, however, at least one way to communicate with others (i.e., it could be head movements, eye blinks, voice intonations, and mimics) that could be enhanced with assistive technology and mobile-based applications. Experts recommend that people with disability should be given access to special computer applications and devices to avoid a digital divide between those who have access to IT and those who do not [5]. One of the strangest aspects of nonverbal autism is the fact that no one really knows why some people with au- tism can't - or don't - use spoken language. True, some people with autism also have childhood apraxia of speech, a neurological disorder that makes spoken language ex- tremely difficult. But many nonverbal individuals on the autism spectrum don't have apraxia -- they just don't speak. Clearly, there are differences in brain function that inhibit spoken language, but at this point there is no agreement on just what those differences are or how they impact any given individual. There are many techniques for encouraging and im- proving spoken language for children with autism, though there is no guarantee that any particular approach will be effective for any given child. Research suggests that speech therapy, behavioral interventions, and even play therapy can improve verbal communication. Some early research also suggests that music therapy and related tech- niques can make a positive impact on speech. Research about people with disabilities has been done primarily in western countries. Governments and private sector companies supported these research projects through specific funds, taxes and grants. Hundreds of researchers were allocated to conduct deep research to build assistive tools for people with disabilities and help them to integrate the society Some of the important projects for people with disabil- ity are: “AmlVital” [4] launched at the University of Gra- nada (2007) which creates devices for elderly and other people with disability; Archimedes [10] which promises accessible technology for the disabled; Chelin et al. [5] proposed a system that uses natural language to assist visually handicapped people in writing compositions. Alper and Raharinirina [2] reviewed literature on assis- tive technology for people with disabilities and offered guidelines for future researchers in this area. First, each person should be given an individual assessment before the assistive technology is given to him/her and the per- son’s needs should be taken into consideration. Second, family involvement in the learning process is critical if the person is to gain maximum benefit from the proposed technology. Jemni and Elghoul [6] presented an untested web based tool called WebSign that translates text to sign language. ALFakheer from King Saud University has announced the development of an educative web- based tool named FAHEEM for non-verb persons with Autism. However, applications are in the testing phase. In [1] a system has been proposed to allow children with iJIM ‒ Volume 9, Issue 4, 2015 49 PAPER A PICTORIAL MOBILE APPLICATION FOR IMPROVING COMMUNICATION SKILLS IN NON-VERBAL AUTISM. disability to learn about new objects/entities by tapping on physical objects through a specially designed RFID- Bluetooth based Tangible User Interface (TUI) tool. In [7] an assistive computerized system has been proposed using the Arabic language for children with intellectual and learning disabilities that are resident at Shafallah Center in Doha, Qatar. The system is flexible and can be used by parents, children and teachers, where they can employ the materials according to specific needs. In [8] a picture- based communication system has been developed to teach Qatari children with autism and limited capability of ex- pressions by means of digital images and pictures. This system was desktop-based and specific to the Shafalah centre. We mention also Mechling [9] has conducted a literature survey (1990–2005) on research on assistive technologies as self-management tools for prompting students with intellectual disability to initiate and com- plete daily tasks. She mentioned that “although Kimball, Kinney, Taylor and Stromer (2003, 2004)” outline instructions on how to create computer-based activity schedules with photographs and video models using Power Point, to date no research base exists to support this new and creative use of high tech systems in provid- ing students with visual, auditory, and animated cues for following and transitioning between activities or use in other forms of self-management. Teaching people with autism is a challenging task. In- structors spend many days to explain to non-verbal people with autism the meaning of images, symbols and photos. They select different shapes and models of the same im- age and show them daily and repeatedly to the autistic people. These images are often taken from the Broadmaker software and the instructors use specific tools to materialize these images (i.e., generation of glossy cards, power point slides, etc.). Figure 1 shows some images taken from the English-based Broadmaker software. Few research projects are carried out in the Arab world for people with disability and the obtained results are moderate [3]. In addition, most of the Arabic software engineering companies in the MEAN region do not de- velop educational applications for special users since they do not see great financial benefits from this market. In this paper, we propose a new mobile-based applica- tion that allows the non-verbal people with autism to learn and communicate using a smart device (i.e., iPhone, iPad). We can then improve their understanding and communication skills and break their isolation. Whenever the non-verbal persons understand very well the meaning of symbols and images, they will be asked to form simple sentences and select the correct images to initiate and complete a specific activity (i.e., go to school, select items for lunch, etc.). The proposed application supports inclu- sion, data exchange, communication and creativity by making communication more accessible and more cost- effective. The application provides ways to choose images and symbols, and their corresponding words to express moods, needs, and thoughts. It is oriented in particular to satisfy the needs of non-verbal persons in countries using Arabic language as medium of communication. It allows the users to from words using images and sym- Figure 1. Images from the Broadmaker Software Figure 2. Distribution of language abilities/disabilities in the Egyptian population bols. The application can help persons who struggle to speak properly due to several conditions such as non- verbal, autism, brain injuries, cerebral palsy, intellectual disability, and stroke to communicate. The application has an educational section which gives the non-verbal the ability to learn new words through reading simple Arabic sentences by getting the corresponding images using a multimedia database and online search engines (i.e., google, yahoo). The application allows the non- verbal people to become more independent and to be better integrated within the sociality, with the ability to communicate whenever they need and in different cir- cumstances. Figure 2 below shows the percentage of the population in Egypt who are Arabic/English speaking, illiterates, disturbed or having a problem in speech communication, and those who are with intellectual and hearing impair- ments. II. OBJECTIVES The following questions have motivated us to propose this research paper: • How can we use technology to help non-verbal? • What are the effects of images, sounds and animation on people with developmental or intellectual disabili- ties? • Can we increase their thinking abilities and skills? • Can technology facilitate communication for people with intellectual disabilities? • Can people with such kind of disabilities be integrat- ed in their society? 50 http://www.i-jim.org PAPER A PICTORIAL MOBILE APPLICATION FOR IMPROVING COMMUNICATION SKILLS IN NON-VERBAL AUTISM. • Can they find suitable jobs? • Can they be productive? • Can we decrease their isolation from the digital age? • Which programs and tools should be offered to them? • How can we help them to express their feelings? • How we can assist them to communicate more effec- tively with their surroundings? Digital inclusion is a concept much talked about these days, but what software and services do people with disa- bilities really need if they are to take part in our infor- mation rich society? In Egypt, there exist many children who suffer learning disability and attention deficit hyperactivity disorder. Thus, it is time to recognize the problem and to propose solutions. The long term research intention at the BUE to use technology to develop augmented communication sys- tems for people with intellectual and learning disabilities is under consideration. Such systems should support col- laboration, data exploration, communication and creativi- ty. The broad objectives of such research could be summa- rized as follows: 1. To develop an Arabic tutorial-based system for peo- ple with disabilities other than non-verbal. We can then unveil the hidden potential within these people and improve their skills. 2. To enhance the thinking and memorization skills of these people. We could propose puzzles of different levels associated with the tutorials. People with disa- bilities can try to solve them and improve their think- ing, writing and memorization skills. 3. To assist the disabled to improve their understanding of electronic text and dialogue. For example, we could try to develop new algorithms and techniques to extract keywords from electronic text (i.e., web sites). We propose to use concepts analysis and Natu- ral Language Processing. We envisage studying other techniques, such as Latent Semantic [Ibrahim et. al, 2007]. 4. To explore the e-learning/M-learning experiences of disabled learners. There are many complex issues and interactions introduced by disabled learners’ re- quirements for accessible e-learning, compatible as- sistive technologies and effective learning support. Assistive technologies should be considered. 5. To make BUE a center of excellence for applied re- search in the field of special education for people with special needs in Egypt and the Arab world. Thus, research can be sustained in Egypt. In addition, our work can benefit others who have the same con- ditions in other countries. Research about people with disabilities has been done primarily in western countries. Governments and private sector companies supported these research projects through specific funds, taxes and grants. Hundreds of researchers were allocated to conduct deep research to build assistive tools for people with disabilities and help them to integrate the society. Very few research projects are carried out in the Arab world for the disabled people and the obtained results are moderate (Al Gain & Al Ab- dulwahab, 2006). In addition, most of the Arabic software engineering companies do not develop programs for users with special needs because they do not see financial bene- fits from this market. It should be kept in mind that Technology can provide motivation, perhaps by allowing success where past expe- riences have only resulted in failure, or by increasing the user's self-esteem. On the other hand, technology is criti- cal because of the rates of poverty among the disabled family. Many families of children with special needs can- not afford adaptive technology; even though it might help their children obtain a better education. In addition, many schools do not allocate the necessary funds to purchase equipment to help children with disabilities. The objective of this paper is restricted to enhance the communication skills of Arabic speaking people with autism. We have gathered the related information about these people from existing documents and from several interviews with stakeholders (i.e., families and teachers). The technical objective is to develop, maintain, and enhance an easy-to-use augmentative communication mobile-based application. The target people are those with autism. In addition, it can also be used by non-verbal Arabic speaking people. III. SERVICES DELIVERED BY THE APPLICATION The application allows non-verbal people better inclu- sion within the community using the state-of-the-art technology. This can certainly unveil the hidden potential within these individuals. It uses augmentative communi- cation technique through a smart mobile device, which can help non-verbal people to communicate with their surroundings effectively. The application can also contribute to the education and employment of non-verbal people in the increase of livelihood opportunities. It is developed in an open archi- tecture where code and contents are totally separated. Thus, teachers can update the contents whenever they need. They may also create specific folders for every per- son according to her/his needs. IV. PROCEDURES In order to fulfill the aforementioned tasks, we built first the general artwork of the application, set the graph- ical user interface and design the structure of the database that will store the links to the information (i.e., symbols, sounds, and images). We tested this application with chil- dren with autism to explain to them simple Arabic stories through images, photos and short clips. We have gathered 20 simple stories for children and annotated manually their text. This annotation process link words in the story with their corresponding images. These images were col- lected from different sources including specific CDs and the Internet. The application is developed based on the techniques of software engineering as depicted in Figure 3a and Figure 3b. We have also developed a component that can annotate simple Arabic sentences automatically. The objective is to get as input an Arabic sentence and convert it automatical- ly to its corresponding images and photos. Thus, the non- verbal people can understand these sentences in an easy way. In addition, the component allow the potential users to form Arabic sentences by selecting images and symbols and the component will generate the corresponding sen- tence. iJIM ‒ Volume 9, Issue 4, 2015 51 PAPER A PICTORIAL MOBILE APPLICATION FOR IMPROVING COMMUNICATION SKILLS IN NON-VERBAL AUTISM. a. Software Engineering Process b. Software Engineering Process Figure 3. A new module of reading loudly the sentence is being developed. This module will allow direct communication between verbal and non-verbal people. They can exchange their ideas through texts and images by using the smart phones. The developed tutorials have been made attractive by using multimedia technology (i.e., Flash, Dreamwaver, Studio). The advantages of a multimedia tutorial to devel- op the skills of non-verbal people with autism are as fol- lows: 1. It is an audiovisual and interactive medium. 2. The treatment or situation can be reproduced, i.e., the same condition can be repeated several times. 3. It can be adapted to the individual needs. 4. It has an effect on more than one sense, and can be more effective. 5. It can help creativity. In addition, the user feels the success. 6. It can include “puzzles and games” to add to a chil- dren’s motivation to use the software. 7. It can be used both in individual and in small-group lessons. 8. Parents can use it with success and participate active- ly in the learning process. 9. Most important thing is that the children should get interested and their interests are sustained for long periods of time. This is not an easy task, but multi- media presentations are very effective in this aspect. The feedback from the users will be taken into consid- eration to improve the general interface of the system and the tutorials. The content of the tutorials and its corre- sponding puzzles will be tested at the end of each tutori- al/puzzles. In addition, we will make a comparison be- tween the non-verbal students who used the system versus those who don't. This comparison will help us to evaluate the effectiveness of the system. V. ARABIC VERSION Arabic icons corresponding to Arabic words are de- veloped as shown in the below snapshot. Every icon show the action and the corresponding word like drink, may I, flower, banana. We have also added voice to every icon. The feedback of special education instructors is very promising. They liked very much the application as it facilitates their works and enhanced the learning process. The non-verbal people feel better and can express their thoughts in a good way. Below are 2 snapshots of Arabic icons screens: Arabic Icon Screen 52 http://www.i-jim.org PAPER A PICTORIAL MOBILE APPLICATION FOR IMPROVING COMMUNICATION SKILLS IN NON-VERBAL AUTISM. Application Screen VI. TECHNICAL DETAILS A. Modules The developed application is composed of four compo- nents as shown in Figure 4 below. The DataStore which is responsible about getting stored information from the database. The Parser which is taking charge of loading the annotated stories, process them, and get relevant images. The SearchEngine which pertains online search to get additional images. The MultimediaLoader which is re- sponsible about loading multimedia items (i.e., symbols, sounds, and images) and display them. Figure 4. System modules B. Whole System Architecture The new system is composed of two parts: an Arabic language processing server and the mobile application as shown in Figure 5. The server is responsible for providing educational services for the mobile application (i.e., pro- cessing Arabic story, loading online images, etc.). The mobile application is a web based application that does not require any connection with the server. The mobile application is created using the HTML 5 technology and connects the server only if the user intended to use the educational section. Figure 5. System architecture C. Educational Model Architecture We have created a database that stores simple Arabic stories. We have annotated diverse useful information (i.e., location, actors, tools, etc.) that are helpful for non- verbal people and can be highlighted automatically while reading. We have collected information about words in contrastive domains (i.e., foods, animals, tools). Figure 6 illustrates the structure of animals information. Figure 6. Animal information structure VII. APPLICATIONS SCENARIOS The system provides a coherent and flexible user inter- face that can be used easily with all users and allows them to navigate easily. Below are 4 different snapshots of the system: A. Snapshot-1 The user can first choose the first icon on the right for “possible” (Arabic: Momken), thereafter click the second icon “drink” (Arabic: Ashrab). Lastly, the user would choose one of the other icons to indicate the type of juice to drink. DataStore Parser Search Engine Multimedia Loader Pars dia r System iJIM ‒ Volume 9, Issue 4, 2015 53 PAPER A PICTORIAL MOBILE APPLICATION FOR IMPROVING COMMUNICATION SKILLS IN NON-VERBAL AUTISM. B. Snapshot-2 After the non-verbal participant finishes the experi- ment, a general feedback will be discussed. C. Snapshot-3 We get this layout by clicking on the icon on the upper row in snapshot-2, namely the boy playing the football. As see in this snapshot, this icon is the middle icon. To get snapshot-2, click the icon on the upper row, namely the boy eating. This icon is the icon on the first row down at the most left corner. D. Snapshot-4 Here the user first chooses the first icon “I” (Arabic: Ana). The user then chooses the second icon “want” (Arabic: Aayes). Thereafter, the user chooses one of the three icons listed below to indicate the action to be taken, namely “eat” (located at the right), “drink” (located at the middle), “bath or WC” (located at the left). E. Snaphost-5 Here the user can read stories and click on the high- lighted word “Ant” (Arabic: Namla) inside the educational section. Then he will get couple information about it with imag- es. VIII. CONCLUSION This paper focuses on innovative practices directly re- lated to non-verbal persons with Autism. The developed system will allow the non-verbal person to learn and communicate using a smart device. The suggested mo- bile application will provide ways to choose images and symbols and their corresponding words to express moods, needs, and thoughts. The developed system will provide !!"#! !"#$ !"#$ !!!" !"" !"# 54 http://www.i-jim.org PAPER A PICTORIAL MOBILE APPLICATION FOR IMPROVING COMMUNICATION SKILLS IN NON-VERBAL AUTISM. direct links for interactions between the static content (i.e., text, images) and the dynamic content (animations and games) and in doing usability studies on the devel- oped user interface to comprehensively assess the applica- tion. REFERENCES [1] Ali Karime et al., “RFID-Based Interactive Multimedia System for the Children”, In the International Journal of Multimedia Tools and Applications, Springer, (ISSN: 1380-7501, Print Version, Journal no. 11042, DOI: 10.1007/s11042-011-0768-3), March 2011. http://www.springerlink.com/content/h4r7470g15 58vv5n/. [2] S. Alper and S. Raharinirina, “Assistive technology for individuals with Disabilities: A Review and Synthesis of the Literature”, Journal of Special Education Technology, 21(2), 47-65, 2006. [3] S. I. Al Gain and S. S. Al-Abdulwahab, “Issues and Obstacles in Disability Research in Saudi Arabia”, Prince Salman Centre for Disability Research Riyadh, Kingdom of Saudi Arabia. AmlVital Project 2007. http://www.innovationsreport.com/html/report s/ communication_media/report-84220.html [4] J. Chelin, L. Kosseim and T. Radhakrishnan, "Using Natural Language Processing to Assist the Visually Handicapped in Writ- ing Compositions", Canadian AI, LNAI 4013, pp. 300-311, 2006. http://dx.doi.org/10.1007/11766247_26 [5] M. Jemni and O. Elghoul, "Towards Web-Based Automatic Inter- pretation of Written Text to Sign Language". In the Proceedings of ICTA’07, April 12-14, Hammamet, Tunisia, pp. 43 – 48, 2007. [6] Jihad Mohamad Alja'am et al., “An Assistive Computerized System for Children with Intellectual and Learning Disabilities”, Lectures Notes in Computer Sciences (Part-I),, Springer Verlag, Ambient Assistive Health and Wellness Management in the Heart of the City - ICOST 2009, pp. 9–16, 2009. http://www.springerlink.com/content/y6536r0233287022/. [7] Jihad Mohamad Alja'am, Osama Al-Ejji, Nasser Al-Khuzaei, and Abdulrahman Al-Rumaihi, “A Picture-Based Communication Sys- tem for Non-Verbal Qataris Children with Autism”, In the Pro- ceedings of The World International Conference on Technology Support Education, WICTE’ 2010, Beirut, Lebanon, December 2010. [8] Mechling L., "Assistive Technology as a Self-Management Tool for Prompting Students with Intellectual Disabilities to Initiate and Complete Daily Tasks: A Literature Review". In Education and Training in Developmental Disabilities, 42(3), pp. 252-269, 2007. [9] Neil G. Scott, Project Archimedes at Stanford University. http://archimedes.standford.edu. [10] Stanford Parser: http://nlp.stanford.edu/software/lex-parser.shtml AUTHORS M. Samir Abou El-Seoud is with the Faculty of In- formatics and Computer Science, The British University in Egypt – BUE (samir.elseoud@bue.edu.eg). AbdelGhani Karkar is with the Dept. of Computer Science and Engineering, Qatar University, Doha, Qatar (a.karkar@qu.edu.qa ). Jihad M. Al Ja'am is with the Dept. of Computer Sci- ence and Engineering, Qatar University, Doha, Qatar (jaam@qu.edu.qa ). Omar H. Karam is with the Faculty of Informatics and Computer Science, The British University in Egypt – BUE (omar.karam@bue.edu.eg). Submitted 06 May 2015. Published as resubmitted by the authors 20 August 2015. iJIM ‒ Volume 9, Issue 4, 2015 55 iJIM – Vol. 9, No. 4, 2015 A Pictorial Mobile Application for Improving Communication Skills in Non-Verbal Autism.