International Journal of Interactive Mobile Technologies(iJIM) – eISSN: 1865-7923 – Vol 16 No 10 (2022) Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia https://doi.org/10.3991/ijim.v16i10.29831 Karim Q. Hussein(), Maha A. Al-Bayati College of Science, Mustansiriyah University, Baghdad, Iraq karimzzm@yahoo.com, karim.q.h@uomustansiriyah.edu.iq Abstract—This research aims at developing an e-spelling system supported by effective multimedia technique of sign language and finger spelling. However the aimed system should enhance the hearing disability for the Deaf student so he can enter his desired paragraph then the spelling process could be accomplished in two stages, the first stage is the word spelling which translates each word into the corresponding sign language video clip (MPEG video cut). Then stopping message appears asking the Deaf student to move to the second stage of translat- ing. The second stage is the letter spelling which translates the current translated word (in sign language) into finger spelling letter by letter in sequence. Both stages of spelling are controlled by the Deaf student. After completing translation of that word into alphabets finger spelling, the next word would be translated into the corresponding sign language video clip. The system has been showed/ tried by some experts to get the valuable feedback from them. As final result it is expected that Deaf students can improve their skills in English reading when they practice, drill and repeat the translating and spelling process of various English sentences and words. Keywords—e-Learning modules, deaf, ASL one-hand American alphabets finger spelling, sign language, reading skills, visual multimedia 1 Introduction Deaf students communicate external sources via visual perception sense. Visual multimedia approach can compensate the audio learning sources. Idea of the current research represents development of an effective Multi-Modes system to translate all the activities of the audio spelling reading skills into the corresponding visual objects. For words translate each word to the corresponding sign language clip, for the letter to translate the concern letter into the corresponding one hand alphabet finger spell- ing. As per the previous approach the authors planned to develop the system into two sequential stages “which would be illustrated by details in the next few pages” from the first word in the sentence, the word is translated into the corresponding sing language clip, then the Deaf student manually hit to next action, it is translating the word letter by letter into finger spelling. The authors tried to create effective control by the Deaf iJIM ‒ Vol. 16, No. 10, 2022 67 https://doi.org/10.3991/ijim.v16i10.29831 mailto:karimzzm@yahoo.com mailto:karim.q.h@uomustansiriyah.edu.iq Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia student word by word and letter by letter. Thus it is expected an active tracing by the Deaf student on the spelling and reading skills. 2 Related work Some selected related work has been considered because those researches have direct relation with the contents of the current paper. The related papers are listed below: i. Karim Q. Hussein, “Evaluating an e-Dictionary for Hearing Impaired Persons Through Case Study (The Effective Role of Speech Visualization Multimedia),” This paper presents an applied e-dictionary which translates any text into multiple modes of communication (sign language and finger spelling simultaneously). To evaluate the dictionary a case study of real HI students used this dictionary, many sessions were held in school of deaf and dumb and a valuable feedback obtained from these experiments [7]. ii. Dr. Karim Q. Hussein, “DEVELOPING AND EVALUATING A MULTIPLE TASK ARABIC E-DICTIONARY FOR ARAB DEAF PERSONS” The author tried to develop effective dictionary for Arab Deaf persons. The dictionary can translate the word into its corresponding sign language. However simultaneously the word under translation is displayed with different color, size and underlining. Thus the deaf person can trace process of translation. Besides effects on the word under process, the corresponding meaning in English language is displayed too. some selected case studies of run-out screens have been displayed and describe the needful details so as to explain the mechanism of the dictionary [8]. iii. Dr. Karim Q Hussein, “AUTHORING SYSTEM OF DRILL & PRACTICE ELEARNING MODULES FOR HEARING IMPAIRED STUDENTS”, Hearing Impaired (HI) persons need to keep on practicing and repeating their lessons as well as their exercises. Teaching methodology of HI students differ than normal students. HI students need to be involved in practicing more and more using their modes of visual communication like sign language to cover their audio disability. Teaching methodology of HI students recommends demonstration and repeating with slow presentations of instructional material. A teacher displays his lesson directly face to face without visual noise. More reinforcement and encouragement to HI students, fun & enjoyment should be strongly included in the e-lessons as well as continuous interaction between teacher and HI students. As per previous factors the decision of researchers is to develop Drill & Practice (D&P) e-learning modules (eLMs) for selected topics like Mathematics. D&P eLMs of Mathematics for HI persons would be the case study of this research including Developing & Evaluating [14]. iv. Dr. Karim Q. Hussein, Dr. Maha A. Al-Bayati, “Comparative Study Between e-One Hand Finger Spelling and e-Sign Language for Hearing Impaired Persons Regard- ing Day-to-Day Activities Using an Object Oriented Approach”. Day-to-day activities are important tasks for Hearing Impaired (HI) persons, par- ticularly when HI persons communicate with others for normal life activities. To develop a modern technology through software development supporting HI persons 68 http://www.i-jim.org Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia is a challenge, because many factors affect such development like visual multimedia components, software engineering fundamentals, e-Learning principals for HI stu- dents, strategies of teaching HI students, modes of communication HI students, visual programming application as well as Object Oriented Approach. However, in the cur- rent research, the authors tried to consider all the previous factors so as to develop two different e-learning systems for HI students. The first is to simulate real world One-Hand-Finger Spelling (OHFS), and the second simulates Sing Language (SL), then applying three experiments for both systems on real HI class school. Results of the experiments were discussed accordingly. However, HI persons communicate one another through multiple kinds of communication like Lip Movement Reading, OHFS, Two Hands Finger Spelling and SL. In the two computer systems, Database system for displaying related video cuts and linking them with words is used. Around 3000 video cuts have been used with corresponding multimedia components via Visual Basic coding, about OHFS. Also similar Database System connected 54 images for OHFS to display spelling of images for each letter in the word by sequence. About Day-to-Day activities, common and most usable activities have been considered to be selected from the vocabulary of video cuts. An Object Oriented Approach is considered strongly to allocate each similar word in selected class. Five students have tried the two systems and tested three major experiments; However, results have been booked and analyzed accordingly. Final conclusion approved the efficiency of SL, the successful O.O.A in teaching SL and OHFS for HI students as well as both are friendly learning tools [15]. 3 Problem statement The problem could be represented by the following three questions: i. How the Deaf student can understand all words into related sign language clips? ii. How Deaf student can understand and represent all English letters into correspond- ing American Spelling Language (ASL) finger spelling? iii. As per the previous questions, the main problem could be formed as: How the Deaf student can improve and modify his skills in spelling and reading English sentences and texts by practicing the system? 4 Objective of research The research aims at: i. Developing a multi-mode of translation process for any word, and sentence into the corresponding sign language clip for the each word and corresponding one hand finger spelling for each letter in the translated word. ii. Applying the system in translating and spelling an experimental selected paragraph. iii. Evaluating the system by showing/trying the experimental spelling samples by the experts in order to discuss their valuable feedback. iv. Because the Multi-Mode e-Learning system is designed to cover all categories of the Deaf. therefore two modes of communication have been considered in this research. iJIM ‒ Vol. 16, No. 10, 2022 69 Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia 5 Translation technique of the system Two types of visual multimedia techniques are used as shown: i) Around 3000 different video clips of sign language which cover mostly the com- mon/public words in English language. If the word is not present in the vocabulary of sign language clips then that word is translated into finger spelling clips. ii) 42 Image files which cover all alphabets of English letters as well as numbers and special samples like >, <, %, &, @, (, etc. 6 Presentation of run-out forms of the system In the next few pages the sequential process of translation of selected English sen- tence would be displayed to show by details the stages, tracing and mechanism of the system. Video cut of the word Finger spelling of the word Fig. 1. 1st form, the deaf student enters his sentence to be translated and spelled 70 http://www.i-jim.org Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia Fig. 2. 2nd form, the word the is to be translated into video cut corresponding then into letter by letter The video clip of the word is displayed here, word by word in sequence. The picture file of the letter in the word is displayed here, The word is displayed via letter by letter in sequence Fig. 3. 3rd form, the word the is to be translated into letter by letter in sequence iJIM ‒ Vol. 16, No. 10, 2022 71 Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia Fig. 4. 4th form, letter spelling stage for the word the letter by letter in sequence (h) Fig. 5. 5th form. video cut of the word deaf 72 http://www.i-jim.org Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia Fig. 6. 6th form. one hand finger spelling of the word Deaf Fig. 7. 7th form. video cut of the word clever iJIM ‒ Vol. 16, No. 10, 2022 73 Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia Fig. 8. 8th form. one hand finger spelling of the word clever (v) 7 Analysis and discussion of comments of experts The system has been tried by some experts. Some of them are professional in English language teaching and others are professional in education of special needs and one of them is specialized in computer science. The authors summarized the feedback of the experts as shown: 1. The system is easy to be used by Deaf students. 2. The experts were interesting while they were practicing spelling process. As a result we can expect such system to be friendly learning to the Deaf student. 3. The system offers multiple modes of spelling method. Therefore it is expected to cover all categories of Deaf students. 4. Some experts asked if we can link lip movement video clips spelling. Actually the authors tried their best to obtain such software dictionary but according to their survey there is no available free lip movement software to be considered. 5. Some experts asked if the system could be published as a free web sit in the inter- net. This is a nice suggestion so that to universal learning tool for the Deaf. But it could be as future work. 6. Some experts confirmed the idea of drill and practice and more reparations could improve the skills of the Deaf students in English language particularly in reading words and sentences. 74 http://www.i-jim.org Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia 7. Some experts asked if there is facility of displaying the picture of the word under process of sign language translation. Deaf students depend completely on the visual features of word. 8. Some experts suggested to link an e-exam section so that Deaf student can test himself and evaluate his progress in reading skills. 9. Some experts suggested another e-spelling enhancing the current e-spelling system. That suggested e-spelling system is to be in opposite to the current e-spelling. It is to display the sign action of video cut, then Deaf would be asked to write the source word of that cut. Also to display the picture of the finger spelling of the letter and the Deaf student must write the letter of that picture file. 10. Some experts reconfirmed the successful effective role of visual multimedia in rep- resentations the meaning action of the word as well as the letters in each word. 11. An expert suggested to evaluate the system using Neural Network in matching between Dataset video clips and entry words to the system. 8 Conclusion The authors summarized their conclusion as shown: 1. The system realizes effective friendly tool for the Deaf students to improve their skills in English reading. 2. Multimedia techniques of translating and spelling processes are successfully rep- resent the audio/oral letters, words and sentences into their corresponding visual multimedia features of pictures and video clips. Therefore the system technically realizes his goals. 3. Due to the previous comments of experts and the facilities of the system to offer multiple modes of e-spelling, word by word spelling and letter by letter spelling. Therefore it is expected that this s-spelling system can support requirements of all categories of Deaf students to support and modify their skills in English language. 4. As per the above items and comments of Experts, we can insure that such system motivates Deaf students to carry out learning, translating and spelling via computer, thus we can say that the system helps in enhancing self-confidence of Deaf students. 5. Due to discussion of viewpoints of experts, Interface of screens were interesting and friendly, Thus Deaf students would be enjoyed on using the experimental modules. 6. It is important that the e-Learning systems for Deaf students can offer multiple modes of communications; because there are multiple categories of Deaf persons, each category has its own requirements for communication with each other. Lip movements reading was not covered in this research because the authors could not obtain the dataset of video cuts for lip movements reading. iJIM ‒ Vol. 16, No. 10, 2022 75 Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia 9 Future works and recommendation 1. As per the suggestions of some experts it is planned to publish this system as free instructional web site for reason of universal benefits. 2. As future work it is recommended to add the lip movement reading spelling to the couple of modes of communication (finger spelling and sign language). 3. A character’s real voice in the video clips of sign language, his sign language action and lip movement can together present an ideal environment for HI student so that HI student can enhance an improve his skills in English reading. This is a success- ful suggestion to modify the system. But the problem that the original source of Indian sign language dictionary is free from voice of characters. Therefore it is highly recommended to link the real voice of the characters of any future work regarding sign language vocabulary. 4. As future work it is suggested to add the picture of the word which is under pro- cess of translation into sign language video clip. So we must add new vocabulary includes around 3000 picture files and modify the source coding of the e-spelling system so that to display the picture besides the video cut. It mean that the video clip of the word and the picture which disrobes the meaning of the that word must be displayed simultaneously. 5. The authors recommended to apply this effective system for primary Deaf schools or even pre-primary schools particularly for Deaf students who learn English as first language. 6. Due to the accelerate developing in Mobile Computing, The authors suggest to develop the current project into Mobile Cloud Application using Android (XML for GUI, interface and JAVA for application coding with Cloud Technology to meet requirements of Big Data for thousands of MPEG Video Cuts). 7. Using ANN or CNN as a modern technique to analyze the Dataset (Training and Testing) then prediction of the entry words. 8. About dataset, the authors recommended in the future work, the researcher may try to create his own dataset to meet the needful status of the Deaf in the society of study. 9. As an idea of future work, the authors suggested to develop chatting system between Deaf persons and normal persons, using mobile cloud system. All modern theories ensure not to isolate disabled people in their own committees. Deaf people must be merged with their corresponding normal persons, chatting system using real time mobile cloud techniques can achieve that target. Deaf chat using sign language as well as one hand finger spelling in the other side normal person can chat in real time using his own oral speech aspect. It a matter of challenge but it an excellent plan for research. 10. Develop a 3D character to take the role of real time translation for sign language. It is important plan for research. 76 http://www.i-jim.org Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia 10 References [1] Ibrahim M. M. Al Emary & Karim Q. Hussein, “Anylyzing the Various Aspects Modules Fro the Hearing Impaired Students,” International Journal of Academic Research, Vol. 4, No. 3, May 2012. [2] Cooke-Plagwitz, Jessamine: Generic Software for Foreign Language Instruction: A Snapshot of Teachers’ Approaches. http://www.readingmatrix.com/conference/pp/proceedings2007/ cooke_plagwitz.pdf [3] Dotter, Franz: Computer for the Deaf (and Hearing-Impaired): Towards an Integrated Solu- tion from a Linguistic Standpoint, 1999–2004 bidok – Behindertenintegration Dokumenta- tion, Institut für Erziehungswissenschaften; Universität Innsbruck. http://bidok.uibk.ac.at/ library/dotter-comp4deaf-e.html [4] Ellis, Kirsten & Blaslki, Kathy: Children, Australian Sign Language and the Web; The Pos- sibilities. http://www.berwick.monash.edu.au/interaction/19952-2/pp6-nielson.pdf [5] Felzer, Laura: Research on How Signing Helps Hearing Children Learn to Read. http:// littlesigners.com/articlell.html [6] Hussein, Karim Q & Warnerkar, C. S: e-Learning Support to Hearing Impaired Persons, ISTE, Punjab 2-3 March 2007. [7] Karim Q. Hussein, Muna Abo-Darwish, & Khalid Al-Atiat, “Evaluating an e-Dictionary for Hearing Impaired Persons Through Case Study (The Effective Role of Speech Visualization Multimedia),” European Journal of Scientific Research, ISSN 1450-216X, Vol. 41, No. 4, pp. 646–652, 2010. [8] Karim Q. Hussein, “Developing and Evaluating a Multiple Task Arabic E-Dictionary for Arab Deaf Persons,” International Journal of Latest Research in Science and Technology, ISSN (Online): 2278-5299, Vol. 4, No. 1, pp. 56–61, January–February 2015. [9] Qabas Abdal Zahraa Jabbar, “Evaluating Model for E-learning Modules According to Selected Criteria: An Object Oriented Approach,” Computer and Information Science, Vol. 5, No. 5, 2012, Published by Canadian Center of Science and Education. https://doi. org/10.5539/cis.v5n5p69 [10] Smith, Deborah Deutsch & Luckasson, Ruth: Introduction to Special Education: Teaching in an Age of Challenge, Allyn & Bacon, 1992. [11] Jeong, Sang-Mok & Song, Ki-Sang: The Community-Based Intelligent e-Learning System, Advanced Learning Technologies, 2005. ICALT 2005. Fifth IEEE International Conference on July 2005, pp. 769–771. [12] Poobrasert, Onintra & Mguine, Brain: Knowledge Engineering in Multimedia Design and Computer Assisted Learning for Special Needs Training: Effectiveness, The 9th World Multi-Conference on Systemic, Cybernetics and Informatics July 10–13, 2005 – Orlando, Florida, USA. http://www.iiisci.org/sci2005/proceedingssci/vol8-2001.asp [13] Nasereddin, H. H., MMLSL: Modelling Mobile Learning for Sign Language. Of Engineering and Computer Science Research and Reviews in Applied Sciences, Vol. 9, No. 2, 2017, pp. 20267–20272. [14] Karim Q. Hussein, “Authoring System of Drill & Practice Elearning Modules for Hearing Impaired Students,” International Journal of Computer Science & Information Technology (IJCSIT), Vol. 7, No. 1, February 2015. https://doi.org/10.5121/ijcsit.2015.7113 [15] Karim Q. Hussein & Maha A. Al-Bayati, “Comparative Study Between e-One Hand Finger Spelling and e-Sign Language for Hearing Impaired Persons Regarding Day-to-Day Activities Using an Object Oriented Approach,” International Journal of Engineering and Innovative Technology (IJEIT), Vol. 4, No. 7, January 2015. iJIM ‒ Vol. 16, No. 10, 2022 77 http://www.readingmatrix.com/conference/pp/proceedings2007/cooke_plagwitz.pdf http://www.readingmatrix.com/conference/pp/proceedings2007/cooke_plagwitz.pdf http://bidok.uibk.ac.at/library/dotter-comp4deaf-e.html http://bidok.uibk.ac.at/library/dotter-comp4deaf-e.html http://www.berwick.monash.edu.au/interaction/19952-2/pp6-nielson.pdf http://littlesigners.com/articlell.html http://littlesigners.com/articlell.html https://doi.org/10.5539/cis.v5n5p69 https://doi.org/10.5539/cis.v5n5p69 http://www.iiisci.org/sci2005/proceedingssci/vol8-2001.asp https://doi.org/10.5121/ijcsit.2015.7113 Paper—Multi-Mode e-Learning System of Reading Skills for Deaf Students Based on Visual Multimedia [16] Hussein, Karim Q., “Evaluating System for E-Learning Modules for Hearing Impaired Students (Technical Criteria in Review),” Chapter 13: pages 259–273, Published by Lam- bert Academic Publishing, ISBN: 978-3-8433-9000-2, Publication Year: 2011. [17] Hyperlink for Strategies of Teaching HI Students, The University of Sheffield: Teaching Deaf and Hearing Impaired Students/Strategies. http://www.shef.ac.uk/disability/teaching/ hearing/5_strategies.html 11 Authors Assist. Prof. Dr. Karim Q. Hussein (Ph.D. in Computer Science), TITLES OF THESES: Ph.D. “Instructional Computer System for Hearing Impaired Persons”, From October 2016 till now: Assistant Professor, Department of Computer Science, Faculty of Science, Mustansiriyah University, Baghdad, Iraq. Teaching Postgraduate as well as Undergraduate. He is guiding many M.Sc. researches in the field of interest as well as Ph.D. activities guiding and examine. Area of research and teaching: Mainly e-learning particularly Multimedia and e-Learning for handicapped persons (Deaf & Blind), 3D Animation using (MAYA), Internet programming, Cloud Computing, Mobile Computing, Mobile Cloud Computing, Data Mining, Deep Learning and Authoring Systems. Dr. Karim Q. Hussein has published more than 52 scientific papers in the field of Computer Science, particularly in Multimedia, e-learning for handicapped students, Deep learning, 3D Animation, Mobile Computing and Mobile Cloud Computing and web design and application. Assist. Prof. Dr. Maha A. Al-Bayati (Ph.D. in Computer Science, 1998), TITLES OF THESES: Ph.D. “Genetic Algorithm – Based Robot Path Planner”, From Octo- ber 2019 till now: Assistant Professor, Department of Computer Science, Faculty of Science, Mustansiriyah University, Baghdad, Iraq. Teaching post graduate programs as well under graduate programs, for post graduate, “AI and Soft Computing (M.Sc. Course), Multi-Agent (Ph.D. Course)” for under graduate “Data Structure and Algo- rithms in C++, OOP, PROLOG, Computational Theory, Operating Systems, System Software, Distributed Information Systems, Computer Graphics, Artificial Intelligence, Microprocessors and Compiler design. Main area of research AI, Algorithms and e-Learning. E-mail: bayati_ma@yahoo.com; mahabayati@uomustansiriyah.edu.iq. Article submitted 2022-01-30. Resubmitted 2022-03-05. Final acceptance 2022-03-06. Final version published as submitted by the authors. 78 http://www.i-jim.org http://www.shef.ac.uk/disability/teaching/hearing/5_strategies.html http://www.shef.ac.uk/disability/teaching/hearing/5_strategies.html mailto:bayati_ma@yahoo.com mailto:mahabayati@uomustansiriyah.edu.iq