International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 14, No. 16, 2020 Paper—Assistive Mobile Application for Visually Impaired People 1 Assistive Mobile Application for Visually Impaired People https://doi.org/10.3991/ijim.v14i16.15295 Sriraksha Nayak, Chandrakala C. B. () Manipal Institute of Technology, Manipal, India chandrakala.cb@manipal.edu Abstract—According to the World Health Organization estimation, globally the number of people with some visual impairment is estimated to be 285 million, of whom 39 million are blind. The inability to use features such as sending and reading of email, schedule management, path finding or outdoor navigation and reading SMS is a disadvantage for blind people in many professional and educa-tional situation. Speech or text analysis can help improve support for visually-impaired people. Users can speak a command to perform a task. The spoken com-mand will be interpreted by the Speech Recognition Engine (SRE) and can be converted into text or perform suitable actions. In this paper an application that allows schedule management, emailing and SMS reading completely based on voice command is proposed, implemented and validated. The System hopes to provide blind people to simply speak the desired functionality and be guided there by the system’s audio instructions. The proposed and designed app is imple-mented to support three languages which are English, Hindi, and Kannada. Keywords—Mobile Application, visually impaired, E-mailing, Text-to-Speech conversion, Speech-to-Text conversion, Outdoor Navigation, Schedule manage- ment. Introduction The world is moving towards digitization, so are the means of communication. Vis- ually impaired people need help in their daily lives. Some of the challenges faced by visually impaired people can be solved by simple, low cost, support software running on android mobile phones. Voice is the main focus; it helps them to control their phone using their voice. In this study, we outline three important features to meet the basic needs of visually impaired users: 1. Converting words that they have spoken into text: In this application, the words that they have spoken are converted into text and are written in PDF format and can be used to send mail to selected id. 52 http://www.i-jim.org https://doi.org/10.3991/ijim.v14i16.15295 mailto:chandrakala.cb@manipal.edu Paper—Assistive Mobile Application for Visually Impaired People 2. Extracting the cash transaction details from the messages in the inbox and converting these text messages into voice messages: It is important to know about Cash trans- action details that are available as part of SMS in written form but visually impaired persons cannot read any details which are in written form. So, extracting all-cash transaction messages and converting them to voice forms of communication can help visually impaired people to know about cash transaction details. Mailing details is one more challenge for them. 3. Outdoor Travelling Guidance: Visually impaired persons rely on the help of others for their daily needs. Traveling in an outdoor environment and finding an independ- ent way is a very dangerous and challenging task for visually impaired people. This application helps them to find a free path for movement. The application is designed to support three languages Kannada, English, and Hindi. Everything in this application is done through voice command. A visually impaired person can make use of this application using their android mobile phones. The paper is organized as follows. Previous work related to the proposed application is summarized in Section 2. Main Goals of the proposed application are outlined in Section 3. UML diagrams and proposed system architecture are presented in Section 4. Result and Conclusion are described in Section 5 and Section 6 respectively. 2 Literature Review Piotr Kardys et al. [1] proposed an android application that supports visually im- paired people and partially sighted people to use their mobile phones. It allows users to make a call, to send and receive SMS, to use phone book and positioning or battery monitoring, through speech commands. Laviniu epelea, Ioan Gavrilu and Alexandru Gacsadi [2] proposed an application which assists visually impaired person. It uses sensors from smartphones and also the information received from some external sensorial modules. It uses text to speech mod- ules to communicate between smartphone and user. This application allows them to call, to get GPS guidance and Indoor guidance, checking the battery level, time and date. Milios Awad et al. [3] proposed an Intelligent Eye android application which assists blind people by providing light detection, color detection, object recognition, and bank- note recognition feature. Developed interface of the application is user- friendly. Kasthuri R et al. [4] proposed an android application which provides features like calling any contact, getting information on live weather reports, transport information and news update. It also guides blinds to travel in an outdoor region. Calling any contact is done through speech command. Through the speech input a user can control a mobile device to perform necessary feature, this speech command then converts to text. Using this application visually impaired people can. Apirak Hoonlor et al. [5] proposed an android application to help visually impaired people in food shopping and other product recognition independently. This is a camera- based application. It identifies the captured image using the UCap annotated image da- tabase. In this application sighted users can add images of the existing products in the iJIM ‒ Vol. 14, No. 16, 2020 53 Paper—Assistive Mobile Application for Visually Impaired People database. For initial testing and evaluation, the author used Infrastructure-as-a-Service on MS Azure cloud server. The visually impaired users can take the photo of a product using the smartphone camera and ask UCap to identify it. Naveen Kumar M et al. [6] proposed an android application that provided educa- tional based Chatbot for blinds and sighted people can also use this application. In this application, visually impaired people get the answer for educational based queries from Wikipedia. Visually impaired people ask questions using voice command and get out- put in both voice and text form. Shagufta Md.Rafique Bagwan and L. J. Sankpal [7] proposed an android application that helps visually impaired people to recognize objects, Detection of Direction of Max- imum Brightness and Color Detection. The result of image recognition is communi- cated to visually impaired people through pre-recorded verbal messages. The author used a hybrid algorithm which reduces computation time and detect object very fast. K Shankar, S Dhaarani, S Kaviya shree [8] proposed an android application that helps visually impaired people to take a snapshot of the things which they want to know and the texts in that snapshot will be converted into speech, using this application user can send and receive SMS. A.J.Kadam, Sharvari Awate, Saba Desai, Rajlakshmi Khese and Gauri Patange[9] proposed an android application to help visually impaired people to recognize and match scanned objects to a database of objects, capable of detecting major colors and locate directions of the maximum brightness regions in the captured scenes. Gowri G, Harish Kumar R, Gokulakrishnan S, Bovaz Prabhu B [10] proposed an android application which helps visually impaired people with call logs, call dialer, Notes, Battery level status, messaging features with inbox, sent items and Maps fea- tures. It is a voice assistant application. Users can send and receive messages through this application. Hussam Elbehiery, M. S. Abdel-Wahab [11] proposed an android application that has three features i.e. Pinout, Call, and Emergency SMS. Pinout will instantly launch a voice notification telling the user his exact location. The application will allow a user to call saved numbers or new persons by clicking the number. Shruti Parkhi, Dr. S. S. Lokhande, and N. D. Thombare [12] proposed an android application that helps visually impaired people. The main module explained in this jour- nal is to recognize and match scanned objects stored in a file on the mobile application using Image Processing and Artificial Neural Network algorithm. In his study authors had used Neural Network algorithm for edge detection, Sobel edge detection and for training Back Propagation. Using text to speech method, object recognized from the image is communicated to visually impaired people. Accuracies for color feature object, shape feature object and both are 94%, 67 and 90% respectively. U. Avanthika, Shyam Sundar, Silviya Nancy [13] proposed an android application for visually impaired people to listen to an audiobook with a handy books’ portal. In this paper, authors had proposed a completely new idea having a portal in which they can store audiobooks and they can use these books whenever they needed. To use this application, the user needs to install it in their smartphones and they can select which- ever book they need to hear. The application contains material related to the course and 54 http://www.i-jim.org Paper—Assistive Mobile Application for Visually Impaired People course-related books, it also contains books for leisure time which includes contempo- rary and classical novels. This application contains backup database which that will start working once the application is on. Initially, the user will get the instructions to select the domain and once the domain is selected it will give voice instruction of list of the books available in that domain Junar Arciete Landicho [14] proposed an android application for blinds and for hear- ing impaired peoples. Using this application, the blind user can hear messages and a deaf user can send messages from inbox. In inbox, it only shows unread messages, once the user reads those messages it will be removed from the inbox. This application also shows all messages sent by the user in the sent box and a deaf user may only forward these messages to others of his contact. This application also shows the contact name and number in alphabetical order. Both deaf and blind users can make use of this mod- ule to contact another person. Eclipse IDE is used to implement this application. Text- To-Speech (TTS) and Speech-To-Text (STT) built in libraries are used to convert text to speech and to convert speech to text. The novelty of the proposed application in this paper is a support for a local Lan- guage i.e. Kannada apart from Hindi and English languages and extracting only the transaction SMSs’ and converting it from text to speech to support visually impaired person. 3 Objective The main aim of the proposed work is to develop a mobile app for Visually Impaired Persons to work on a smartphone. The main objectives are: 1. A functionality to support emailing which involves speech to text conversion, send- ing email and support reading mail by converting text to speech. 2. A function to create audios of the cash transaction SMSs’ available as part of mes- saging service inbox. 3. A feature to assist path finding in the Outdoor region. 4. A schedule managing feature that involves marking events in the calendar. 4 Mobile Application Architecture In this section, the use-case diagram of the application which gives a brief insight into the various stakeholders and the main uses of the App. The main functional re- quirements of the application along with the behavioral aspects and the order in which interaction between the objects of the proposed system is illustrated using activity dia- gram and sequence diagram respectively. Tools and Technologies which were used in the proposed system are described at the end of this section. iJIM ‒ Vol. 14, No. 16, 2020 55 Paper—Assistive Mobile Application for Visually Impaired People 4.1 Use case diagram The mobile tool use case involves five actors: the visually impaired person, the GPS application, the TTS application, the Google database and the mobile database as shown in Figure 1. Each use-case in the use case diagram represents the functionalities of a system and it is represented by an oval shape in figure 1. 4.2 Functional requirement The main Functional requirements of the Application are as follows: • Speech to text conversion • Emailing function • Text to speech conversion • Marking Events • Find Independent pathway to support outdoor navigation Fig. 1. Use case Diagram of the Application 56 http://www.i-jim.org Paper—Assistive Mobile Application for Visually Impaired People 4.3 Proposed system architecture The Architecture involves three layers Interface layer, Logical Layer and Commu- nication Layer as shown in figure 2. Interface Layer consists of classes that are respon- sible for interaction with visually impaired users. Logical Layer consists of classes that are responsible for the implementation of a particular function and the communication layer consists of classes that are responsible for data exchange between application and operating system. On application launch, four processes are initiated. User interface processes are carried out in the Main Activity java class. Four processes of listening for transaction-related SMS, getting an Independent way in the outdoor region and mailing information, as well as managing schedule takes place simultaneously. Fig. 2. Architecture of Mobile Tool for Visually Impaired Persons Figure 3 illustrates the class diagram of the proposed system. The Speech recognizer class provides access to the speech recognition service. The speech input will be ini- tially sent to a remote server, which converts speech to text and the result will be sent back to the requested app. To convert speech to text, Recognizer Intent class is used. It takes Action Recognize speech method to start an activity that will prompt user for speech and send it through speech Recognizer Class. It also uses Extra Language Model and Language Model Free form methods. Extra Language Model informs recognizer which speech model to prefer while performing Action Recognize speech and Lan- guage Model Free form is a file used by speech recognizer to recognize speech. It con- tains large list of words and their probability of occurrence. Along with this it also uses iJIM ‒ Vol. 14, No. 16, 2020 57 Paper—Assistive Mobile Application for Visually Impaired People Extra Language which specifies which language to use and Extra Prompt this is a text prompt to show to user when asking them to speak. SMS Manager Class will extract all transaction-related messages from inbox and the Text to Speech class will convert the text form of SMS to speech. It will be useful for visually impaired persons to hear transaction related SMS. To extract SMS from inbox it uses Async Task abstract class which provide liberty to perform heavy task in back- ground. As android run on single thread it takes longer time to fetch the response and it will make app non responsible. To avoid this Async Task abstract class is used. It takes three methods like OnPreExecute, doInBackground and OnPostExecute. On- PreExecute method contains the code which need to be executed before background process starts. doInBackground method contains the code which need to be executed in background. OnPostExecute method is called after background method completes. Re- sult from background is passed to this method. Text to Speech class is used to convert from Text to Speech. This class takes three parameters. First parameter is text that would be spoken. Second one is a constant, it may be Queue Flush or Queue Add. Queue Flush will remove all previous inputs from playback queue and is replaced by new entry. In Queue Add, new entries are added to the end of previous Playback queue. Text to Speech class takes many methods some of the methods used in the project are setLanguage which is used to select language. Error method is used to find error. Stop method denotes stop requested by client. When the user speaks the destination address, the Location Manager Class will get the destination location from the Global Positioning System. It will guide the VIPs to find Independent ways in the outdoor Region. To add event to calendar, Intent Object is used. To add new event o user’s calendar ACTION_INSERT method is used and to specify other event detail EXTRAs are used like to specify event start and end time EXTRA_EVENT_BEGIN_TIME and EXTRA_EVENT_END_TIME is used respectively. To compose an email with an attachment, ACTION_SEND is used and to compose an email with multiple attachment ACTION_SEND_MULTIPLE is used and to specify other details EXTRAs are used. EXTRA_EMAIL consists of a string array of recipient email address. EXTRA_SUBJECT consists of a string with email subject. EXTRA_STREAM consists of Uri pointing to the attachment. EXTRA_TEXT is the body of the email. 58 http://www.i-jim.org Paper—Assistive Mobile Application for Visually Impaired People Fig. 3. Class Diagram Figure 4 illustrates the behavioral aspects of the proposed system. When the user opens the application, it first instructs the user to select the language in which they wish to operate. On selection of the language it redirects the user to a particular activity based on the speech input. Application supports three languages: English, Kannada, and Hindi. Instructions are given to navigate through different activities after the preferred language is selected. On Email module selection, the user needs to give details through voice command, then the application will convert speech to text and puts the generated text as the email body. On selection of SMS module, it will extract all SMSs related to transaction from inbox which will be in the form of text, TTS will convert SMSs’ text to speech so that visually impaired persons can hear all SMS related to transactions. Once the Remainder module is selected, a user needs to specify a date and an event. The application will store the event’s date and information in the mobile database and shows a notification on that date. On the selection of Outdoor Guidance module, users need to specify des- tination address, GPS will find the way to the destination location and will provide travel direction. iJIM ‒ Vol. 14, No. 16, 2020 59 Paper—Assistive Mobile Application for Visually Impaired People Fig. 4. Activity Diagram Figure 5 shows the sequence diagram of the Email functionality. Initially, the appli- cation will give instructions to the users and it asks the user to select one of the follow- ing modules and the modules are Email Module, SMS Module, Remainder Module, and Outdoor Guidance Module. When a user selects the Email module, the user needs to provide information to the system. Sent mail is stored in the Google database. 60 http://www.i-jim.org Paper—Assistive Mobile Application for Visually Impaired People Fig. 5. Sequence Diagram (Email Module) Figure 6 shows the sequence diagram of the SMS Module. When a user selects the SMS module, Transaction related SMS are extracted from a mobile database and dis- played in one activity. TTS will convert these text messages into speech message and visually impaired persons can easily hear the messages. Fig. 6. Sequence Diagram (SMS Module) iJIM ‒ Vol. 14, No. 16, 2020 61 Paper—Assistive Mobile Application for Visually Impaired People Figure 7 shows the sequence diagram of the Outdoor Guidance Module. When a user selects the outdoor guidance module, a user needs to speak the destination address. Application will search Destination address in Global Positioning System and it sends destination address location to the application. TTS will convert text information into speech and guide visually impaired persons to travel in an outdoor region. Fig. 7. Sequence Diagram (Outdoor Guidance Module) Figure 8 shows the sequence diagram of the Remainder Module. When a user selects the Remainder module, a user needs to speak the date and event. The application will store events date and information in the mobile database and shows a notification on that date. 62 http://www.i-jim.org Paper—Assistive Mobile Application for Visually Impaired People Fig. 8. Sequence Diagram (Remainder Module) 4.4 App development tool and technologies Tools and Technologies that have been used are as follows: Mobile Smart Phone, Android Studio, Android SDK, Phone on Map, Global Positioning System android ap- plication. Application supports mobile phones having version number 4.4 or more and minimum SDK version or API level 19 or more. Android Studio 3.1 is used to imple- ment the application. 5 Result and Discussion When the user opens the application, it shows the activity as shown in figure 9. It first instructs the user to select the language in which they wish to operate. Once the audio instructions get over, it shows the activity as shown in figure 10 that allows users to speak the language or help command. On selection of the language it redirects the user to a particular activity based on the speech input. There are three language options available: English, Kannada, and Hindi. iJIM ‒ Vol. 14, No. 16, 2020 63 Paper—Assistive Mobile Application for Visually Impaired People Fig. 9. Help Module Fig. 10. Main Activity When the user says English it will redirect the users to the English Module as shown in figure 11 and when the user says Hindi it will redirect to the Hindi Module as shown in figure 12. When the user says Kannada it will redirect to the Kannada Module as shown in figure 13. Fig. 11. English Module Fig. 12. Hindi Module Fig. 13. Kannada Module When the user says SMS, it will redirect the user to the SMS module as shown in figure 14. It will extract all transaction-related SMS from the inbox and display it in 64 http://www.i-jim.org Paper—Assistive Mobile Application for Visually Impaired People one activity. TTS will convert these text messages into speech message and visually impaired persons can easily hear the messages. Fig. 14. SMS Module Fig. 15. Document Module When the user says Document, it will redirect the user to the Document Module as shown in figure 15. When the user is in this module, initially it will give instructions and then the user can provide details for creating PDF. When the user completes the information, it will create a pdf of the given information and it will pass the information to Mail Module. The mail Module is as shown in figure 16. The user needs to specify recipient id through the voice command and the mail will be sent to the specified id. Fig. 16. Mail Module Fig. 17. Remainder Module iJIM ‒ Vol. 14, No. 16, 2020 65 Paper—Assistive Mobile Application for Visually Impaired People When the user says Reminder, it will redirect users to the Reminder Module as shown in figure 17. Users need to give dates in the year, month, date format and need to specify event title by voice command. And the user also needs to say an open calen- dar so that the application will open the calendar and mark the event to the specified date. Document creation for Kannada and Hindi is shown in figure 18 and figure 19 respectively. Fig. 18. Kannada document Module Fig. 19. Hindi document Module Email Module for Kannada and Hindi are shown in figure 20 and figure 21 respectively. Fig. 20. Kannada Email Module Fig. 21. Hindi Email Module 66 http://www.i-jim.org Paper—Assistive Mobile Application for Visually Impaired People Fig. 22. Kannada Schedule Manager Module Fig. 23. Hindi Schedule Manager Module When the user says the outdoor region, it will redirect users to the Direction Screen Module as shown in figure 24. Fig. 24. Direction Screen Fig. 25. Direction Screen with marker The user needs to specify the destination address by voice command. Initially, it will show a point at our location and on specifying a destination address it will show the path to reach the destination as shown in figure 25. Along with this, it will give instruc- tions to reach the destination. iJIM ‒ Vol. 14, No. 16, 2020 67 Paper—Assistive Mobile Application for Visually Impaired People 6 Conclusion The proposed and designed application helps the visually challenged people by providing an easy way of accessing everything without anyone's help. By using this application, the visually impaired people can create pdf and send mail to specified email id and also, they can hear transaction related SMS. The visually impaired persons can make use of this application for managing events using calendar API. The visually im- paired can walk in the streets and reach their desired destination with the help of GPS based navigation provided in this application. 7 References [1] P. Kardys, A. Dąbrowski, M. Iwanowski and D. Huderek (2016), A new Android application for blind and visually impaired people, Signal Processing: Algorithms, Architectures, Ar- rangements, and Applications (SPA), Poznan, pp. 152-155.L https://doi.org/10.1109 /SPA.2016.7763604 [2] Ţepelea, I. Gavriluţ, and A. Gacsadi, "Smartphone application to assist visually impaired people," 14th International Conference on Engineering of Modern Electric Systems (EMES), Oradea, 2017, pp. 228-231. https://doi.org/10.1109/EMES.2017.7980421 [3] M. Awad, J. E. Haddad, E. Khneisser, T. Mahmoud, E. Yaacoub and M. Malli, "Intelligent eye: A mobile application for assisting blind people", IEEE Middle East and North Africa Communications Conference (MENACOMM), Jounieh, 2018, pp. 1-6. https://doi.org/10.1109/MENACOMM.2018.8371005 [4] R. Kasthuri, B. Nivetha, S. Shabana, M. Veluchamy and S. Sivakumar, "Smart device for visually impaired people", Third International Conference on Science Technology Engineer- ing & Management (ICONSTEM), Chennai,2017, pp. 54-59. https://doi.org/10.1109 /ICONSTEM.2017.8261257 [5] Hoonlor, S. P. N. Ayudhya, S. Harnmetta, S. Kitpanon, and K. Khlaprasit, "UCap: A crowdsourcing application for the visually impaired and blind persons on Android smartphone", International Computer Science and Engineering Conference (ICSEC), Chiang Mai, 2015, pp. 1-6. https://doi.org/10.1109/ICSEC.2015.7401406 [6] M. N. Kumar, P. C. L. Chandar, A. V. Prasad and K. Sumangali, "Android-based educational Chatbot for visually impaired people", IEEE International Conference on Computational In- telligence and Computing Research (ICCIC), Chennai,2016, pp. 1-4. https://doi.org/10.1109/ICCIC.2016.7919664 [7] S. M. R. Bagwan and L. J. Sankpal, "VisualPal: A mobile app for object recognition for the visually impaired", International Conference on Computer, Communication and Control (IC4), Indore, 2015, pp. 1-6 https://doi.org/10.1109/IC4.2015.7375665 [8] K Shankar, S Dhaarani, S Kaviya shree, "Android Application for Visually Impaired", In- ternational Journal for Research Trends and Innovation, Volume 2, Issue 4, ISSN: 2456- 3315, 2017. [9] A.J.Kadam, Sharvari Awate, Saba Desai, Rajlakshmi Khese and Gauri Patange, "Android Application For Visually Impaired People", Imperial Journal of Interdisciplinary Research, Vol-3, Issue-2, ISSN: 2454-1362, 2017. [10] Gowri G, Harish Kumar R, Gokulakrishnan S, Bovaz Prabhu B, "A Survey on Smart App for Visually Impaired", International Journal for Research in Applied Science & Engineering 68 http://www.i-jim.org https://doi.org/10.1109/SPA.2016.7763604 https://doi.org/10.1109/SPA.2016.7763604 https://doi.org/10.1109/EMES.2017.7980421 https://doi.org/10.1109/MENACOMM.2018.8371005 https://doi.org/10.1109/ICONSTEM.2017.8261257 https://doi.org/10.1109/ICONSTEM.2017.8261257 https://doi.org/10.1109/ICSEC.2015.7401406 https://doi.org/10.1109/ICCIC.2016.7919664 https://doi.org/10.1109/IC4.2015.7375665 Paper—Assistive Mobile Application for Visually Impaired People Technology, Volume 7 Issue 3, ISSN: 2321-9653, 2019. https://doi.org/10.22214/ijra- set.2019.3143 [11] Hussam Elbehiery, M. S. Abdel-Wahab, "Smart Touch Phones Blind Assistant System", American Journal of Systems and Software, Vol. 2, No. 3, 72-80, 2014. [12] Shruti Parkhi, Dr.S.S.Lokhande, and N.D.Thombare , "Vocal Vision Android Application for Visually Impaired Person", International Journal of Science, Engineering and Technol- ogy Research, Volume 5, Issue 6, ISSN: 2278 - 7798, 2016. [13] U.Avanthika, Shyam Sundar, and Silviya Nancy, "An Interactive Mobile Application for the Visually Impaired to Have Access to Listening Audio Books with Handy Books Portal", International Journal of Interactive Mobile Technologies, Volume 9, Issue 1, 2015. https://doi.org/10.3991/ijim.v9i1.4326 [14] Junar Arciete Landicho, "Voisee Communicator: An Android Mobile Application for Hear- ing-impaired and Blind Communications", International Journal of Interactive Mobile Tech- nologies, Volume 10, Issue 4, 2016. https://doi.org/10.3991/ijim.v10i4.5859 8 Authors Sriraksha is a Post Graduate student of Manipal Institute of Technology, (MIT), Manipal Academy of Higher Education, India. Dr. Chandrakala CB is currently working as Associate Professor - Senior Scale in Department of Information & Communication Technology at Manipal Institute of Technology, (MIT), Manipal Academy of Higher Education, India. Her research areas are Distributed Computing, Ad-hoc networks, Software Engineering. She has around 20 years of teaching experience. Article submitted 2020-04-29. Resubmitted 2020-06-03. Final acceptance 2020-07-08. Final version pub- lished as submitted by the authors. iJIM ‒ Vol. 14, No. 16, 2020 69 https://doi.org/10.22214/ijraset.2019.3143 https://doi.org/10.22214/ijraset.2019.3143 https://doi.org/10.3991/ijim.v9i1.4326 https://doi.org/10.3991/ijim.v10i4.5859