Journal of Applied Engineering and Technological Science Vol 4(2) 2023: 992-1011 992 SMART_EYE: A NAVIGATION AND OBSTACLE DETECTION FOR VISUALLY IMPAIRED PEOPLE THROUGH SMART APP Bhasha Pydala1*, T. Pavan Kumar2, K. Khaja Baseer3 Research Scholar, Department of CSE, KLEF (Deemed to be University) Guntur, India-5225021 Professor, Department of CSE, KLEF (Deemed to be University) Guntur, India2 Professor, Department of IT SVEC, Mohan Babu University, Tirupati, India-5171023 Basha.chanti@gmail.com1*, pavankumar_ist@kluniversity.in2, baseer.kk@vidyanikethan.edu3 Received : 26 April 2023, Revised: 31 May 2023, Accepted : 05 June 2023 *Corresponding Author ABSTRACT the purpose of this study was to devise an efficient solution, known as SMART_EYE, aimed at assisting visually impaired individuals in navigating unfamiliar environments and detecting obstacles. The motivation behind this research stemmed from the significant population affected by vision impairment and the limitations of existing navigation alternatives, which are often heavy and expensive and thus infrequently adopted. To address this problem, we employed a method that utilized smart applications with AI and sensor technology. The smart app captured and classified images, while obstacle detection was performed using ultrasonic sensors. Voice commands were used to provide users with real-time information about obstacles in their path. The results of this study demonstrated the effectiveness of the SMART_EYE model in improving both qualitative and quantitative performance measures for visually impaired individuals. The model offered a cost-effective alternative that enabled independent navigation and obstacle detection, thereby enhancing the quality of life for this population. The practical implications of this study are twofold: the SMART_EYE model provides a viable solution for visually impaired individuals to navigate unfamiliar environments, and its cost-effectiveness addresses the limitations of existing assistive devices. From a theoretical perspective, this study contributes to the field of assistive technology by integrating AI and sensor technology in a smart app to aid visually impaired individuals. Furthermore, the evaluation and ranking of different systems based on their impact on the lives of visually impaired people provide a basis for further research and development in this area. In conclusion, this study's value lies in its contribution to both theory and practice, showcasing the potential for future advancements in assistive technology for visually impaired individuals, ultimately improving their quality of life. Keyword: Visually Impaired, Objects, Detection, Recognition, Artificial Intelligence, Sensors 1. Introduction Vision is an essential aspect of human life, providing us with the ability to perceive and interact with the world around us. It enables us to navigate through various environments, recognize objects, and interpret visual information. However, vision impairment is a prevalent global issue, affecting a significant proportion of the population. According to the World Health Organization (WHO), over 283 million people worldwide suffer from sight problems, including 39 million blind individuals and 228 million with low vision. (Andrius Budrionis, 2022) The loss or impairment of vision can have profound consequences, impacting an individual's independence, mobility, and overall quality of life. For individuals with visual disabilities, navigating unfamiliar environments can be particularly challenging (Iskander, 2022). The ability to move around independently and safely is crucial for their well-being and participation in society. While various navigation alternatives have been developed over the years to assist visually impaired individuals, they often face limitations that hinder their widespread adoption and effectiveness. The existing landscape of navigation alternatives for visually impaired individuals is characterized by devices and systems that are either too heavy or too expensive for universal use. These limitations pose significant barriers, preventing individuals with visual disabilities from accessing the necessary assistance and support they require for efficient navigation (Real, 2019). Many available devices are bulky and burdensome to carry, imposing physical and practical challenges for the user. Additionally, the high costs associated with these devices make them inaccessible to a substantial portion of the target population, further exacerbating the problem. Pydala et al … Vol 4(2) 2023 : 992-1011 993 The primary problem addressed by this research is the lack of accessible and affordable navigation alternatives for visually impaired individuals (Lee, 2016). While numerous efforts have been made to develop devices and systems to assist this population, their adoption and implementation remain limited. The current options are often impractical, either due to their weight or cost, preventing visually impaired individuals from fully benefiting from the advancements in assistive technology (R. Gnana Praveen, 2013). Visually impaired individuals face significant challenges when it comes to navigating unfamiliar environments independently. Without reliable and accessible navigation solutions, they are often reliant on the assistance of others or restricted in their mobility (Ilag, 2019). This limitation not only affects their ability to engage in various activities but also hampers their independence and self-confidence. The existing navigation alternatives fail to meet the needs of visually impaired individuals in terms of portability, affordability, and effectiveness (Jafri, 2018). Bulky and heavy devices impose physical and practical constraints, making them inconvenient for daily use. Furthermore, the high costs associated with these devices create a financial barrier, preventing a considerable segment of the population from accessing the necessary assistance. Therefore, there is an urgent need for a lightweight and cost-effective navigation system that can address the challenges faced by visually impaired individuals. Such a system would empower them to navigate unfamiliar environments independently, enhancing their mobility, self- reliance, and overall quality of life. By providing an accessible and affordable solution, visually impaired individuals would have the opportunity to participate more fully in society, engage in various activities, and experience a greater sense of freedom and empowerment. In light of these challenges, this research aims to develop a navigation system, called SMART EYE, that is specifically designed to meet the needs of visually impaired individuals. The SMART_EYE system will be lightweight, portable, and cost-effective, offering a practical and innovative solution for independent navigation and obstacle detection. By addressing the limitations of existing alternatives, this research seeks to bridge the gap and provide visually impaired individuals with an efficient and accessible navigation system that can significantly improve their daily lives. To overcome the limitations of existing navigation alternatives, this research proposes the SMART_EYE system, which aims to provide a lightweight and cost-effective solution for visually impaired individuals to navigate independently and detect obstacles in their path. The SMART_EYE system utilizes a smart application integrated with AI and sensor technology to capture and classify images, while obstacle detection is achieved through the use of ultrasonic sensors. Real-time assistance and obstacle information are provided to the user through voice commands, enabling them to navigate unfamiliar environments more confidently and safely. The precise objective of this research is to develop and evaluate the effectiveness of the SMART_EYE system in addressing the navigation challenges faced by visually impaired individuals. This objective encompasses the following key aspects: 1. Designing a lightweight and portable system application that is accessible to visually impaired individuals, promoting ease of use and mobility. 2. Integrating AI and sensor technology within the smart application to enable real-time obstacle detection and classification, enhancing the user's situational awareness. 3. Evaluating the performance and effectiveness of the SMART_EYE system through rigorous testing and validation to determine its potential impact on the lives of visually impaired individuals. 4. Assessing the cost-effectiveness of the SMART_EYE system to ensure its affordability and widespread accessibility. By achieving these objectives, this research aims to contribute to the field of assistive technology by providing a practical and innovative solution for visually impaired individuals to navigate unfamiliar environments independently. The development of the SMART_EYE system addresses the existing limitations of heavy and expensive navigation alternatives, offering a lightweight, cost-effective, and efficient solution that can significantly improve the lives of visually impaired individuals. Pydala et al … Vol 4(2) 2023 : 992-1011 994 The structure of the paper is as follows: Section 2 provides a Literature Survey, Section 3 presents the methodology of the Proposed Work, Section 4 presents the Results and Discussion, and finally, Section 5 concludes the paper with future scope. 2. Literature Survey The Literature Survey of the research highlights various studies and approaches that have been proposed to aid visually impaired individuals in navigation and obstacle detection. The studies use different technologies such as sensor-based and computer vision, ultrasonic sensors, IR sensors, IoT, and deep learning algorithms to develop assistive devices. These assistive devices aim to improve indoor and outdoor mobility and create a functional system for individuals with visual impairments, including those who are blind. The studies use different feedback mechanisms such as auditory commands, vibrations, and object identification to provide directional information and obstacle detection. Elmannai W. M., (2018) proposes a data fusion framework for guiding visually impaired individuals. The framework combines data from various sensors such as ultrasonic sensors, depth sensors, and cameras to accurately detect obstacles and provide directional information. The proposed framework also employs machine learning algorithms to enhance the accuracy of obstacle detection and to classify the type of obstacle. The authors conducted experiments to validate the accuracy and reliability of the framework, and the results demonstrate a significant improvement in obstacle detection compared to existing systems. The proposed framework has the potential to enhance the mobility of visually impaired individuals and improve their quality of life. Limitations with the above study is: The approach is incapable of showing indoor and outdoor coverage area of obstacles, as well as a directional facility which lacks the ability to recognize longer objects such as doors, walls, and so on. According to Shah (2006) presents a study on a novel sensory direction model for the visually impaired. The system uses ultrasonic sensors to detect obstacles and direction and transmits this information to the user through vibrations of varying intensity and patterns on a handle. The handle is designed to provide feedback to the user based on the vibration's intensity, sensor position, and signal pulse length. The study involved 15 visually impaired individuals of different age groups who were blindfolded and tested in various navigation scenarios. The device was found to be flexible, lightweight, and ergonomically designed to fit different hand sizes. However, one limitation of this technique is that it cannot be connected to a camera for image processing, which restricts its ability to detect information about crosswalks and traffic signals in the outdoor environment. Many electronic devices that aid people with vision loss use information gathered from the environment and provide feedback through touchable or auditory signals. However, the preferred feedback form is still a matter of debate, and opinions vary among individuals. Despite this (Elmannai W. &., 2017), there are certain essential components that any electronic system assisting blind or visually impaired people must have to ensure its reliability and usefulness. These characteristics can be used to evaluate the dependability and effectiveness of the system. However, a drawback of some of these systems is that they may have difficulty detecting objects at certain ranges and are limited to static object detection rather than dynamic object detection. Islam (2019) reviews the development of walking assistants for visually impaired people and discusses recent innovative technologies in this field, along with their merits and demerits. The review aims to draw a schema for upcoming development in the field of sensors, computer vision, and smartphone-based walking assistants. The goal is to provide a basis for different researchers to develop walking assistants that ensure the movability and safety of visually impaired people. Ponnada (2018) presents a prototype of mobility recognition using feature vector identification and sensor computed processor Arduino chips to assist visually challenged people in recognizing staircases and manholes. The prototype provides more independence to the sightless people while walking on the roads and helps them pass through without any assistance. The model is developed using an Arduino kit and a low-weight stick to recognize obstacles, with the chip programmed and embedded in the stick to detect manholes and staircases using a bivariate Gaussian mixture model and speeded up robust features algorithm for feature extraction. Pydala et al … Vol 4(2) 2023 : 992-1011 995 The developed model shows an accuracy of around 90% for manhole detection and 88% for staircase detection. Ahmad (2018) proposes a model-based state-feedback control strategy for a multi-sensor obstacle detection system in a smart cane. The accuracy of the sensors and actuator positions is critical to ensuring correct signals are sent to the user. Low-cost sensors can result in false alerts due to noise and erratic readings. The proposed approach uses a linear quadratic regulator-based controller and dynamic feedback compensators to minimize false alerts and improve accuracy. Real-time experiments showed significant improvements in error reductions compared to conventional methods. According to Bai (2018) presents a novel wearable navigation device to assist visually impaired people in navigating indoor environments safely and efficiently. The proposed device consists of essential components such as locating, way-finding, route following, and obstacle avoiding modules. The authors propose a novel scheme that utilizes a dynamic subgoal selecting strategy to guide users to their destination while avoiding obstacles in a complex, changeable, and possibly dynamic indoor environment. The navigation system is deployed on a pair of wearable optical see-through glasses for ease of use, and it has been tested on a collection of individuals and found effective for indoor navigation tasks. The device's sensors are of low cost, small volume, and easy integration, making it suitable as a wearable consumer device. The study (Tiponut, 2010) focuses on electronic travel aids (ETAs) developed using sensor technology and signal processing to improve the movement of visually impaired people (VIPs) in constantly changing environments. Despite efforts to create effective ETAs, VIPs still rely on traditional aids like white canes and guide dogs. The study proposes an ETA tool with an Obstacles Detection System (ODS) and a Man-Machine Interface, inspired by the visual system of locusts and flies. However, the tool has limitations in identifying obstacle labels and providing auditory navigation for blind users. Chaitali M. Patil (2016) propose a system framework that can help remove communication barriers for people with visual, auditory, and speech disabilities, allowing them to communicate with each other and non-disabled individuals using various modes of communication such as American Sign Language, audio, braille, and regular text. The system aims to improve the individual's capacity and desire to convey and transmit messages. However, the system lacks an effective prototype using the latest technologies. The paper (Bhasha, 2020) proposes a new smart cane for visually impaired people (VIP) which can detect obstacles, water, and light environments in front of, and to the left and right of, the user. The smart cane is constructed using an Arduino Mega 2560 Microcontroller, Ultrasonic Sensors, Light Sensor, and Soil Moisture Sensor. The device generates an audio feedback signal to the user if any obstacle or water or light environment is present in their walking path. The proposed smart cane is more affordable than existing electronic sticks, and the VIPs find it comfortable to use because it is very familiar like a traditional stick. The detection algorithms used in this paper are simple and efficient for detecting the obstacles, water, and light environment of the user’s path, thus helping the user to travel independently from source to destination. Ran (2004) states that few visually impaired assistance aids can provide lively communications and responsiveness to the user and that even if such a system existed, it would likely be sophisticated and not take into account the demands of a blind person, such as simplicity, ease of use, and less complexity. Patil (2018) discusses NavGuide, which employs ultrasonic sensors to classify obstacles and environmental conditions and uses vibration and audio alerts to provide information to the user. However, NavGuide has limitations, including the inability to detect downhill slopes and the detection of damp floors only after the user steps on them. Marzec (2019) describes a navigation system that uses IR sensors to detect walls, buildings, and other objects. The system requires the user to hold the device in their arms, and vibrations are used to convey navigation signals about potential movements and nearby threats. Parimal A. Itankar (2016) aimed to enhance the experience of visually impaired people by using weakly supervised learning to match ambient music selected by a deep neural network. They suggested a multifaceted strategy for measuring ambiguous concepts related to music, including availability, implicit senses, immersion, and subjective fitness. The authors conducted in-depth trials involving 70 individuals and collected feedback on the features of their model. However, the investigation Pydala et al … Vol 4(2) 2023 : 992-1011 996 had three significant flaws: the performance was not cutting-edge, the music database was limited in terms of genre and size, and each experiment involved only 10 people, making it impossible to extrapolate the findings. To generalize the findings, large-scale experiments are necessary, and improved auditory feature representation through devices is needed to enhance accuracy. Sangpal (2019) describes a system that uses Python and AIML to create an intelligent chatbot assistant that mimics the behavior of a human assistant. The system is designed to respond to queries or issues with spoken word remedies. Python programs are used to convert audio commands to text format and for audio reply and voice recognition, similar to Google text-to- speech. AIML is used to match instructions or text to existing dialogues and conversations using predefined audio syntax. The Python interpreter forms the core of the system. This system represents the state-of-the-art in intelligent chatbot assistants. Ashraf *2020) describes an IoT-powered smart stick developed by Ayesha Ashraf et al. to assist people with vision impairments. The stick has an ultrasonic sensor and a buzzer for detecting obstacles and sounding an alarm. An Android app is also developed that can send essential notifications and GPS location to saved phone numbers. The device is lightweight and portable, making it easier for people with disabilities to walk around more easily and comfortably without the risk of injury. The authors also explore how image processing and interaction with the aid can help the user understand the structure of obstacles and objects before providing advice from the aid. Overall, the smart stick is designed to enhance the mobility and safety of visually impaired individuals. Rahman (2021) presents a smart device for visually impaired people (VIPs) that utilizes deep learning and the Internet of Things (IoT)(Kurniawan & Saputra, 2022). The device is divided into three parts: an IoT-based smart stick that monitors the blind person's movement in real-time through the cloud, deep learning algorithms for detecting obstacles, and a virtual assistant to manage the integration. The paper uses the Mask R-CNN model for object detection, which allows for accurate object detection in a short processing time. However, due to the wide range of obstacles in the real world, this model uses a limited number of sensors and devices and a pre- trained model in object recognition with a limited number of real-world images. Bhavani (2021) proposes an approach to provide visually impaired individuals with direction finding, directional help, walking path notification, and an understanding of their surroundings. The proposed approach utilizes highly sensitive sensors and a comfortable and flexible carbon material to construct the stick. The study identifies various disabilities and provides an auditory output that the blind can use to understand the awareness of the buzzer's position. The paper discusses an assistance aid proposed by (Salama, 2019) which uses an ultrasonic sensor to detect obstacles in the path of visually impaired individuals. The sensor measures the height and distance of the obstacle and communicates the information to a microcontroller. The system can sense a distance of up to 12 feet with a resolution of 0.3 cm. However, the paper highlights that such a system may not be suitable for visually impaired individuals as it may be too complex and not meet their requirements for simplicity and ease of use. The article discusses assistive technology, which refers to tools or equipment that help people with impairments to participate fully in society (Foley, 2012) (Mountain, 2004) (Pentland, 1998). Smart aids are a type of assistive technology that can come in mobile computerized forms, such as mobile phones, and are more covert than conventional assistive technologies, reducing social stigma. Navigation aids for the blind are limited in their ability to detect and alert users to the types of obstacles in front of them, and RFID-based systems are expensive and prone to damage. To address these limitations, the article proposes a navigation system that uses deep learning algorithms and a smartphone to identify various obstacles. This system does not require the deployment of RFID chips and is not limited to particular indoor or outdoor settings, thus expanding the locations where it can be used and providing visually impaired people with more information about their surroundings. Table 1 - Comparison of Assistance Aids for Visually Impaired People Reference Metrics Advantages Disadvantages (Elmannai W. M., 2018) Hurdle avoiding approach and data transformation algorithm Significant improvement and qualitative development in conflict warning domain Incapable of showing indoor and outdoor Pydala et al … Vol 4(2) 2023 : 992-1011 997 coverage area of obstacles (Shah, 2006) Biologically inspired obstacle detection system First to be biologically inspired by an insect's visual system Incapable of assessing obstacle label and providing auditory navigation for VIP (Elmannai W. &., 2017) Fundamental components for electronic devices that assist visually impaired people Provides guidance and direction capabilities None mentioned (Islam, 2019) Framework for a communication device for people with vision, hearing, and speech impairments Can transmit message based on user's capacity and desire Lacks an effective prototype with the latest technologies (Ponnada, 2018) Assistance aid for obstacle detection Can identify obstacles on the user's path None mentioned (Ahmad, 2018) Few VIP assistance aids offer lively communications and responsiveness Provides lively communications and responsiveness None mentioned (Tiponut, 2010) NavGuide classifies obstacles and environmental conditions Includes wet floors, obstructions at floor level, and obstacles at knee level Unable to detect a pit or downhill (Chaitali M. Patil, 2016) Infrared sensors for navigation system design Uses low-cost, low-power infrared sensors User must hold the device for sensing vibrations (Sangpal, 2019) Matching ambient music for enhancing VIP experience Uses deep neural network and weakly supervised learning Low accuracy in some models, narrow range of music genres, small sample size (Ashraf, 2020) IoT-powered sensitive smart stick for obstacle detection Highly lightweight and portable None mentioned (Rahman, 2021) Smart device for VIPs utilizing deep learning and IoT Includes real-time monitoring through the cloud Limited number of sensors and devices (Bhavani, 2021) Direction finding, directional help, walking path notification, and comprehension of surroundings for VIPs Uses incredibly sensitive sensors and comfortable, flexible carbon material None mentioned (Salama, 2019) Ultrasonic sensor for obstacle detection Able to sense distance up to 12 feet away with a resolution of 0.3 cm None mentioned Overall, the literature survey includes a variety of proposed solutions for assisting visually impaired individuals, ranging from obstacle detection systems to communication devices. Each reference has its own unique set of advantages and disadvantages, and the metrics used to evaluate them also vary. Some of the common advantages include lightweight and portable design, real- time monitoring, and deep learning algorithms. On the other hand, some of the disadvantages include inability to assess obstacle label and provide auditory navigation, limitations in detecting certain types of obstacles, and lack of an effective prototype with the latest technologies. Research gaps identified in the literature include limitations in identifying obstacle labels, difficulty detecting objects at certain ranges, and limited dynamic object detection. Several studies propose navigation aids that are not connected to a camera for image processing, which restricts their ability to detect information about crosswalks and traffic signals in the outdoor environment. Many studies involve an only small sample size, which makes it difficult to generalize findings, and the performance of some systems is not cutting-edge. Moreover, the lack of simplicity and ease of use of some devices hinders their usefulness and reliability. To overcome these gaps, future research could focus on developing navigation systems that integrate low-cost, small-volume sensors with deep learning algorithms for accurate and reliable obstacle detection, while providing simple and easy-to-use interfaces for visually impaired individuals. Large-scale experiments could be conducted to test the effectiveness of such systems, and more attention could be paid to developing navigation aids that are suitable for both indoor and outdoor use. 3. Methodology Pydala et al … Vol 4(2) 2023 : 992-1011 998 The proposed methodology aims to design a lightweight and portable system application that is accessible to visually impaired individuals, promoting ease of use and mobility. Additionally, it involves integrating AI and sensor technology within the smart application to enable real-time obstacle detection and classification, enhancing the user's situational awareness. Proposed System The system design includes a wearable device and a voice-based navigation system that assists visually impaired individuals in recognizing, detecting, and avoiding obstacles. The system leverages a combination of computer vision and sensor-based technology to achieve its objectives. The methodology employed in this research aims to address the objectives of designing a lightweight and portable system application and integrating AI and sensor technology for real- time obstacle detection and classification. The proposed system utilizes a combination of computer vision and sensor-based technology to assist visually impaired individuals in recognizing, detecting, and avoiding obstacles. The methodology includes the following steps: 1. Designing a Wearable Device and Voice-Based Navigation System: The first step in the methodology involves designing a wearable device and developing a voice-based navigation system specifically tailored for visually impaired individuals. The device should be lightweight and portable, ensuring ease of use and mobility. It should be comfortable for the user to wear and provide convenient access to the navigation features. 2. Implementing Computer Vision and Sensor Technology: To enable real-time obstacle detection and classification, the proposed system integrates computer vision and sensor technology. Computer vision algorithms are used to process images captured by the device's camera and detect objects in the environment. Sensor technology, such as ultrasonic sensors or depth sensors, is utilized to measure the proximity and depth of obstacles. 3. Proximity Measurement Method: The proposed model introduces a novel proximity measurement method for estimating the distance of obstacles based on their depth. This strategy overcomes the limitations of the current system by enabling the detection of multiple objects simultaneously, including longer obstacles like doors and walls. The proximity measurement method enhances the user's situational awareness by providing accurate information about the distance of obstacles in real-time. 4. Implementation Tools: The proposed work has been implemented using various tools and technologies. Raspberry Pi, a small and affordable single-board computer, is utilized as the hardware platform for the wearable device. Android Studio, an integrated development environment, is employed to develop the Android app for the navigation system. Google COLAB Tool, a cloud-based platform for machine learning, is utilized for object detection algorithms. 5. Lightweight, Efficient, and Cost-Effective System: The developed system is designed to be lightweight, efficient, and cost-effective. The hardware components are carefully selected to ensure portability and minimize the overall weight of the device. The software algorithms are optimized for efficient processing and real-time performance. By utilizing affordable and readily available hardware and software resources, the system aims to provide a cost-effective solution for visually impaired individuals in need of navigation assistance. User Testing and Evaluation: Once the system is implemented, user testing and evaluation are conducted to assess its performance and usability. Visually impaired individuals participate in the testing process, providing feedback on the system's effectiveness in assisting with navigation and obstacle detection. User feedback is valuable for refining and improving the system's design and functionality. Pydala et al … Vol 4(2) 2023 : 992-1011 999 Fig. 1. Block diagram of the Proposed Methodology The primary objective of the proposed work is to develop an Intellectual IoT System that can aid in obstacle detection for visually impaired individuals in society. The proposed system is designed to be cost-effective and efficient, making it accessible to a broader range of users. The system is built using the Tensor Flow and Deep Learning frameworks, enabling visually impaired individuals to perform obstacle acknowledgement and autonomous walking in crowds while navigating different modes of transportation. The system consists of a Raspberry Pi 3, a speaker, and a Pi camera, and it can be placed in a pocket with the camera mounted outside. Designing a Wearable Device and Voice-Based Navigation System Fig. 2. Design Process Designing a wearable device and a voice-based navigation system involves creating a hardware and software solution that is specifically tailored for visually impaired individuals. The goal is to develop a device that can be easily worn by the user and provide them with intuitive navigation capabilities through voice commands. Here is an explanation of the key aspects involved in the design process as shown in figure 2: 1. Hardware Design: The hardware design focuses on creating a wearable device that is lightweight, portable, and comfortable for the user to wear. The device may include components such as a camera for capturing images, sensors for obstacle detection, a microphone for voice input, and a speaker for voice output. The size, shape, and placement of these components should be carefully considered to ensure convenience and usability. 2. User Interface Design: The user interface design involves creating an intuitive and accessible interface for visually impaired individuals to interact with the device. Since visual cues may not be applicable, the interface primarily relies on voice-based interactions. This includes designing a system for voice commands, feedback, and prompts. The interface should be Pydala et al … Vol 4(2) 2023 : 992-1011 1000 simple, clear, and easy to navigate, enabling users to control the device and receive information effectively. 3. Voice Recognition and Natural Language Processing: To enable voice-based navigation, the system needs to incorporate voice recognition and natural language processing capabilities. Voice recognition algorithms are employed to accurately interpret and understand the user's voice commands. Natural language processing techniques help in understanding the context and intent of the user's instructions, allowing the system to respond appropriately. 4. Navigation and Routing Algorithms: The design also involves developing navigation and routing algorithms that can guide visually impaired individuals through different environments. These algorithms consider factors such as the user's current location, desired destination, available paths, and obstacle information. By utilizing mapping data and real-time feedback from the obstacle detection system, the device can provide step-by-step directions, alert users about obstacles in their path, and suggest alternative routes when necessary. 5. Integration of Sensor Technology: Sensor technology plays a crucial role in obstacle detection and enhancing situational awareness. Sensors like ultrasonic or depth sensors can be integrated into the wearable device to detect the presence and proximity of obstacles. The data from these sensors is processed and used to provide real-time feedback to the user about the distance and location of obstacles. 6. Accessibility and Ergonomics: Accessibility and ergonomics are critical considerations in the design process. The device should be accessible to individuals with visual impairments, taking into account factors such as tactile feedback, braille labels, and adjustable straps for fitting different body sizes. The device should also be ergonomic, ensuring that it is comfortable and unobtrusive for the user to wear for extended periods. 7. Iterative Design and User Feedback: The design process typically involves multiple iterations and user feedback. Prototype versions of the wearable device and navigation system are tested with visually impaired individuals to gather insights and improve the design. User feedback helps in refining the interface, addressing usability issues, and enhancing the overall user experience. By considering these aspects and leveraging advancements in technology, the design of a wearable device and voice-based navigation system can provide visually impaired individuals with a user-friendly and effective means of navigating their surroundings independently. Fig. 3. The proposed system Architecture of static/dynamic objects detection and navigation process. The proposed work has been implemented using Raspberry Pi, Android Studio, and Google COLAB Tool for object detection. The system is lightweight, efficient, and cost-effective, making it a suitable option for visually impaired individuals who need assistance in navigating their surroundings. The device can also assist regular walkers in detecting obstacles and avoiding accidents. Pydala et al … Vol 4(2) 2023 : 992-1011 1001 Voice based navigation System development: Voice Recognition and Natural Language Processing: Voice recognition algorithms: Let x be the audio input (user's voice command), and let y be the recognized text output. Voice recognition algorithms aim to estimate the most likely text transcription y given the audio input x. This can be represented as: y = VoiceRecognition (x) Natural language processing techniques: Let z be the interpreted intent or context of the user's instructions. Natural language processing techniques involve analyzing the recognized text output y to understand the context and intent of the user's instructions. This can be represented as: z = NaturalLanguageProcessing(y) Fig. 4. Voice recognition flow model Algorithm: Voice Recognition Inputs: • Audio Input: An audio signal represented as a sequence of samples Outputs: • Recognized Text: A sequence of words or transcription of the spoken input 1. Preprocessing: • The audio input signal, 𝑥(𝑡), is typically represented as a discrete-time sequence of samples, where t is the time index. • Various preprocessing techniques can be applied to enhance the quality of the audio signal. Let's denote the preprocessed audio signal as 𝑥𝑝𝑟𝑒(𝑡). 2. Feature Extraction: • The preprocessed audio signal is transformed into a sequence of feature vectors that capture relevant acoustic characteristics. • Let's denote the feature vector sequence as 𝑋 = {𝑥1, 𝑥2, … , 𝑥𝑁 }, where 𝑁 is the total number of feature vectors. • Each feature vector, 𝑥𝑖 , represents a snapshot of the audio characteristics at a specific time frame. 3. Acoustic Modeling: Pydala et al … Vol 4(2) 2023 : 992-1011 1002 • Acoustic modeling aims to estimate the conditional probability of the observed feature vectors given the underlying phonetic units or subword units. • Let 𝐻 represent the set of possible phonetic units or subword units, and 𝑃(𝑋|𝐻)represent the probability of observing feature vectors 𝑋 given a phonetic unit or subword unit sequence H. • Acoustic modeling techniques, such as Hidden Markov Models (HMMs) or Deep Neural Networks (DNNs), are employed to learn and estimate these probabilities. 4. Language Modeling: • Language modeling focuses on estimating the likelihood of word sequences or phrases in a given language. • Let W represent the set of possible word sequences, and P(W) represent the probability distribution over word sequences. • Language models learn the statistical patterns and context of words to estimate the probability of a particular word sequence. 5. Decoding: • Decoding combines the acoustic and language models to find the most probable word sequence given the observed feature vectors. • The decoding process involves finding the word sequence that maximizes the joint probability 𝑃(𝑊, 𝑋|𝐻), where H represents the phonetic or subword unit sequence. • Decoding algorithms, such as Hidden Markov Model Decoding or Beam Search, are employed to find the most probable word sequence. 6. Postprocessing: • Postprocessing techniques are applied to refine the recognized text and improve its accuracy. • These techniques may involve language-specific rules, grammar checks, spell checking, or statistical methods to correct common recognition errors. 7. Output: • The output of the voice recognition algorithm is the recognized text, which represents the transcription of the spoken input. In mathematical terms, the voice recognition algorithm involves estimating conditional probabilities of feature vectors given phonetic or subword units (acoustic modeling), estimating the likelihood of word sequences (language modeling), decoding to find the most probable word sequence, and applying postprocessing techniques. The recognized text is the final output of the algorithm. Proximity Measurement Method Novel proximity measurement method for estimating the distance of obstacles based on their depth Algorithm: Novel Proximity Measurement Method Inputs: • Image: Captured image of the environment • Calibration Parameters: Parameters for calibration and scaling Outputs: • Distance: Estimated distance of the closest obstacle Steps: 1. Initialize the system and calibration parameters. 2. Capture an image of the environment using the device's camera. • Input: Image 3. Apply computer vision algorithms to detect and segment obstacles in the image. • Input: Image • Output: Segmented Obstacles 4. For each segmented obstacle: a. Extract depth information using sensor technology (e.g., ultrasonic or depth sensors). • Input: Segmented Obstacle Pydala et al … Vol 4(2) 2023 : 992-1011 1003 • Output: Depth Information b. Calculate the distance of the obstacle based on the depth information and calibration parameters. • Input: Depth Information, Calibration Parameters • Output: Distance c. Store the distance information associated with the obstacle. 5. Repeat steps 4a to 4c for all detected obstacles in the image. 6. Analyze the stored distance information to identify the closest obstacle. • Input: Stored Distance Information • Output: Closest Obstacle 7. Provide real-time feedback to the user about the distance of the closest obstacle. • Input: Closest Obstacle • Output: Distance 8. Update the system continuously to adapt to changes in the environment and obstacle positions. 9. If there are more images to process, return to step 2; otherwise, proceed to step 10. 10. End the algorithm. Fig. 5. Novel Proximity Measurement Method flow chart Calibration Parameters: Calibration parameters are variables or settings used to adjust and scale the measurements obtained from the sensors and cameras. These parameters help in calibrating the system to ensure accurate distance calculations and obstacle detection. The specific calibration parameters depend on the sensor technology used and the characteristics of the camera or sensors in the system. Examples of calibration parameters may include focal length, lens distortion coefficients, or sensor alignment values. Distance Calculation Formula: The distance calculation formula depends on the specific sensor technology used for depth extraction. For example, if ultrasonic sensors are employed, the distance can be estimated using the time-of- flight principle: Pydala et al … Vol 4(2) 2023 : 992-1011 1004 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 = (𝑆𝑝𝑒𝑒𝑑 𝑜𝑓 𝑆𝑜𝑢𝑛𝑑 × 𝑇𝑖𝑚𝑒) 2 If depth sensors like LiDAR (Light Detection and Ranging) or Time-of-Flight (ToF) cameras are used, the distance can be directly measured by the sensor technology. Analysis of Stored Distance Information: • The stored distance information can be analyzed to identify the closest obstacle. • This analysis typically involves comparing the stored distances associated with each obstacle and selecting the obstacle with the shortest distance as the closest one. • By evaluating the stored distance information, the algorithm can determine which obstacle is in close proximity to the user and provide appropriate feedback or warnings. This algorithm takes an input image of the environment and applies computer vision algorithms to detect and segment obstacles. Depth information is extracted using sensor technology, and the distance of each obstacle is calculated based on the depth information and calibration parameters. The algorithm then identifies the closest obstacle and provides real-time feedback to the user about the estimated distance. The system continuously updates to adapt to changes in the environment, allowing for accurate obstacle detection and distance estimation. The algorithm can be repeated for multiple images if needed. 4. Results And Discussion The hardware prototype of the proposed model is shown below. Fig. 6. System prototype (wearable device) Fig 6 shows a system prototype of the proposed model, which is a wearable device. The wearable device is designed to be compact and lightweight, allowing visually impaired individuals to comfortably wear it during their daily activities. a wearable device for visually impaired individuals includes the following components: 1. Main Unit: The main unit of the wearable device houses the necessary electronics, including the processing unit, memory, and power source. It may resemble a small device that can be attached to the user's clothing or worn on the body. 2. Camera: The wearable device may feature an integrated camera or a camera module that captures images of the user's surroundings. The camera is an essential component for capturing the visual information required for obstacle detection and navigation. 3. Sensor Technology: The wearable device may incorporate various sensor technologies to enhance its functionality. For example, it may include ultrasonic sensors or depth sensors to detect the presence and proximity of obstacles in the user's environment. 4. User Interface: The wearable device may have a user interface, which could include buttons, touch-sensitive areas, or voice command capabilities. The user interface allows visually impaired individuals to interact with the device, provide input, and receive feedback. 5. Connectivity: The prototype may include wireless connectivity options, such as Bluetooth or Wi-Fi, to enable communication with other devices or to connect to a companion mobile application. The smart App Home page: Speak Data is the name of the application. The Home page contains different buttons like connect Bluetooth, view data, speak data, Touch speak, Stop. Pydala et al … Vol 4(2) 2023 : 992-1011 1005 Fig. 7. Speak Data (Smart App Icon) Fig. 8. Home Page of Smart App Bluetooth Pairing: In the proposed system the Bluetooth device used is “project18”. As soon as the Bluetooth is “ON” the mobile app gets a list of wireless devices and the user should connect with the required Bluetooth device by clicking on the “Connect Bluetooth” button on the Home page from the user's mobile app “Speak Data”. Fig. 9. Bluetooth Connection Text to speech direction navigation: When the user connects to the system Bluetooth device, the App automatically gets installed in the user's mobile and when an object is detected through an ultrasonic sensor the app gets the direction data via Bluetooth and it helps to guide the visually impaired people (VIP). If any user wants to check the direct data they can view by clicking the “view data” button in the home page. Pydala et al … Vol 4(2) 2023 : 992-1011 1006 Fig. 10. Speech output Data Dynamic Object Identification voice prompt: When the user clicks on the “Speak dl” button on the home page, the app automatically redirected to the mobile camera and starts identifying an object in the surroundings if any. If any object is identified, it gives voice prompt as an output with a number of objects detected, label (name) of the object. The app can also detect multiple objects in one scan. In Fig. 7. The number of ObjectDetected[0] as per the array index i.e label: ”teddy bear”. Fig 11. Dynamic no. of object identifications is '1' with label(s) 'teddy bear' Moreover the further object(s) identified samples are : Fig. 12. Dynamic Object Detection count is '1' '2' with label(s) 'cell phone'. Fig 13. Dynamic Object Detection count with label(s) 'person & tv'. In Fig. 12. The no. of ObjectDetected[0] as per the array index i.e label: ”cell phone”. In Fig. 13. The no. of ObjectDetected[1] as per the array index i.e label: ”person and tv”. Pydala et al … Vol 4(2) 2023 : 992-1011 1007 Fig 14. Dynamic Object Detection count is 1 with label(s) 'Door'. Fig 15. Dynamic Object Detection count is 1 with label(s) 'chair ' In Fig. 14. The number of ObjectDetected[0] as per the array index i.e label: ”door”. In Fig. 15. The number of ObjectDetected[0] as per the array index i.e label: ”chair”. Static Object Identification: Clicking on “Touch speak” from the Home page it redirects to static page. When users click on each button it opens the camera and identified a particular static object. Fig. 16. Static object detection “Stop” Button to terminate: To exit from App/to stop getting data the user is provided with the button “Stop” which makes the user terminate from App. Fig. 17. Stop button to terminate from App Pydala et al … Vol 4(2) 2023 : 992-1011 1008 Working process of Algorithms with Input and output INPUT OUTPUT Fig. 18. Working process of Algorithms with Input and output In the above working process, the ultrasonic sensor transmits and receives the signals, and the Bluetooth with wireless communication gets connected to the App gives the text the data through which the blind person gets the text to speech as output to guide them. INPUT OUTPUT Fig. 19. Working process of Algorithms with Input and output In the above working process, the trained phone will be input to the App to get data. As soon as the object is detected with the respective trained label of it, the App gets the text as input and it converts it into the speech as output to let blind people interact with the environment. Fig. 20. Detection of Obstacles Accuracy The figure 20 provided represents the accuracy of obstacle detection for various objects under different lighting conditions, specifically during the day and at night. The accuracy values are presented as percentages, ranging from 0% to 100%. • Daytime Detection Accuracy: This column represents the accuracy of detecting obstacles during the daytime. It indicates the likelihood of correctly identifying an object as the specified obstacle during daylight conditions. • Night Detection Accuracy: This column represents the accuracy of detecting obstacles at night. It indicates the likelihood of correctly identifying an object as the specified obstacle in low- light or nighttime conditions. • Object Categories: Each row represents a specific object category, such as "Human," "Two Wheeler," "Couch," "Chair," and so on. 0 0,2 0,4 0,6 0,8 1 1,2 H u m a n T w o W h e e le r C o u ch C h a ir C o w La p to p P la n t B e n ch C a r D o g U m b re lla S u it ca se B o tt le Detection of Obstacles Accuracy Day time Night Pydala et al … Vol 4(2) 2023 : 992-1011 1009 • Accuracy Values: The values in the table indicate the detection accuracy for each object category under the respective lighting conditions. For example, in the "Human" row, the accuracy for daytime detection is given as 0.95 (or 95%), while the accuracy for nighttime detection is 0.76 (or 76%). • Interpretation: Higher accuracy values suggest that the system has a better ability to accurately detect the specified obstacles. For instance, an accuracy value of 0.95 (or 95%) indicates a high likelihood of correctly identifying the obstacle, while a value of 0.76 (or 76%) suggests a relatively lower accuracy. It's important to note that the accuracy values in the table are provided without additional information regarding the specific methodology or dataset used to calculate them. The accuracy of obstacle detection can vary depending on the specific algorithms, training data, and evaluation metrics employed in the system. 4. Conclusion And Future Work This paper presents a navigation and obstacle detection system called SMART_EYE, designed to assist visually impaired individuals through a smart application. The proposed system aims to address the challenges faced by visually impaired people in navigating unfamiliar environments. By incorporating AI and sensor technology, SMART_EYE provides real-time assistance in detecting and classifying obstacles, enabling users to navigate independently. The system captures and analyzes images using computer vision algorithms and detects obstacles using ultrasonic sensors. Users receive feedback through voice commands, enhancing their situational awareness. The proposed model offers a cost-effective and efficient solution, providing qualitative and quantitative performance measures to evaluate its impact on visually impaired individuals' lives. The SMART_EYE system lays the foundation for further advancements and improvements in assisting visually impaired individuals. Future research can focus on the following aspects: Enhanced Object Classification: Improving the accuracy and efficiency of object classification algorithms can enable SMART_EYE to recognize a wider range of objects and provide more detailed information about the environment. 1. Advanced Obstacle Detection: Incorporating advanced sensor technologies, such as LiDAR or depth cameras, can enhance the system's obstacle detection capabilities, providing more precise distance measurements and identifying complex obstacles with greater accuracy. 2. Indoor Navigation: Expanding the system's functionality to include indoor navigation can assist visually impaired individuals in navigating within buildings, such as offices, shopping malls, or public facilities. 3. Integration with Assistive Technologies: Integrating SMART_EYE with other assistive technologies, such as wearable haptic devices or smart canes, can provide a comprehensive solution for visually impaired individuals, offering multiple modalities for obstacle detection and navigation. 4. Accessibility and User Experience: Further research can focus on improving the accessibility and user experience of SMART_EYE by considering factors like tactile feedback, voice- guided instructions, and personalized settings to accommodate individual user preferences and needs. By addressing these areas of future research, the SMART_EYE system can continue to evolve, providing increasingly accurate and efficient navigation and obstacle detection capabilities to improve the quality of life for visually impaired individuals References Ahmad, N. S. (2018). Multi-sensor obstacle detection system via model-based state-feedback control in smart cane design for the visually challenged. IEEE Access, 64182-64192. Andrius Budrionis, D. P. (2022). Smartphone-based computer vision travelling aids for blind and visually impaired individuals: A systematic review. Assistive Technology, 178-194. Ashraf, A. N. (2020). Iot Empowered Smart Stick Assistance For Visually Impaired People. International Journal of Scientific & Technology Research, 356-360. Pydala et al … Vol 4(2) 2023 : 992-1011 1010 Bai, J. L. (2018). Virtual-blind-road following-based wearable navigation device for blind people. IEEE Transactions on Consumer Electronics, 136-143. Bhasha, P. K. (2020). A Simple and Effective Electronic Stick to Detect Obstacles for Visually Impaired People through Sensor Technology. Journal of Advanced Research in Dynamical and Control Systems, 18-25. Bhavani, R. (2021). Development of a smart walking stick for visually impaired people. Turkish Journal of Computer and Mathematics Education, 999-1005. Chaitali M. Patil, A. P. (2016). A Mobile Robostick for Visually Impaired People. International Journal of Computer Engineering In Research Trends, 183-186. Elmannai, W. &. (2017). Sensor-based assistive devices for visually-impaired people: Current status, challenges, and future directions. Sensors, 565-577. Elmannai, W. M. (2018). A highly accurate and reliable data fusion framework for guiding the visually impaired. IEEE Access, 33029-33054. Foley, A. &. (2012). Technology for people, not disabilities: Ensuring access and inclusion. Journal of Research in Special Educational Needs, 192-200. Ilag, B. N. (2019). Design review of Smart Stick for the Blind Equipped with Obstacle Detection and Identification using Artificial Intelligence. International Journal of Computer Applications, 55-60. Iskander, M. H.-A. (2022). Health Literacy and Ophthalmology: A scoping review. Survey of Ophthalmology, 1225-1233. Islam, M. M. (2019). Developing walking assistants for visually impaired people: A review. IEEE Sensors Journal, 2814-2828. Jafri, R. &. (2018). User-centered design of a depth data based obstacle detection and avoidance system for the visually impaired. Human-centric Computing and Information Sciences, 1- 30. Kurniawan, B., & Saputra, H. T. . (2022). Telegram Implementation on Security and Monitoring of Home Door Keys Based on Wemos and Internet of Things . Journal of Applied Engineering and Technological Science (JAETS), 4(1), 244–250. https://doi.org/10.37385/jaets.v4i1.1042 Lee, Y. H. (2016). RGB-D camera based wearable navigation system for the visually impaired. Computer vision and Image understanding, 3-20. Marzec, P. &. (2019). Low energy precise navigation system for the blind with infrared sensors. 2019 MIXDES-26th International Conference "Mixed Design of Integrated Circuits and Systems (pp. 394-397). IEEE. Mountain, G. (2004). Using the evidence to develop quality assistive technology services. Journal of Integrated Care, 19-26. Parimal A. Itankar, H. R. (2016). Indoor Environment Navigation for Blind with Voice Feedback. International Journal of Computer Engineering In Research Trends, 609-612. Patil, K. J. (2018). Design and construction of electronic aid for visually impaired people. IEEE Transactions on Human-Machine Systems, 172-182. Pentland, B. (1998). Living in a State of Stuck: How technology impacts the lives of people with disabilities. Spinal Cord, 210-219. Ponnada, S. Y. (2018). A hybrid approach for identification of manhole and staircase to assist visually challenged. IEEE Access, 41013-41022. R. Gnana Praveen, R. P. (2013). Blind Navigation Assistance for Visually Impaired based on Local Depth Hypothesis from a Single Image. Procedia Engineering, 351-360. Rahman, M. W. (2021). The architectural design of smart blind assistant using IoT with deep learning paradigm. Internet of Things, 1126-1132. Ran, L. H. (2004). Drishti: an integrated indoor/outdoor blind navigation system and service. Second IEEE Annual Conference on Pervasive Computing and Communications (pp. 23- 30). hyderabad: IEEE. Real, S. &. (2019). Navigation systems for the blind and visually impaired: Past work, challenges, and open problems. Sensors, 3404. Salama, R. &. (2019). Design of smart stick for visually impaired people using Arduino. In New Trends and Issues Proceedings on Humanities and Social Sciences (pp. 58-71). IEEE. Pydala et al … Vol 4(2) 2023 : 992-1011 1011 Sangpal, R. G. (2019). JARVIS: An interpretation of AIML with integration of gTTS and Python. 2nd International Conference on Intelligent Computing (pp. 486-489). IEEE. Shah, C. B. (2006). Evaluation of RU-netra-tactile feedback navigation system for the visually impaired. International Workshop on Virtual Rehabilitation (pp. 72-77). kgm: IEEE. Tiponut, V. I. (2010). Work directions and new results in electronic travel aids for blind and visually impaired people. WSEAS Transactions on Systems, 1086-1097.