Wheelchair Control and Home Automation Using Hand Gestures 1 With the advancement of technology in recent times, mobility is one field which is not left behind. The difficulty and dependency for mobility is the major drawback of mechanical wheelchair. But in recent decades, there are many technologies which have been implemented to overcome different disabilities of people. The proposed system uses air gesture to control both home automation and wheelchair movement. Arduino UNO controls the movement of the wheelchair and to control specific electrical appliances at home. The video of hand gesture by user will be processed to obtain the input for Arduino board. The proposed system is aimed at providing ease of use for disabled with minimal hand movement. Wheelchair Control and Home Automation Using Hand Gestures Nithya BG1, Sanjay S1, Thanush R1,*, Thejas BK1, Venugeetha Y1 1Department of Computer Science, Global Academy of Technology, Bengaluru, India Abstract Keywords: Video processing, Home automation, Gesture navigation, Smart wheelchair Received on 06 March 2020, accepted on 15 May 2020, published on 21 May 2020 Copyright © 2020 Nithya BG et al., licensed to EAI. This is an open access article distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/3.0/), which permits unlimited use, distribution and reproduction in any medium so long as the original work is properly cited. doi: 10.4108/eai.13-7-2018.164663 *Corresponding author. Email: thanushr405@gmail.com 1. Introduction Over 65 million people out of the global population suffer from disability in mobility. The invention of wheelchair was a major milestone in this field. A wheelchair can be used for providing movability to physically disabled people. For the disabled, the road to locomotion was majorly built by the invention of a mechanical wheelchair. Previously there was no option other than manual wheelchair which involved much physical effort and the dependency constraints made it not feasible to everyone. With the advancement of technology in recent times, mobility is one field which is not left behind. Human computer interface (HCI), as a great assistive technology, helps people with motor disabilities to ease their day-to-day activities. There are many interfaces that help in the movement of the wheelchair but are not ideal for independent mobility. There is scope for an interface with minimal time delay and user-friendly features for navigation and portability of the wheelchair for physically disabled. Accessing home appliances without dependency is just as necessary as self-reliant mobility for basic independent life- style. Home automation is the process of automating and controlling all the home appliances like fans, lights, TVs among others, using a single interface. Bluetooth and Wi-Fi technology are predominantly used in home automation. An interface that provides easy-to-use features for home automation with mobility using wheelchair is essential for physically disabled people in order to be self-sustained. This paper provides a brief study on various interfaces used for gesture recognition, home automation and wheelchair navigation. Gesture recognition techniques include video processing, image processing and sensors. This study provides an overview of methods that can be used to provide an optimized interface that facilitates physically disabled people. 2. Problem Definition Independent mobility is crucial for development of physical, cognitive, communicative and social skill for physically impaired people. In addition, the high price of the electric wheelchair makes it infeasible for most of the people. The problem of the wheelchair is the type of aid needed by a disabled person to move about is dependent, to a large extent, on the level of his incapacity. People in wheelchairs EAI Endorsed Transactions on Smart Cities Research Article EAI Endorsed Transactions on Smart Cities 07 2020 - 08 2020 | Volume 4 | Issue 12 | e3 http://creativecommons.org/licenses/by/3.0/ mailto:10.4108/thanushr405@gmail.com 2 Nithya BG et al. encounter accessibility issues with doors every day. It is difficult to get a door to open. In addition, the high price of the electric wheelchair makes it infeasible for most of the people. Thus, we chose to investigate whether we could develop a wheelchair that is cost effective and user friendly. 3. Review In Nobuyuki Otsu [9], the paper demonstrates how the image is segmented using automatic thresholding. Here a selection of optimal threshold increases the separability in levels of gray in the obtained result. This method from the discriminant examination, provides practicability for evaluating the edge associated mechanically choosing a best threshold. At the start, only the 0th and 1st order accruing moments of the gray- level histogram are preserved. An ideal threshold is selected automatically, based on the integration and not on differentiation. In Julian Balcerek et. al. [7], a video processing approach is used to help pedestrians in vehicle detection. The system proposed detects the vehicle by a video camera placed on the back of the pedestrian and then notifies them by sending a signal. Video sequences are analyzed using histograms. The efficiency of this system is very high else the delay would not help the pedestrian in moving quickly. The system mainly uses the video processing technique and Machine Learning. In Jochen Triesch et.al.[6], a vision system was developed which is used for recognition of hand posture. The model is based on Elastic Graph Matching (EGM). Computer science has many techniques to recognize patterns. One of which is Elastic matching (EM). EM is also recognized as nonlinear template matching, deformable template, or flexible matching. EM is explained as an idealization problem of 2D distorting identifying respective pixels between targeted images. Main concepts covered are image processing and video processing. In B.G.Lee et. al.[2], a glove kind of device is used to detect and interpret sign language. The device consists of 5 flex- sensors, 2 pressure sensors and a 3-axis inertial motion sensor to differentiate the features in the ASL alphabet. These sign languages are converted into text and sent to the receiver. A mobile application which can run on Android was built with a text-to- speech feature that interprets the received text into voice output. Research outcomes indicate that classification of correct sign language is 65.7% accurate in the 1st version with no pressure sensors. A 2nd version of the approached glove kind of system with the mixture of pressure sensors on the center finger maximizes the classification accuracy rate drastically to 98.2%. The implementation mainly relies on gesture recognition, ML and Android Application. In Francesco Camastra et. al.[5], the system provides a hand identification using Learning Vector Quantization (LVQ). LVQ consists of 2 modules. The 1st module is used for feature extraction from a data glove. The 2nd module is the identifier implemented by LVQ. This test is performed on collection of 3900 hand gestures performed by diverse human beings. Gesture recognition and ML are the domains which are under focus. In Arathi P.N. et. al.[1], most of the home devices are automated and simply handled by gestures. The patterns are captured by the image capturer and are used for processing. Programming part of the implementation is done using algorithms based on MATLAB. The proposed work is said to use an algorithm for object detection. Initially, the picture is captured by the image capturer and MATLAB is used for processing, if the existing pattern is matched with the given gesture the data will be forwarded to the microcontroller, then the home devices are controlled. The project mainly works on image processing, AI and ML. In Rakib Hyder et. al.[11],the paper uses Recursive Circular Hough Transform (RCHT) to demonstrate pupil movement detection which automatically tunes the sensitivity and radius. It uses a low-resolution mobile phone camera to take pictures, from which pupil point is detected. This wheelchair could be controlled by people with working eyes only. Here video is continually captured, and the data is forwarded to the Personal Computer for execution. Then picture frames are drawn from the video frequently. It then converts resized RGB image to Grayscale image. Image Adjustment is done to increase the contrast of the output image. In Celia Shahanaz et. al.[4], the paper demonstrates the necessity for automated electric wheelchairs for disabled people. Here an electric wheelchair is implemented, where the device cost is decreased for a large group of people. There are also different features such as - rough surface detection, torque adjustments, slope and obstacle detection and hence safety is guaranteed. The system receives input from a microphone, joystick, sonar sensors, and rotary encoders and then processes these using a microcontroller and substantially drives the motors with the help of a motor driver. Automatic speed control is implemented which can gradually increase and decrease the speed. There is also a Sonar enabled for obstacle detection. In Keerthi Kumar N et. al.[8], a wheelchair was developed that moves based on the user's brain wave. Systems like Brain Computer Interface (BCI) are used for transactions between the physical devices and the human brain. It uses a concept called Electroencephalography (EEG) to acquire the brain waves and pass it on to the physical device for processing the input and moving the wheelchair accordingly. The data acquisition can be done only when the person is wearing the specified headset. It works on Machine Learning, Brain Computer Interface and Electroencephalography. In Prannah Dey et. al.[10], This project introduces an intelligent wheelchair which will move with respect to various head gestures. Micro-controller is used as the main control part. It also makes use of micro-controller, accelerometer sensor, LDR and ultrasonic sensor. Accelerometer sensor is used to navigate the wheelchair in 5 various directions according to the various head gestures by the user. EAI Endorsed Transactions on Smart Cities 07 2020 - 08 2020 | Volume 4 | Issue 12 | e3 3 The wheels use two DC motors to move in different directions. Relays are used has driver motor in Arduino UNO for wheelchair directions. It makes use of solar panel. A photon detection sensor is used which would facilitate the user to navigate in dim light. Ultrasonic sensor identifies the hurdles, LDR releases the light and sends the indicator to the microcontroller. Sensor for seat belt is used as a provision to provide secure movement. In Biswajeet Champaty et. al.[3], a wheelchair was developed using eye signals. This machine was developed for people with very minimal movements. An EOG signal detection system was first built and then signals received were processed to give the control signals to the wheelchair based on the amplitude and timing of the signals. Electrooculogram signals are preferred in this system as it has a unique pattern for each eye movement. It mainly concentrates on the field Electro Oculography (EOG). In Abhijit M et. al.,[13] the home automation is done using a hand gesture. The gestures are recognized using the gyroscope, magnetometer and accelerometer on the 3- Dimensional axis. The user needs to wear a removable glove for the gesture recognition. A hub is used to control all the home appliances. The input is sent to the hub through a wireless channel like the Bluetooth. In Siri. T. Bhat et. al.,[14] the wheelchair is controlled by the use of hand gestures. The user wears a glove called as sparsh gloves which consists of sensors to analyze the user input. The wheelchair seat height is also made adjustable. This wheelchair is also used to warn the users when there is an obstacle in front of the tire and also behind the wheelchair. In Ms. S. D. Suryawanshi et. al.,[15] voice commands are for the movement of the wheelchair. These voice commands are compared with the prestored voice commands and respective actions are performed on the wheelchair. An ARM processor is used for the control of the wheelchair. The main domain on which this wheelchair works on is Voice recognition. In Sayeed Shafayet Chowdhury et. al.[12],the proposed wheelchair can be useful for the people with disabilities making use of easy finger actions on a small white background paper. A sturdy scheme of finger motion detection is proposed for the direction control of a wheelchair. Concept of background subtraction and morphological operations are used to acquire the finger-tip area using a finger-motion detection scheme. The method used in this paper is able to figure out finger motion in real-time from photos taken. Images are taken using simple cellular phone cameras and fingertip is detected using image processing techniques. The control operation among the cellular-phone and wheelchair is completed wirelessly with the help of Bluetooth era. Appropriate control indicators are dispatched via Bluetooth to the wheelchair control board which helps the users to control the wheelchair in different directions with the help of voice commands. This paper specifies humanitarian applications such as control of flexible bed for patients and even helps in home automation for ease of life. 4. Comparative Analysis In Prannah Dey et. al.[10], This project introduces an intelligent wheelchair which will move with respect to various head gestures. Head gesture is impractical because many people might feel nauseous while moving their heads around. When ever the person wants to travel for a long distance, he/she needs to either keep moving their head or keep their head in still position which might be difficult. In Rakib Hyder et. al.[11], The paper uses Recursive Circular Hough Transform (RCHT) to demonstrate pupil movement detection. Since this approach uses a low resolution camera to obtain the pupil movements, the person should always have an eye contact to the camera. If there is no direct eye contact, then the wheelchair could be uncontrollable. This wheelchair could be controlled by people with working eyes only. 100% efficiency cannot be provided in dim environment. If any other person has a direct eye contact with the camera, then the wheelchair might be in a dilemma. In Keerthi Kumar N et. al.[8], A wheelchair was developed that moves based on the user's brain wave. This interface is that it is very costly and a slight change in concentration also makes the wheelchair to move haphazardly. The person can be able to control the wheelchair only if the person is wearing the specified headset. This is a very sensitive approach and hence if there are any other thoughts occurring in the brain, it may cause problems in movement of the wheelchair. Table 1 gives a brief description of how our proposed system overcomes the challenges faced by the existing systems. Sl. No Existing System Disadvantages Proposed System 1 Head gesture movement Nauseous / sickness feeling after a period of time. Hand gesture movement 2 Pupil movement detection Blocks the actual view of the person. Hand gesture movement 3 Home Automation using sensored gloves Difficulty in remembering all symbols. User defined patters for home automation. Table 1. Comparative analysis 5. Proposed System The air gestures made by the disabled person are recorded by the camera. The air gestures can be made either using only fingers or complete hand where in the arm of the person will rest on the wheelchair. The person sitting on the wheelchair should just wave his/her finger or hand and hence no additional efforts are necessary. The person can draw the Wheelchair Control and Home Automation Using Hand Gestures EAI Endorsed Transactions on Smart Cities 07 2020 - 08 2020 | Volume 4 | Issue 12 | e3 4 gestures without using any gloves. The web camera sends the live video frames to the video processor that houses an algorithm that can understand the air gestures. If the gesture corresponds to wheelchair motion, this information is sent to motor control board and it controls the wheel, while, if the gesture corresponds to controlling home appliances, then the home appliances are controlled through a Bluetooth interface. Specific home appliances are controlled with the help of a relay. Inside a relay there will be a coil, when the voltage is applied through the coil, electromagnetic field is generated and the coil starts to act as a magnet, pulling the armature towards itself which drives the home application. Here, designing a video processing algorithm using software as specified below which will identify the air gestures. Finally, patterns recognizable by the camera help in the moving the wheelchair, controlling the home appliances and opening/closing of doors. 6. Methodology 6.1 Existing System Figure 1. 3D printed wearable device that holds the hardware components, which include an Android Pro Mini microcontroller, a flex sensor, a motion sensor, and a Bluetooth low energy (BLE) module.[2] Here a wearable hand device [Figure 1] is used for sign language detection and interpretation. The device consists of five flex-sensors containing two pressure sensors and a three-axis inertial motion sensor to distinguish the characters in the American Sign Language alphabet [Figure 2]. Figure 2. 3D printed finger holder using a flexible filament that can accommodate different finger sizes, providing flexibility: (a) front view, (b) back view, and (c) holder with a flex sensor.[2] Figure 3. Overview of sign interpretation system that consists of three modules, namely, sensors module, processing module, and application module.[2] The entire system [Figure 3] mainly consists of three modules: a wearable device with a sensor module and a processing module, and a display unit mobile application module. Sensor data are collected and analyzed using a built-in embedded support vector machine classifier. The recognized alphabet is further transmitted to a mobile device through Bluetooth. An Android-based mobile application was developed with a text-to-speech function that converts the received text into audible voice output. (b) (a) Nithya BG et al. EAI Endorsed Transactions on Smart Cities 07 2020 - 08 2020 | Volume 4 | Issue 12 | e3 5 Wheelchair Control and Home Automation Using Hand Gestures Figure 4. Improved version (second prototype) of 3D printed wearable device (see Fig. 1) with fusion of pressure sensor added to the middle finger.[2] The results [Figure 4] indicate that a true sign language recognition accuracy rate of 65.7% can be achieved on average without pressure sensors. A second version of the proposed wearable system with the fusion of pressure sensors on the middle finger increased the recognition accuracy rate dramatically to 98.2%. 6.2 Proposed System Figure 5. Block diagram of Air Gesture Interface for Wheelchair motion Fig 5 gives the outline of our proposed system. Once the video is captured, it is processed and sent to the Arduino board. Based on the type of input, it is sent to the motor control board for wheelchair movement or sent to another Arduino board which are connected through Bluetooth. These signals are then sent either to motor control board for door control or to relays for accessing home applications. 7. Conclusion This paper exhibits different interfaces developed for the movement of wheelchair, gesture detection and home automation. As the researchers are working and making systems better compared to the previous versions, yet there is scope for improvement. The independent mobility is not sufficient for independent lifestyle. Accessing home appliances without dependency is just as necessary as independent mobility. This paper has the proposed idea of an interface for the wheelchair that can support wheelchair motion and home automation with door controls. The proposed gesture interface would ease the lives of disabled. References [1] ArathiP.N, S.Arthika ,S.Ponmithra ,K.Srinivasan, V.Rukkumani “Gesture Based Home Automation System”, IEEE International Conference on Nextgen Electronic Technologies – 2017 [2] B. G. Lee, Member, IEEE, and S. M. Lee, “Smart Wearable Hand Device for SignLanguage Interpretation System with Sensors Fusion”, DOI 10.1109/JSEN.2017.2779466, IEEE Sensors Journal, December 2017 [3] Biswajeet Champaty, Jobin Jose, Kunal Pal, Thirugnanam A, “Development of EOG Based Human Machine Interface control System for Motorized Wheelchair”, International Conference on Magnetics, Machines & Drives (AICERA) – 2014 [4] Celia Shahanaz, Ahmed Maksud, Shaikh Anowarul Fattah and Sayeed ShafayetChowdhury,” Low-cost Smart Electric Wheelchair with Destination Mapping and Intelligent Control Features”, IEEE International Symposium on Technology in Society (ISTAS) – 2017 Conference Proceedings [5] Francesco Camastra and Domenico De Felice, “LVQ- based Hand Gesture Recognition Using a Data Glove”, DOI: 10.1007/978-3-642-35467-0_17 – May2012 [6] Jochen Triesch and Christoph von der Malsburg, “A System for Person-Independent Hand Posture Recognition against Complex Backgrounds”, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 23, NO. 12, DECEMBER 2001 [7] Julian Balcerek, Adam Konieczka, Tomasz Marciniak, Adam Dbrowski, Krzysztof Mackowiak, Karol Piniarski, “Video processing approach for supporting pedestrians in vehicle detection”, IEEE - September 2014 [8] Keerthi Kumar M, Chaitra Rai, Manisha R, Priyanka C B, Syeda Saniya Anis, “EEG Controlled Smart Wheelchair for Disabled People”, International Journal of Engineering Research & Technology (IJERT) NCRACES - 2019 Conference Proceedings. [9] Noboyuki Otsu, “A Threshold Selection Method from Gray Level Histograms”, IEEE Transactions on systems, man, and cybernetics, val.SMC-9, no.1, January 1979 [10] Prannah Dey, Md.Mehedi Hasan, SrijonMostofa, Ariful Islam Rana, “Smart wheelchair integrating head gesture navigation”, 2019 International Conference on Robotics,Electrical and Signal Processing Techniques (ICREST). [11] Rakib Hyder, Sayeed Shafayet Chowdhury and Shaikh Anowarul Fattah, “Real-Time Non-intrusive Eye-gaze Tracking Based Wheelchair Control For The Physically Challenged”, IEEE EMBS Conference on Biomedical Engineering and Sciences (IECBES) - 2016 [12] Sayeed Shafayet Chowdhury, Rakib Hyder, Celia Shahanaz and Shaikh Anowarul Fattah, “Robust Single Finger Movement Detection Scheme for Real Time Wheelchair Control by Physically EAI Endorsed Transactions on Smart Cities 07 2020 - 08 2020 | Volume 4 | Issue 12 | e3 6 Challenged People”, IEEE Region 10 Humanitarian Technology Conference (R10-HTC).,2017 [13] Abhijit M. ,Anjana Nair, Jikhil John, ShabasBasheer, Munna Basil Mathai, "Hand Gesture Based Home Automation", International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering, DOI:10.15662/IJAREEIE.2017.0603127 [14] Siri.T.Bhat, B. Surekha, Shreesha Raghavan, "Sparsh Glove: A Gesture-Based Hardware Control for a Multipurpose Wheelchair", International Conference on Computer Science and Technology Allies in Research-March 2016 by IJCSE, E-ISSN: 2347-2693 [15] Ms. S. D. Suryawanshi, Mr. J. S. Chitode, Ms. S. S. Pethakar, "Voice Operated Intelligent Wheelchair", International Journal of Advanced Research in Computer Science and Software Engineering, ISSN:2277 128X Nithya BG et al. EAI Endorsed Transactions on Smart Cities 07 2020 - 08 2020 | Volume 4 | Issue 12 | e3 Abstract 1. Introduction 2. Problem Definition 3. Review 4. Comparative Analysis 5. Proposed System 6. Methodology 6.1 Existing System 6.2 Proposed System 7. Conclusion References