1 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) ISSN (Print) 2313-4410, ISSN (Online) 2313-4402 © Global Society of Scientific Research and Researchers http://asrjetsjournal.org/ ASL Recognition Quality Analysis Based on Sensory Gloves and MLP Neural Network Assistant Prof. Dr. Firas A. Raheema*, Hadeer A.Raheem b aAssistant Prof. Dr. ,University of Technology , Baghdad , Iraq bEngineer ,University of Technology , Baghdad , Iraq aEmail: 60124@uotechnology.edu.iq bEmail: Hadeerasaad@gmail.com Abstract A simulated human hand model has been built using a virtual reality program which converts printed letters into a human hand figure that represents American Sign Language (ASL), this program was built using forward and inverse kinematics equations of a human hand. The inputs to the simulation program are normal language letters and the outputs are the human hand figures that represent ASL letters. In this research, a hardware system was designed to recognize the human hand manual alphabet of the ASL utilizing a hardware glove sensor design and using artificial neural network for enhancing the recognition process of ASL and for converting the ASL manual alphabet into printed letters. The hardware system uses flex sensors which are positioned on gloves to obtain the finger joint angle data when shown each letter of ASL. In addition, the system uses DAQ 6212 to interface the sensors and the PC. We trained and tested our hardware system for (ASL) manual alphabet words and names recognition and the recognition results have the accuracy of 90.19% and the software system for converting printed English names and words into (ASL) have 100% accuracy. Keywords: ASL; Artificial neural network; forward kinematics; inverse kinematics; deaf; DOF. 1. Introduction Deaf people have the same needs as the normal people of communicate with other people but they have the problem of describing that because they cannot speak. ------------------------------------------------------------------------ * Corresponding author. http://asrjetsjournal.org/ American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 2 There are diverse sign languages throughout the world, similarly as there are diverse talked languages American Sign Language and British Sign Language are distinctive, commonly ambiguous Since the American and British Deaf people group were not in contact with each other, the two languages grown autonomously. French Sign Language, Danish Sign Language, Taiwan Sign Language, Australian Sign Language, Thai Sign Language, Finnish Sign Language, Brazilian Sign Language, what's more, numerous others have developed in communities of Deaf people, similarly as talked languages have developed in groups of hearing people. Each displays the sorts of basic contrasts from the nation’s talked languages that show it to be a language in its own particular right [1]. This languages help deaf people to communicate with each other and normal people who knew that languages. This work will depend on ASL and focus on recognition of one constrained but important part of the Language: ASL finger-spelling alphabet as shown in (Figure 1). In which signers spell out a word as a sequence of hand shapes or hand trajectories corresponding to individual letters. Unfortunately the majority of normal people do not understand that language this causes the isolation of deaf people from general community. This language is expressed by using hand gesture .The human hand is a great complex system because its extensive number of degrees of freedom (DOF) inside an essentially small space it’s composed of 19 links corresponding to the human bones and 24 DOF’s [2]. In the last decade numerous researchers have studied communication through signing some of them take the way of converting normal English language into sign language [3-5], and the others take the reverse way by converting sign language into normal language, sign language recognition and Gesture recognition have been studied by large number of researchers, in any case, there are significant challenges because of complexity of hand and body movement in sign language expressions. Gesture and Sign language recognition researches can be ordered into two classes: • Computer vision based [6-10]. • Data glove and movement sensor based [11-14]. In this paper we will using data glove to perform all gestures of (ASL) 28 letters both static and dynamic by processing the data using artificial neural network algorithms to recognize ASL letters, using new and economy hardware design by decreasing the sensors needed, and we will build a software program to covert normal English language into ASL using kinematics equations and MATLAB programming .so that we will gathering the two approaches in this paper. 2. American Sign Language American Sign Language (ASL) developed by Thomas Hopkins Gallaudet who brought the sign language from Spain to America [15]. It is a complex language that utilizes signs made by moving the hand. It’s the essential language of Americans who are deaf or hard of hearing and is one of several communication options used by deaf people. There are about two million deaf people in the USA. ASL is the second most widely used non- English language in the United States after Spanish.ASL consists of 36 hand shapes, 6000 words, and 26 letters [16]. These can be performed by using hand and body gestures. American Sign Language alphabet is shown in (figure 1).it used in performing names and spelling words. We will depend on ASL alphabet as a reference in American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 3 our project. Figure 1: American Sign Language alphabet 3. Kinematic Modeling of human hand The Human hand is one of the most complicated systems because of its ability to perform multiple tasks and its wide range of flexibility so that the human hand has 24 DOF and 19 links .in this project we will simplify this system to 20 DOF and 15 link because some links and joints does not effect on performing the letters of (ASL), we will derive one model for the thumb and for the (index, middle, ring, little) fingers. Where the thumb has 3 links (metacarpal, proximal, and distal) links and three joints (metacarpophalangeal (MCP), interphalangeal (IP) and trapeziometacarpal (TMC)).as shown in (figure 2) The MCP and IP joints has one DOF for each one but TMC joint is universal joint and has two DOF one for adduction/abduction and one for flexion /extension (figure 3), (figure 4) shows the difference between adduction/abduction and flexion/extension. The other fingers (proximal, middle, and distal) also have 3 links. And three joints (proximal interphalangeal PIP Distal interphalangeal DIP and metacarpophalangeal MCP, The DIP and PIP joints has one DOF for each one but MCP joint is universal joint and has two DOF one for adduction /abduction and one for flexion/extension [2]. Figure 2: human hand skeleton [17]. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 4 Figure 3: MCP abduction (A) and adduction (B) [18]. Figure 4: MCP flexion (A), PIP flexion (B), DIP flexion(C), and MCP, PIP, and DIP extension (D) [18]. 3.1 forward kinematics The forward kinematic was used to find the finger tip orientation and position depending on the finger joint angels. Model equations are calculated by using the Denavit-Hartenberg (D-H) parameters [19-20]. the translation and rotation of each joint is found by the transformation matrix 𝑇𝑇𝑖𝑖𝑖𝑖−1 (𝜃𝜃𝑖𝑖) 𝑇𝑇𝑖𝑖𝑖𝑖−1 (𝜃𝜃𝑖𝑖) =� 𝐶𝐶(𝜃𝜃𝑖𝑖) 𝑆𝑆(𝜃𝜃𝑖𝑖) 0 0 −𝐶𝐶(𝛼𝛼𝑖𝑖)𝑆𝑆(𝜃𝜃𝑖𝑖) 𝐶𝐶(𝛼𝛼𝑖𝑖)𝐶𝐶(𝜃𝜃𝑖𝑖) 𝑆𝑆(𝛼𝛼𝑖𝑖) 0 𝑆𝑆(𝛼𝛼𝑖𝑖)𝑆𝑆(𝜃𝜃𝑖𝑖) −𝑆𝑆(𝛼𝛼𝑖𝑖)𝐶𝐶(𝜃𝜃𝑖𝑖) 𝐶𝐶(𝛼𝛼𝑖𝑖) 0 𝑎𝑎𝑖𝑖𝐶𝐶(𝜃𝜃𝑖𝑖) 𝑎𝑎𝑖𝑖𝑆𝑆(𝜃𝜃𝑖𝑖) 0 1 � Where C=cosine (𝜃𝜃𝑖𝑖) , S=sin (𝜃𝜃𝑖𝑖) The D-H parameter for a single finger is shown in table(1) where joints defined by the variable θ. links are represent by the parameter a which is the length of the bone .d parameter is always zero since links(bones) are aligned, and α is the twist angel. θ1, θ2, θ3, θ4 are the angels of rotation for adduction\abduction of TMC joint ,flexion\extension of TMC joint ,MCP joint, and IP joint for the thumb finger and the angels of rotation for adduction \abduction of MCP joint , flexion\extension of MCP joint, The DIP, and PIP (for the rest of fingers)respectively. And L1, L2, L3 are the length of bones. Frame -1 is the wrist frame and represents the base American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 5 frame for all fingers. Table 1: D-H parameter for a single joint Ө d a α 1 θ1 0 0 pi/2 2 θ2 0 L1 0 3 θ3 0 L2 0 4 θ4 0 L3 0 Equation 1 represents the direct kinematics equation for finger p = T(u) T(𝜃𝜃1)1 0 0 −1 T(𝜃𝜃2)2 1 T(𝜃𝜃3)3 2 T(𝜃𝜃4)4 3 (1) The direct kinematics equation can be solve by finding Homogeneous matrixes for the finger which are 𝑇𝑇10 (𝜃𝜃1) =� 𝐶𝐶(𝜃𝜃1) 𝑆𝑆(𝜃𝜃1) 0 0 0 0 1 0 𝑆𝑆(𝜃𝜃𝑖𝑖) −𝐶𝐶(𝜃𝜃𝑖𝑖) 0 0 0 0 0 1 � (2) 𝑇𝑇21 (𝜃𝜃2) =� 𝐶𝐶(𝜃𝜃2) 𝑆𝑆(𝜃𝜃2) 0 0 −𝑆𝑆(𝜃𝜃2) 𝐶𝐶(𝜃𝜃2) 0 0 0 0 1 0 𝐿𝐿1𝐶𝐶(𝜃𝜃2) 𝐿𝐿2𝑆𝑆(𝜃𝜃2) 0 1 � (3) 𝑇𝑇32 (𝜃𝜃3) = � 𝐶𝐶(𝜃𝜃3) 𝑆𝑆(𝜃𝜃3) 0 0 −𝑆𝑆(𝜃𝜃3) 𝐶𝐶(𝜃𝜃3) 0 0 0 0 1 0 𝐿𝐿2𝐶𝐶(𝜃𝜃3) 𝐿𝐿2𝑆𝑆(𝜃𝜃3) 0 1 � (4) 𝑇𝑇43 (𝜃𝜃4) = � 𝐶𝐶(𝜃𝜃4) 𝑆𝑆(𝜃𝜃4) 0 0 −𝑆𝑆(𝜃𝜃4) 𝐶𝐶(𝜃𝜃4) 0 0 0 0 1 0 𝐿𝐿3𝐶𝐶(𝜃𝜃4) 𝐿𝐿3𝑆𝑆(𝜃𝜃4) 0 1 � (5) To find the position and orientation of the thumb or other fingers with respect to the base (wrist) frame we use the transformation identity matrix for the rotational part and the position of frame zero with respect to frame -1 so that the transformation matrix become 𝑇𝑇0−1 (𝑈𝑈𝑡𝑡ℎ𝑢𝑢𝑢𝑢𝑢𝑢) = � 1 0 0 0 0 1 0 0 0 0 1 0 𝑢𝑢𝑡𝑡,𝑥𝑥 𝑢𝑢𝑡𝑡,𝑦𝑦 𝑢𝑢𝑡𝑡,𝑧𝑧 1 � (6) 𝑇𝑇40 = � 𝑛𝑛𝑡𝑡,𝑥𝑥 𝑛𝑛𝑡𝑡,𝑦𝑦 𝑛𝑛𝑡𝑡,𝑧𝑧 0 𝑠𝑠𝑡𝑡,𝑥𝑥 𝑠𝑠𝑡𝑡,𝑦𝑦 𝑠𝑠𝑡𝑡,𝑧𝑧 0 𝑎𝑎𝑡𝑡,𝑥𝑥 𝑎𝑎𝑡𝑡,𝑦𝑦 𝑎𝑎𝑡𝑡,𝑧𝑧 0 𝑝𝑝𝑡𝑡,𝑥𝑥 𝑝𝑝𝑡𝑡,𝑦𝑦 𝑝𝑝𝑡𝑡,𝑧𝑧 1 � (7) American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 6 Where: 𝑛𝑛𝑡𝑡,𝑥𝑥=(𝐶𝐶1𝐶𝐶2𝐶𝐶3 − 𝐶𝐶1𝑆𝑆2𝑆𝑆3)𝐶𝐶4 +(-𝐶𝐶1𝐶𝐶2𝑆𝑆3 − 𝐶𝐶1𝑆𝑆2𝐶𝐶3)𝑆𝑆4 (8) 𝑛𝑛𝑡𝑡,𝑦𝑦 =(𝑆𝑆1𝐶𝐶2𝐶𝐶3 − 𝑆𝑆1𝑆𝑆2𝑆𝑆3)𝐶𝐶4 + (−𝑆𝑆1𝐶𝐶2𝑆𝑆3 − 𝑆𝑆1𝑆𝑆2𝐶𝐶3)𝑆𝑆4 (9) 𝑛𝑛𝑡𝑡,𝑧𝑧 =(𝑆𝑆2𝐶𝐶3 − 𝐶𝐶2𝑆𝑆3)𝐶𝐶4 + (−𝑆𝑆2𝑆𝑆3 + 𝐶𝐶2𝐶𝐶3)𝑆𝑆4 (10) 𝑠𝑠𝑡𝑡,𝑥𝑥 =-(𝐶𝐶1𝐶𝐶2𝐶𝐶3 − 𝐶𝐶1𝑆𝑆2𝑆𝑆3)𝑆𝑆4 +(-𝐶𝐶1𝐶𝐶2𝑆𝑆3 − 𝐶𝐶1𝑆𝑆2𝐶𝐶3)𝐶𝐶4 (11) 𝑠𝑠𝑡𝑡,𝑦𝑦= -(𝑆𝑆1𝐶𝐶2𝐶𝐶3 − 𝑆𝑆1𝑆𝑆2𝑆𝑆3)𝑆𝑆4 + (−𝑆𝑆1𝐶𝐶2𝑆𝑆3 − 𝑆𝑆1𝑆𝑆2𝐶𝐶3)𝐶𝐶4 (12) 𝑠𝑠𝑡𝑡,𝑧𝑧 =-(𝑆𝑆2𝐶𝐶3 − 𝐶𝐶2𝑆𝑆3)𝑆𝑆4 + (−𝑆𝑆2𝑆𝑆3 + 𝐶𝐶2𝐶𝐶3)𝐶𝐶4 (13) 𝑎𝑎𝑡𝑡,𝑥𝑥 = 𝑆𝑆1 (14) 𝑎𝑎𝑡𝑡,𝑦𝑦 = −𝐶𝐶1 (15) 𝑎𝑎𝑡𝑡,𝑧𝑧 = 0 (16) 𝑝𝑝𝑡𝑡,𝑥𝑥=(𝐶𝐶1𝐶𝐶2𝐶𝐶3 − 𝐶𝐶1𝑆𝑆2𝑆𝑆3)𝐶𝐶4𝐿𝐿3 +(-𝐶𝐶1𝐶𝐶2𝑆𝑆3 − 𝐶𝐶1𝑆𝑆2𝐶𝐶3)𝑆𝑆4𝐿𝐿3 +(𝐶𝐶1𝐶𝐶2𝐶𝐶3 − 𝐶𝐶1𝑆𝑆2𝑆𝑆3)𝐿𝐿2 + 𝐶𝐶1𝐶𝐶2𝐿𝐿1 (17) 𝑝𝑝𝑡𝑡,𝑦𝑦= (𝑆𝑆1𝐶𝐶2𝐶𝐶3 − 𝑆𝑆1𝑆𝑆2𝑆𝑆3)𝐶𝐶4𝐿𝐿3 + (−𝑆𝑆1𝐶𝐶2𝑆𝑆3 − 𝑆𝑆1𝑆𝑆2𝐶𝐶3)𝑆𝑆4𝐿𝐿3 +(𝑆𝑆1𝐶𝐶2𝐶𝐶3 − 𝑆𝑆1𝑆𝑆2𝑆𝑆3)𝐿𝐿2 + 𝑆𝑆1𝐶𝐶2𝐿𝐿1 (18) 𝑝𝑝𝑡𝑡,𝑧𝑧 =(𝑆𝑆2𝐶𝐶3 − 𝐶𝐶2𝑆𝑆3)𝐶𝐶4𝐿𝐿3 + (−𝑆𝑆2𝑆𝑆3 + 𝐶𝐶2𝐶𝐶3)𝑆𝑆4𝐿𝐿3 +(𝑆𝑆2𝐶𝐶3 + 𝐶𝐶2𝑆𝑆3)𝐿𝐿2 + 𝑆𝑆2𝐿𝐿1 (19) Figure 5: D-H coordinate assignment for one finger of human hand 3.2 Inverse kinematics The solution of the inverse kinematics can be derived from geometric methods [21], such as the relation of American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 7 triangles. The hand can reproduce positive or negative movements with regard to a reference line for some joints. Or algebraic methods [19] by finding relations between the elements of the final transformation matrix that derived in the forward kinematics. In the solution of the inverse kinematics first we find Xc, Yc, Zc which denoted the component of the base frame (frame 1) by using transformation matrix (Equ.7).and we can find φ from it. 𝜑𝜑 = 𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛2 𝑛𝑛1𝑧𝑧 𝑠𝑠1𝑧𝑧 (20) Where 𝜑𝜑 = 𝜃𝜃2 + 𝜃𝜃3 + 𝜃𝜃4 (21) By using geometrical method and as shown in (figure 6): 𝜃𝜃1 = 𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛2 𝑌𝑌𝑌𝑌 𝑋𝑋𝑌𝑌 (22) Figure 6: projection of the finger onto x0 –z0 plane. Figure 7: projection of the finger onto x0 –y0 Plane. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 8 From basic trigonometry and (figure 7), the position and orientation of the finger tip can be written in terms of the joint coordinates in the following way: 𝑋𝑋 = 𝐿𝐿1 cos 𝜃𝜃2 + 𝐿𝐿2 cos(𝜃𝜃2+𝜃𝜃3) + 𝐿𝐿3 cos(𝜃𝜃2 + 𝜃𝜃3 + 𝜃𝜃4) (23) 𝑌𝑌 = 𝐿𝐿1 sin 𝜃𝜃2 + 𝐿𝐿2 sin( 𝜃𝜃2+𝜃𝜃3) + 𝐿𝐿3 sin( 𝜃𝜃2 + 𝜃𝜃3 + 𝜃𝜃4) (24) To find the joint coordinates for a given set of finger tip coordinates (𝑋𝑋,𝑌𝑌,𝜑𝜑) , one needs to solve the above nonlinear equations for𝜃𝜃2,𝜃𝜃3 𝑎𝑎𝑛𝑛𝑎𝑎 𝜃𝜃4.Substituting the last of the three above equations into the other two we can eliminate 𝜃𝜃4 . Then, we have two equations in 𝜃𝜃2 𝑎𝑎𝑛𝑛𝑎𝑎 𝜃𝜃3. 𝑋𝑋 − 𝐿𝐿3 cos(𝜑𝜑) = 𝐿𝐿1 cos 𝜃𝜃2 + 𝐿𝐿2 cos(𝜃𝜃2+𝜃𝜃3) (25) 𝑌𝑌 − 𝐿𝐿3 sin(𝜑𝜑)= 𝐿𝐿1 sin𝜃𝜃2 + 𝐿𝐿2 sin(𝜃𝜃2+𝜃𝜃3) (26) The unknowns have been grouped on the right hand side. The left hand side depends only on the finger tip Cartesian coordinates and is therefore known. Now, renaming the left hand sides, 𝑋𝑋′ = 𝑋𝑋 − 𝐿𝐿3 cos(𝜑𝜑) and 𝑌𝑌′ = 𝑌𝑌 − 𝐿𝐿3 sin(𝜑𝜑), regrouping terms, squaring both sides in each equation and adding them, we get a single nonlinear equation in 𝜃𝜃2 : 2𝐿𝐿1𝑋𝑋′ cos𝜃𝜃2 + 2𝐿𝐿1 𝑌𝑌′ sin𝜃𝜃2 + (𝐿𝐿22 − 𝐿𝐿12 − 𝑋𝑋′2 − 𝑌𝑌′2) = 0 (27) There are two solutions for 𝜃𝜃2in the above equation given by 𝜃𝜃2 = 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛2( 𝑌𝑌′,𝑋𝑋′) ± arccos �𝐿𝐿1 2+𝑋𝑋′2+ 𝑌𝑌′2−𝐿𝐿2 2 2𝐿𝐿1�𝑋𝑋′2+ 𝑌𝑌′2 � (28) Substituting any of these solutions gives us 𝜃𝜃3 = 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛2( 𝑌𝑌′ − 𝐿𝐿1 sin 𝜃𝜃2 ,𝑋𝑋′ − 𝐿𝐿1 cos𝜃𝜃2) −𝜃𝜃2 (29) Substituting 𝜃𝜃3and 𝜃𝜃2 in (21) to find𝜃𝜃4. Thus, for each solution for𝜃𝜃2, there is one solution for 𝜃𝜃3and𝜃𝜃4. 4. Simulation of human hand manual alphabet (ASL). Depending on the derived kinematics of human hand simulation of every (ASL)letter was build using matlab programming where each finger have specific position and orientation in each letter representation these values were used to implement every part of the simulated human hand by substitute these values in the kinematics equations for example letter(D). American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 9 Table 2: angels of fingers that perform letter D Figure 8: representation of letter (D) in Matlab 5. Hardware Design methodology The sensory glove circuit was designed to generate voltage data which are different according to the ASL manual alphabet letter gesture through the use of bending sensors, resistances, capacities, impedance buffer and data acquisition device which is the interface between the sensory glove and the PC, (Figure 9) shows the hardware circuit. Figure 9: sensory glove hardware circuit angels thumb index middle ring little Ө1 0 0 0 0 0 Ө2 85 85 85 0 -38 Ө3 90 90 90 0 15 Ө4 15 15 15 0 105 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 10 5.1 Block diagram Flex sensors sending data that depending on the bending of human hand and fingers to the analog signal processing circuit and then to the DAQ which is the interface between the sensory glove and the PC, as shown in (figure 10). Figure 10: block diagram of the sensory glove system 5.2 Flex Sensor Testing To find the most efficient circuit for the flex sensor that provides the widest range of voltage the sensors were tested in two circuits in each of which the sensor is connected with two capacitors to remove the ripple from the output voltage and it is connected with a resistance which was changed four times with different values (1 kΩ ,5 kΩ, 10 kΩ and 22 kΩ) to check the most suitable resistance that increases the sensor output voltage range. Then the output voltage passes through the impedance buffer to minimize noise. Circuit 1) In the first circuit the flex sensor was connected to the ground and the resistance was connected to the 5 volt power supply as shown in figure (11) R=resistance. V=voltmeter. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 11 OSC=oscilloscope. Table (3) shows the range of the output voltage for the flex sensor with the maximum bending value and with the minimum bending value. Figure 11: the first electrical circuit for flex sensor Table 3: output voltages of the first circuit Resistance (R) Maximum o/p voltage Minimum o/p voltage 1 kΩ 5.3 V 4.8 V 5 kΩ 4.5 V 2.5 V 10 kΩ 3.7 V 1.7 V 22 kΩ 3.8 V 1.6 V Circuit 2) In the second circuit the flex sensor was connected to the 5 volt supply and the resistance was connected to the ground as shown in figure (12). Figure 12: the second electrical circuit for flex sensor American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 12 R=resistance. V=voltmeter. OSC=oscilloscope. Table (4) shows the range of the output voltage for the flex sensor with the maximum bending value and with the minimum bending value. Table 4: output voltages for the second circuit Resistance (R) Maximum o/p voltage Minimum o/p voltage 1 kΩ 5.3 V 4.8 V 5 kΩ 4.5 V 2.5 V 10 kΩ 4 V 1.6 V 22 kΩ 3.8 V 1.6 V The widest range for voltage can be seen from figure (13) when circuit 2 was used with 10 kΩ , so the designed circuit for the flex sensor in this project will be the second circuit with R=10kΩ. Figure 13: range of the output voltage for circuit 1 and 2. 6. Proposed Neural Network Design A multi-layer ANN was proposed to recognize the manual alphabet of ASL. The algorithm used to train the ANN model was a backpropagation algorithm [22]. The inputs to the ANN are the NI DAQ 6212 output voltages which are the values of output voltage of six flex sensors. the designed ANN was a feedforward network having multilayer, the architecture of this network is summarized in Figure (14) where (n, j , h and k) neurons in input, first hidden, second hidden and output layers respectively, n=6 number of inputs of the neural network and the output layer having single neuron k=1. Figure (15) shows the block diagram of the system after American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 13 using ANN for enhancing the recognition process. Inputs to Neural Network Outputs of Neural Network bj bh bk ne t m Ok x1 x2 xn wjm whj wkh bnj bnh bnk Figure 14: the proposed neural network design structure which is a multilayer neural network with two hidden layers Figure 15: block diagram of sensory glove system after using ANN 6.1 Normalization Neural network training can be made more efficient if certain preprocessing steps are performed on the network inputs and targets. Before training, it is often useful to scale the inputs so that they always fall within a specified range. The available data for each variable are scaled to a specific rang of measurements, so as to remove any bias from the combination of its influence. So that a function would be used to normalize the output data of DAQ to the neural network rang and this function is: American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 14 𝑆𝑆 𝑜𝑜=�2∗𝑆𝑆𝑖𝑖−2.8 2.25 �−1 (30) Where: 𝑆𝑆𝑜𝑜=Sensor output after normalization. 𝑆𝑆𝑖𝑖= Sensor output before normalization. 6.2 GUI Software Design for ASL Recognition A graphical user interface window was designed in MATLAB program as a part of the overall system to show the recognized ASL manual alphabet as printed letters or words. Figure (16) shows the designed GUI window. Figure 16: designed GUI window. 6.3 Testing Error for the Proposed Neural Network After training the ANN using the training set of data, the ANN was tested by using testing set of data to check the quality of the three proposed structures of ANN in order to choose the best structure that gives minimum testing error. Figures (17.a, 17.b and 17.c) show the testing error curves for the three proposed structures of ANN. Where the horizontal axis from 1 to 26 represents the letter from A to Z, respectively, and the vertical axis represents error values, the testing set of data was chosen by adding randomly error values to the learning set with the rang of (-0.05, 0.05). The testing error curves indicate that the third proposed ANN (which has 30 and 15 neurons in the 1st and 2nd hidden layers, respectively) having the least testing error values, so that this ANN will be used for the recognition process. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 15 a b c Figure 17: testing error curves for a:(15,7) neurons ,b:(40,20)neurons ,c:(30,15)neurons. 7. Results and discussion 7.1 Simulation results The ASL letters were represented successfully in the MATLAB where the equations of forward kinematics for human hand were used to performing shape of letter and then we used the program to perform complete word and the result was successful and the aim was achieved from this simulation which was converting typed normal language into animated sign language. Figure (18) shows the results of typing word (MECHA) in the simulation program, and figures (19) shows the trajectories of angels for every joint in the thumb finger. M E C H A Figure 18: simulation of word (MECHA) American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 16 Figure 19: The trajectories of thumb joints angles. 7.2 normalization results The sensor outputs were normalized before using neural network. The Sensors output voltage before normalization where shown in figure (20) and figure (21) shows the sensors output voltage after normalization. Figure 20: sensors output voltage before normalization Figure 21: sensors output voltage before normalization American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 17 7.3 Neural Network results In this section, the ASL recognition system is examined for all the 26 letters of ASL, 10 pattern were saved in database. By using the proposed algorithm, the results are listed in Table (5), Figure (22) shows the graph for results of 26 letters. The total rate is 90.19% using neural network for the recognition process. Table 5: results of recognition for 26 ASL letter using NNT. gesture true gesture true A 90 N 95 B 86 O 86 C 90 P 90 D 85 Q 92 E 100 R 90 F 85 S 93 G 90 T 90 H 92 U 85 I 85 V 100 J 86 W 96 K 90 X 85 L 92 Y 92 M 90 Z 90 Figure 22: The recognition graph of 26 ASL letters. 7.4 hardware and neural results The developed ASL alphabet recognition ANN model was first trained with data as one for every letter. When American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 18 that was not successful, we trained with two, three, and ten, finally, five readings for each letter which was effective and the neural network operate successfully and the ASL alphabet letters were recognized figure (23) shows the recognition of letters (A,D,A,M) of ASL alphabet. A D A M Figure 23: ASL alphabet recognition References [1] D. M. Perlmutter. “what is sign language,” Linguistic Society of America. Washington, DC 20036- 6501. Available: https://www.lsadc.org/. [2] Cobos, Salvador, et al. "Simplified human hand models for manipulation tasks." Cutting Edge Robotics 2010. InTech, 2010. [3] Huenerfauth, Matt. "A multi-path architecture for machine translation of English text into American Sign language animation." Proceedings of the Student Research Workshop at HLT-NAACL 2004. Association for Computational Linguistics, 2004. [4] Zhao, Liwei, et al. "A machine translation system from English to American Sign Language." Envisioning machine translation in the information future (2000): 191-193. [5] Morrissey, Sara, and Andy Way. "An example-based approach to translating sign language." (2005). [6] Kim, Taehwan, Karen Livescu, and Gregory Shakhnarovich. "American sign language fingerspelling https://www.lsadc.org/ American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 19 recognition with phonological feature-based tandem models." Spoken Language Technology Workshop (SLT), 2012 IEEE. IEEE, 2012. [7] Pugeault, Nicolas, and Richard Bowden. "Spelling it out: Real-time asl fingerspelling recognition." Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE, 2011. [8] Wysoski, Simei G., et al. "A rotation invariant approach on static-gesture recognition using boundary histograms and neural networks." Neural Information Processing, 2002. ICONIP'02. Proceedings of the 9th International Conference on. Vol. 4. IEEE, 2002. [9] Pansare, Jayashree R., Shravan H. Gawande, and Maya Ingle. "Real-time static hand gesture recognition for American Sign Language (ASL) in complex background." Journal of Signal and Information Processing 3.03 (2012): 364. [10] Dong, Cao, Ming C. Leu, and Zhaozheng Yin. "American sign language alphabet recognition using microsoft kinect." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2015. [11] Allen, Jerome M., Pierre K. Asselin, and Richard Foulds. "American Sign Language finger spelling recognition system." Bioengineering Conference, 2003 IEEE 29th Annual, Proceedings of. IEEE, 2003. [12] Shembade, Yateen P. Design and Simulation of a Mechanical Hand. Rochester Institute of Technology, 2012. [13] De Marco, Robert Michael, and R. A. Foulds. "Data recording and analysis of American Sign Language." Bioengineering Conference, 2003 IEEE 29th Annual, Proceedings of. IEEE, 2003. [14] Wang, Honggang, Ming C. Leu, and Cemil Oz. "American Sign Language Recognition Using Multi- dimensional Hidden Markov Models." Journal of Information Science and Engineering 22.5 (2006): 1109-1123. [15] Butterworth, Rod R. The Perigee Visual Dictionary of Signing: An A-to-Z Guide to Over 1,350 Signs of American Sign Language. Penguin, 1995. [16] Sternberg, Martin LA. American sign language dictionary. HarperPerennial, 1998. [17] van der Hulst, Frank PJ, et al. "A functional anatomy based kinematic human hand model with simple size adaptation." Robotics and Automation (ICRA), 2012 IEEE International Conference on. IEEE, 2012. [18] Lowe, Whitney. Orthopedic Assessment in Massage Therapy. Daviau Scott, 2006. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 47, No 1, pp1-20 20 [19] Schilling, Robert J. Fundamentals of robotics: analysis and control. Simon & Schuster Trade, 1996. [20] Hind Z. Khaleel, ''Inverse Kinematics Solution for Redundant Robot Manipulator using Combination of GA and NN,” Al-Khwarizmi Engineering Journal, Vol. 14, No. 1, pp. 136-144, 2018. [21] Spong, Mark W., Seth Hutchinson, and Mathukumalli Vidyasagar. Robot modeling and control. Vol. 3. New York: Wiley, 2006. [22] Firas A. Raheem, Azad R. Kareem and Amjad J. Humaidi, 'Inverse Kinematics Solution of Robot Manipulator End-Effector Position Using Multi-Neural Networks', Eng. &Tech.Journal, Vol.34,Part (A), No.7, 2016.