 Kurdistan Journal of Applied Research (KJAR) | Print-ISSN: 2411-7684 – Electronic-ISSN: 2411-7706 | kjar.spu.edu.iq Volume 2 | Issue 3 | August 2017 | DOI: 10.24017/science.2017.3.37 Human gait identification using Kinect sensor Mohammed H. Ahmed Department of Computer Science University of Raparin Ranyah ,Sulaimanyah Kurdistan Region of Iraq mohammedhussein@raparinuni.org Abdulbasit K. Faeq Department of Software Engineering FENG, Koya University, Koya, Iraq abdulbasit.faeq@koyauniversity.org Azhin Tahir Sabir Department of Software Engineering FENG, Koya University, Koya, Iraq azhin.sabir@koyauniversity.org Halgurd S. Maghdid Department of Software Engineering FENG, Koya University, Koya, Iraq halgurd.maghdid@koyauniversity.org Abstract: This study investigates a novel three- dimension gait recognition approach based on skeleton representation of motion by the cheap consumer level camera Kinect sensor. In this work, a new exemplification of human gait signature is proposed using the spatio-temporal variations in relative angles among various skeletal joints and changing of measured distance between limbs and land. These measurements are computed during one gait cycle. Further, we have created our own dataset based on Kinect sensor and extract two sets of dynamic features. Nearest Neighbors and Linear Discriminant Classifier (LDC) are used for classification. The results of the experiments show the proposed approach as an effective and human gait recognizer in comparison with current Kinect-based gait recognition methods. Keywords: Gait recognition, Kinect, Angle Features (AF), Distance Feature (DF), motion analysis. 1. INTRODUCTION Biometrics uses human biological characteristics for person identity verification. It can deal with physiological characteristics such as: iris, face, fingerprint, finger knuckles, DNA, and hand Geometry. Biometric also deals with human behavioral characteristics, like human voice, signature, and gait [1]. The automatic identification of any person has widely and vitally grown recently, especially in areas with high demands of security such as check points, airports and banks [2]. The importance of biometric has been appeared based on the fact that each person has recognizable characteristics that can uniquely identify a human. Gait is one of the biometrics, that can be used to distinguish individuals (from distance) based on human walking style. Psychological studies show that humans is capable to recognize people from their way and style of walking [1]. Unlike other biometrics, gait has the advantage of offering the ability to recognize human from a distance at a non-high resolution representation. Additionally, it requires no user cooperation, at any certain level [3]. Gait recognition models analyzes video stream sequence recordings. Kinect sensor device (produced by Microsoft) is one of the tools that supports this kind of analysis. The Microsoft Kinect tool is a product used initially with Xbox 360 gaming console, where the user can command games by body motion from a distance with no connected sensor. Kinect is also able to track various types of human involved information such as: face geometry, human skeleton and also capturing the depths axis of the image. Kinect camera could be more applicable with Kinect SDK for Windows along with the set of APIs, such that it offers an effective tool to improve a gait recognition application [4]. The current applications of gait recognition are mostly using normal cameras that records a video sequences. However, these kind of recordings face many difficulties in the feature extraction level. Consequently, some researchers locate dot points in manual way on the subject in each frame[5] and [6]. Determining points manually along all these motion based methods is time consuming and complex approach at the feature extraction stage. On the other hand, Kinect can offer these feature more accurately in addition to provide body skeleton details. 2. RELATED WORK Various gait recognition approaches have been applied in the literature which are two categories: i) model-based approaches and ii) model-free approaches [7]. Basically, person's body parts (legs, arms, etc.) are represented by explicit models in model based approaches [8]. Along these models, the gait signature is represented through the parametric values differentiating over time, that are estimated in each frame. However, model construction, model fitting and parameter estimation, in the model based approaches, are time-consuming and costly in terms of computation [8]. Consequently, they are unsuitable for a wide range of real life applications. The early parametric gait recognition method was proposed by BenAbdelkader et al. [9], where they estimated two spatiotemporal parameters of gait, namely stride length and cadence as two distinctive biometric traits. Following to that, Urtasun and Fua [10] use 3-D temporal motion model on a synchronized video sequences to analyze gait. The individual gait signature is characterized later through the recovered motion parameters. In a similar way suggested by Yam et al. [11] human leg motion and structure is modeled to differentiate between gait signatures extracted from walking and running samples. This method offers an effective way to view and scale independent gait representation, however, the model costly in terms of computation and sensitive to the quality of the gait sequences [12]. A model free approach utilizes the silhouette as a whole in order to construct a compact representation of walking motion Instead of modeling individual body parts [8]. Another tow representation are Gait Energy Image (GEI) [13] and Motion Energy Image (MEI) [14] that are two widely used model-free in gait recognition methods. The MEI approach basically depends on a temporal vector mailto:mohammedhussein@raparinuni.org mailto:abdulbasit.faeq@koyauniversity.org mailto:halgurd.maghdid@koyauniversity.org image. In MEI, the vector axis wieght, is a function of the motion characteristics at the corresponding frame sequences [14]. While, GEI collect all the silhouette motion sequences of one gait cycle in a single image. Here, the temporal information is preserved as well [13]. Researchers that use model-free methods extend recently the GEI to a more effective exemplification. For instance, in [15] the authors proposed Frame Difference Energy Image (FDEI), to apply de-noising and clustering to decrease the effect of uncompleting of silhouette. While, in [16], the authors utilize both Head Energy Image (HEI) and Foot Energy Image (FEI) in order to structure a rich information representable energy image. Model-free methods are cheap in terms of computation, however, they are sensitive to view and scale variations, and as a consequence, not convenient in out of control situations. In the past twenty years, gait recognition has been studied and Kinect has shared in to the spike in the interest in Kinect data-based gait recognition. Kinect is a cheap consumer-level device contain an array of sensors, regarding a color camera, a depth sensor, and a multi- array microphone setup. Additionally, it can record and structure a real time 3D virtual skeleton of human body [17], such that it disposes the time consumer video processing stages. These characteristics of Kinect have make it applicable in different real-life problems, such as smart house [18], health care [19], surveillance [20], etc. The simple real-time skeleton feature extracting has impact on some recent gait recognition studies. For instance, the approach proposed by Ball et al. [21] adopt gait data clustering using Kinect. Here, the lower half of the body is used for feature extraction. Another example is the work of Preis et al. [1] who proposed a Kinect skeleton-based approach for gait recognition using 13 mostly static biometric parameters: height, the length of legs, torso, the two lower legs, the two thighs, the two upper arms, the two forearms, step-length, and speed. Unlike gate biometric features, these features are not behavioral. Gait is more related to the movement patterns of body parts during locomotion. Gabel et al. [22] used the changes in the location of these skeletons between consecutive frames as a feature. However, the work is only evaluated for gait parameter extraction not for identification. 3. PROPOSED METHOD The proposed model in this study, aim to recognize gait using the skeleton provided by Kinect sensor. Kinect sensor along with SDK v 1.6, offers a human skeleton for up to two persons in an acceptable quality at the same time. Additionally, Kinect provides depth dimension image and RGB image . However, this study utilizes the skeleton model alone. Basically, the proposed system includes four components: Firstly, create a database based on skeleton information using Kinect. Secondly, detects gait cycle for each subject. Third, feature will be extracted in two different sets: Distance Features and Angle Distance Features described later. Finally, Nearest Neighbour (NN) and LDC used as classification method shown in Figure 1 Figure 1 System overview. 3.1.Database Based on the fact that the Kinect is somewhat new, there is no available public standard database. Hence, it was compulsory to develop a specific dataset using Kinect sensor. We check the accuracy of the Kinect camera according to the distance between the camera and the person. Ten short videos are recorded for 5 subjects at various distance like 3.5m, 3m and 2.5m. By a comparison of how accurate these distances are, we select 2.5m as the distance between Kinect and the person, where the skeleton information at this distance appear clearer than the use of other distances and offers more gait cycles. In addition to that, the Kinect sensor was fixed at 0.6m height, with angle of zero and recording speed at around 30 frames per second. The conducted experiment uses 16 persons who were walking in front the Kinect sensor, from right to left (angle 90). The subjects are told to walk “naturally” for 5 times, end up with an 80 videos. See Figure 2 Figure 2 Kinect dataset [23] 3.2. Gait Cycle A normal human style of walking starts from a specific heel-strike of a one foot to the same heel-strike after a period of walking. The stance phase of the gait cycle represent the first portion which locate 60 percent of the cycle. However, the swing phase is the rest 40 percent of the cycle [24]as in figure 3. Figure 3: Gait cycle detection.[25] In general, gait cycle can be detected based on two main techniques. The first technique is called double support method, in which both feet move away from each other. The second technique is the local minima, where the distance between the two feet is at the minimum [27]. In this study, double support phase technique is used to detect the gait cycle and the ankles distances is used instead of feet. Ankles distance =|| right ankle (x) –left ankle (x)|…(1) 3.3. Feature Extraction Kinect sensor with SDK provides human body skeleton and tracking of 20 different skeletal points. Therefore, extracting feature based on Kinect sensor is totally different from the techniques that used in normal cameras. As a result, our database includes (x,y) point values for 20 joints, and totally produces 40 attributes. Based on these points, we have extracted a set of dynamic features: will be explained in the next section. 3.3.1. Distance Feature (DF) The DF features is extracted according to the changes of the distance between skeleton joint and the floor toward the Y-axis. The suggested gait features here are: human height DF1 and the height of (right wrist DF2, right shoulder DF3, right and left Ankles DF4 and DF5). Then, the mean and the standard deviation for all type of these distances in one gait cycle is computed. The final proposed feature is the triangle based area of the (Hip center and straight line between the left and the right Feet DF6), mean of DF6 is computed in one gait cycle and concatenated to the feature vector. The following equations shows the way of constructing proposed gait feature vectors: (see figure 4). DF1= Di (Vy (4) - L (0)). ………………...…(2) ……………….…(3) ……………........(4) …………..…...(5) ……………...…...(6) DF6= 1/2 *(| Vx(20) – Vx(16)|) * Vy(1). …..…(7) { } { } DFF = {MeanDF, StdDF}. ……….……….……(8) Where Di is distance, L is land line, Vy is vertical joint, and m is the number of frames during one gait cycle. 3.3.2. Angle Feature (AF) The proposed Angle Feature set is based on the spatio- temporal variation in proportional angles among various skeletal joints. Here we suggested eight features to represent human body (see figure 4) which are: right and left ankle angles (AF1,AF2), right and left knee angles (AF3,AF4), right and left hip angles (AF5, AF6) right and left elbow angles (AF7, AF8). Then, the mean and the standard deviation for all angle values are computed in one gait cycle. Equations 9 show how to calculate the angles between vectors that starts from each skeletal joints. | || | ……………………………………..(9) Figure 4 (DF) and (AF) features. 4. EXPERIMENTAL RESULTS AND DISCUSSION We created our dataset to test the performance of the proposed method, the dataset is recorded based on Kinect sensor and contain 16 subjects captured from side view, and each subject has 5 records. The data set includes a collection of (x, y) point values that demonstrate the position of each joint in one gait cycle. Next to collecting the date and detecting the gait cycle. Now we extract two set of significant features which are ready to be used in the process of gait recognition, called DF and AF (discussed before). To create training (gallery) and test (probe) set, 2 samples from each subject have been selected for the test set and the remaining 3 samples for the training set. To provide realistic scenario we have used all the possible probability of selecting the samples for the training and test set. In our method NN and LDC are used to classify human gait. The system is tested based on each set of feature individually then we fused them based on the future level fusion. In the case of NN classifier, when DF tested the system achieved 74.64% recognition accuracy, while AF provided 67.18% recognition accuracy. The same pattern of being DF behave better than AF is appeared in the case of LDC classifier (89.68%, 67.18%). Which shows the better discriminative ability of DF for identity than AF features. However, by combining both of the DF and AF Features, the recognition rate increased to 85.62% and 94.37% for both NN and LDC respectively. This improvement in the accuracy shows the complementary characteristic of one AF to DF and vice versa. Moreover, LDC classifier, seem to be more superior to NN using this data in all of the cases. LDC classifier is a simple classifier and behave very well with data that have Gaussian distribution. And generally all of the data for a specific class will participate in making whether the new point belong to the current class or another one. This might have influence on the improved accuracy of LDC over NN classifier. Table 1: Recognition Rate Feature set Accuracy (LDC) Accuracy (NN) DF 89.68% 74.64% AF 67.18% 67.18% DF and AF 94.37% 85.62% 5. CONCLUSION This paper started by collecting the skeleton information for 16 persons who were used to walk in front of the Kinect sensor. Kinect provides new field of gait parameters and more powerful skeleton based features for gait recognition. In this paper we study two new features for gate recognition: Distance Features (DF) and Angle Features (AF). Two different classification method (NN and LDC) are used to test the performance of the proposed method. DS and AF are tested separately, then combined based on the feature level fusion. In the case of using NN, The proposed method achieved 74.64%, 67.18%, and 85.62% as a recognition rate for DF, AF, and (DF+AF) respectively. Father when method tested based on LDC the recognition rate increased for each of DF, AF and (DF+AF) and provided 89.68%, 67.18% and 94.37% respectively. In future work we aim to expand the database and record more participant with carrying bag or wearing coat to see how the system can deal with these challenges. 6. REFERENCE [1] J. Preis, M. Kessel, M. Werner, and C. Linnhoff- Popien, "Gait recognition with kinect," in 1st international workshop on kinect in pervasive computing, 2012, pp. P1-P4. [2] R. Das, "An introduction to biometrics," Military Technology, vol. 29, pp. 20-27, 2005. [3] M. Kumar and R. V. Babu, "Human gait recognition using depth camera: a covariance based approach," in Proceedings of the Eighth Indian Conference on Computer Vision, Graphics and Image Processing, 2012, p. 20. [4] A. Jana, Kinect for windows SDK programming guide: Packt Publishing Ltd, 2012. [5] J. P. Singh and S. Jain, "Person identification based on gait using dynamic body parameters," in Trendz in Information Sciences & Computing (TISC), 2010, 2010, pp. 248-252. [6] A. K. Jhapate and J. P. Singh, "Gait Based Human Recognition System using Single Triangle," International Journal of Computer Science and Technology, pp. 128-131, 2011. [7] J. Han and B. Bhanu, "Statistical feature fusion for gait-based human recognition," in Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, 2004, pp. II-II. [8] J. Wang, M. She, S. Nahavandi, and A. Kouzani, "A review of vision-based gait recognition methods for human identification," in Digital Image Computing: Techniques and Applications (DICTA), 2010 International Conference on, 2010, pp. 320-327. [9] C. BenAbdelkader, R. Cutler, and L. Davis, "Stride and cadence as a biometric in automatic person identification and verification," in Automatic Face and Gesture Recognition, 2002. Proceedings. Fifth IEEE International Conference on, 2002, pp. 372- 377. [10] R. Urtasun and P. Fua, "3D tracking for gait characterization and recognition," in Automatic Face and Gesture Recognition, 2004. Proceedings. Sixth IEEE International Conference on, 2004, pp. 17-22. [11] C. Yam, M. S. Nixon, and J. N. Carter, "Automated person recognition by walking and running via model-based approaches," Pattern Recognition, vol. 37, pp. 1057-1072, 2004. [12] A. Sinha, K. Chakravarty, and B. Bhowmick, "Person identification using skeleton information from kinect," in Proc. Intl. Conf. on Advances in Computer-Human Interactions, 2013, pp. 101-108. [13] J. Man and B. Bhanu, "Individual recognition using gait energy image," IEEE transactions on pattern analysis and machine intelligence, vol. 28, pp. 316- 322, 2006. [14] A. F. Bobick and J. W. Davis, "The recognition of human movement using temporal templates," IEEE transactions on pattern analysis and machine intelligence, vol. 23, pp. 257-267, 2001. [15] C. Chen, J. Liang, H. Zhao, H. Hu, and J. Tian, "Frame difference energy image for gait recognition with incomplete silhouettes," Pattern Recognition Letters, vol. 30, pp. 977-984, 2009. [16] X. Li and Y. Chen, "Gait recognition based on structural gait energy image," Journal of Computational Information Systems, vol. 9, pp. 121-126, 2013. [17] J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, M. Finocchio, A. Blake, M. Cook, and R. Moore, "Real-time human pose recognition in parts from single depth images," Communications of the ACM, vol. 56, pp. 116-124, 2013. [18] E. E. Stone and M. Skubic, "Evaluation of an inexpensive depth camera for passive in-home fall risk assessment," in Pervasive Computing Technologies for Healthcare (PervasiveHealth), 2011 5th International Conference on, 2011, pp. 71-77. [19] Y.-J. Chang, S.-F. Chen, and J.-D. Huang, "A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities," Research in developmental disabilities, vol. 32, pp. 2566-2570, 2011. [20] M. Popa, A. K. Koc, L. J. Rothkrantz, C. Shan, and P. Wiggers, "Kinect sensing of shopping related actions," in International Joint Conference on Ambient Intelligence, 2011, pp. 91-100. [21] A. Ball, D. Rye, F. Ramos, and M. Velonaki, "Unsupervised clustering of people from'skeleton'data," in Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction, 2012, pp. 225-226. [22] M. Gabel, R. Gilad-Bachrach, E. Renshaw, and A. Schuster, "Full body gait analysis with Kinect," in Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE, 2012, pp. 1964-1967. [23] M. H. Ahmed and A. T. Sabir, "Human Gender Classification Based on Gait Features Using Kinect Sensor," in Cybernetics (CYBCONF), 2017 3rd IEEE International Conference on, 2017, pp. 1-5. [24] A. Sabir, N. Al-jawad, and S. Jassim, "Gait recognition using spatio-temporal silhouette-based features," in Proc. of SPIE Vol, 2013, pp. 87550R- 1. [25] M. H. Ahmed, "Kinect-Based Human Gait Recognition Using Static and Dynamic Features," International Journal of Computer Science and Information Security, vol. 14, p. 425, 2016.