FACTA UNIVERSITATIS Series: Electronics and Energetics Vol. 34, No 4, December 2021, pp. 557-567 https://doi.org/10.2298/FUEE2104557R © 2021 by University of Niš, Serbia | Creative Commons License: CC BY-NC-ND Original scientific paper FEATURE EXTRACTION FOR PERSON GAIT RECOGNITION APPLICATIONS Adnan Ramakić1, Zlatko Bundalo2, Željko Vidović3 1Rectorate, University of Bihać, Bihać, Bosnia and Herzegovina 2Faculty of Electrical Engineering, University of Banja Luka, Banja Luka, Bosnia and Herzegovina 3University of East Sarajevo, Faculty of Transport and Traffic Engineering, Doboj, Bosnia and Herzegovina Abstract. In this paper we present some features that may be used in person gait recognition applications. Gait recognition is an interesting way of people identification. During a gait cycle, each person creates unique patterns that can be used for people identification. Also, gait recognition methods ordinarily do not need interaction with a person and that is the main advantage of these methods. Features used in a person gait recognition methods can be obtained with widely available RGB and RGB-D cameras. In this paper we present a two features which are suitable for use in gait recognition applications. Mentioned features are height of a person and step length of a person. They may be extracted and were extracted from depth images obtained from RGB-D camera. For experimental purposes, we used a custom dataset created in outdoor environment using a long-range stereo camera. Key words: Gait Recognition, Gait Energy Image, Backfilled Gait Energy Image, Height of a Person, Step Length of a Person. 1. INTRODUCTION People may be identified using different biometric methods. Examples of these methods are fingerprint, retina and iris recognition (identification based on eye features), facial recognition, keystroke dynamics, voice recognition etc. Generally, they may be divided in physiological and behavioral biometric methods. Physiological biometric methods include fingerprint, retina and iris recognition, hand geometry, facial recognition etc., while behavioral biometric methods include methods like keystroke dynamics, voice recognition, person signature recognition, gait recognition etc. Received April 3, 2021; received in revised form August 14, 2021 Corresponding author: Zlatko Bundalo Faculty of Electrical Engineering, University of Banja Luka, 5 Patre, 78 000 Banja Luka, Bosnia and Herzegovina E-mail: zlatbun2007@gmail.com 558 A. RAMAKIĆ, Z. BUNDALO, Ž. VIDOVIĆ Most of the above listed methods need some kind of interaction with a person during an identification process. On the other hand, gait recognition is a method that ordinarily does not need any interaction with a person during identification process. Using some type of long- range cameras (e.g. ZED stereo camera) some facial recognition methods also may be conducted without interaction with a person. Today’s gait recognition approaches, which are in use, are model-based or appearance-based. Model-based approach ordinarily exploits different parts of human body to create a model that are in use for identification purposes. Some of human body parts that are ordinarily in use with model-based approaches are legs, arms, etc. In other words, some measures related to mentioned body parts are in use (e.g. arm length). Appearance-based approach ordinarily uses persons’ silhouette representations. Research related to gait recognition usually has been done using RGB or RGB-D cameras (also sometimes called RGB and RGB-D sensors) and datasets that were created with them. Earlier were used RGB cameras, but today in use also are RGB-D cameras. RGB-D camera provides depth data along with RGB data. The most of research that has been done using RGB-D cameras were realized using Kinect sensor developed by Microsoft. Kinect sensor provides RGB data along with depth data. In this paper we analyze some features that may be used along with well-known gait recognition methods. These features are height of a person and step length of a person. Both features may be obtained from depth images of RGB-D camera. Gait recognition methods that were used along with mentioned features are, appearance-based methods, GEI (Gait Energy Image) [1] and BGEI (Backfilled Gait Energy Image) [2]. GEI is an image that contains silhouettes (aligned, normalized and averaged) of a person over a gait cycle. BGEI is similar to the GEI, also represents an image with a person silhouettes, but silhouettes of a person are back filled from front most pixels. 2. RELATED WORK In the field of gait recognition there is a large number of works. Approaches that deal with gait recognition are usually divided in two types: model-based approach and appearance-based approach. Model-based approach uses explicit models to represent and track different parts of human body (such as, e.g. legs or arms) over time while appearance- based approach ordinarily uses human silhouettes that are extracted from RGB or depth images. Appearance-based approach usually does not use explicit models. In this paper we generally focused on some appearance-based approaches. Han and Bhanu [1] presented a spatiotemporal gait representation called Gait Energy Image (GEI). GEI is an image that contains averaged silhouettes, normalized and aligned, of a person during a gait cycle. Sivapalan et al. [3] presented a Gait Energy Volume (GEV). Authors [3] extended GEI with a 3D and used reconstructed voxel volumes instead of temporally averaging segmented silhouettes. Sivapalan et al. [2] also presented Backfilled Gait Energy Image (BGEI). BGEI is a feature that may be constructed using side-view silhouettes or frontal depth images. BGEI is an image, such as GEI, with a person silhouettes, but silhouettes of a person are back filled from front most pixels. Feature Extraction for Person Gait Recognition Applications 559 Hofmann et al. [4] used depth information with a GEI in a way that GEI required silhouettes were calculated using a depth data. Also, authors [4] proposed and a feature defined as Depth Gradient Histogram Energy Image (DGHEI). Arora and Srivastava [5] presented Gait Gaussian Image (GGI), a period based gait technique that is used for feature extraction of gait image during a gait cycle. Iwashita et al. [6] presented an approach in which an image that contains a human body is divided in multiple areas. For every mentioned area features are extracted and used in gait recognition process. Ramakić et al. [7] and Lenac et al. [8] presented approaches where they used appearance- based methods, such as GEI and BGEI, and height feature obtained from depth images for gait recognition tasks. Lenac et al. [8] presented HGEI-i and HGEI-f methods where in case of first method, HGEI-i, early fusion of information is realized while HGEI-f performs late fusion of information. HGEI-i represents a method for gait recognition where are combined GEI features and feature height of a person in a way that height of a person is added as one of a features alongside features obtained from GEI. Then, classification process is realized after integration of GEI features and height of a person feature. In HGEI-f method, GEI and height of a person are separately considered and based on results from each classifications single prediction is made. Ramakić et al. [9] also presented a method for gait recognition that exploits silhouettes of a person along with height of a person and step length of a person. Chattopadhyay et al. [10] presented a Pose Depth Volume (PDV) feature for frontal gait recognition. Bashir et al. [11] proposed a gait representation called Gait Entropy Image (GEnI). GEnI encodes in a single image the randomness of pixel volumes in the silhouette images over a gait cycle. Portillo-Portillo et al. [12] presented an approach for a gait recognition. Mentioned approach exploits GEI and Direct Linear Discriminant Analysis (DLDA) in order to create a view invariant model for identification. Lishani et al. [13] proposed an approach which is based on the Haralick features extracted from GEI. Rudek et al. [14] presented a method for a gait classification based on analysis of the trajectory of pressure centers extracted from a feet contact point with a ground. 3. FEATURE EXTRACTION Features that were extracted and used in gait recognition process are real height of a person and step length of a person. The main idea in this paper is using additional features along with well-known appearance-based gait recognition methods such as GEI and BGEI. 3.1. GEI and BGEI GEI represents person’s silhouettes (aligned, normalized and temporally averaged) over a gait cycle in one image. GEI is defined as: 560 A. RAMAKIĆ, Z. BUNDALO, Ž. VIDOVIĆ  = = N t tjiI N jiG 1 ),,( 1 ),( (1) where N is the silhouette frames number in gait cycle, t is the frame number in gait cycle at moment of time, I(i, j) is the original silhouette image with (i, j) values in the image coordinate. Examples of GEI images for three people and for three different datasets are shown in Fig. 1. First row shows GEI images from own custom dataset created with a long-range stereo camera, second row shows GEI images from well-known Casia Dataset B [15] [16] [17] and third row shows GEI images also from well-known TUM Gait from Audio, Image and Depth (GAID), TUM-GAID [18] dataset. Fig. 1 Examples of GEI images, for three different people, from own custom dataset, Casia Dataset B [15] [16] [17], and TUM-GAID [18] dataset BGEI is created as well as GEI, with difference that BGEI is constructed by first back filling the binary silhouettes, where the front most pixel on each row is found and from it Feature Extraction for Person Gait Recognition Applications 561 filled to the back of the image. Examples of BGEI images, for three different people, for our own custom dataset and TUM-GAID dataset are shown in Fig. 2. Fig. 2 Examples of BGEI images, for three people, from own custom dataset and TUM-GAID [18] dataset 3.2. Height of a person and step length of a person Height of a person and step length of a person were extracted from depth images. Depth images contain information about distance of specific object from a camera. Using mentioned information real height of a person and step length of a person may be estimated. These two features are conceptually simple and robust features which can be easily extracted and combined with some gait recognition methods. Independent use of height of a person or step length of a person does not represent reliable way of identification, but in a combination with some gait recognition method can improve overall identification score. Height of a person may be estimated from depth image as a distance between top- most point on person’s silhouette and ground plane. Top-most point on silhouette is represented as vector that contains the fallowing values (Eq. (2)): 2 1 x y P w      =       (2) where x and y are 2D image pixel coordinates. Label w represents a disparity value. Ground plane can be detected using Random Sample Consensus (RANSAC). RANSEC represents 562 A. RAMAKIĆ, Z. BUNDALO, Ž. VIDOVIĆ a plane detection method that are often in use in point cloud data. Output from this step are A, B, C, and D plane parameters. In order to obtain 3D coordinates for the top-most point in Cartesian coordinate system perspective transformation matrix was used. Mentioned matrix is obtained through camera calibration process. This is shown in Eq. (3): , 1 0 0 0 1 0 0 0 0 1 0 0 x y x x x x c c fQ c c T T −    −    =   −−      (3) If a perspective transformation matrix is known, matrices multiplication should be done. This is shown in Eq. (4): 3 2 P Q P=  (4) Result is a fallowing 3D point vector as shown in Eq. (5): 3 ' ' ' ' X Y P Z W      =       (5) In order to obtain metric system values and real values for distance it is necessary to divide all vector elements with W as shown in Eq. (6) and Eq. (7): 3 3 ' /P P W= (6) 3 1 X Y P Z      =       (7) In order to obtain distance between top-most point (Eq. (7)) and ground plane, Eq. (8) is used. 2 2 2 A X B Y C Z D d A B C  +  +  + = + + (8) Obtained distance d represents a height of a person. Also height of a person may be estimated based on height of a person’s silhouette in depth image. In that case, two points are necessary: top-most point and bottom point between legs of a person. Both points can be calculated as described in previous text for the top-most point. Distance between these Feature Extraction for Person Gait Recognition Applications 563 two points represents a height of a person. In this paper height of a person was estimated as a distance amongst two points, top-most point and bottom point. This is illustrated in Fig. 3. Step length of a person was estimated as a distance between two points, defined on left and right leg which is also illustrated in Fig. 3. Average value for step length of a person, which is obtained over a gait cycle, was used. Fig. 3 Depth image with estimated height of a person and step length of a person (in meters) 4. EXPERIMENTAL SETUP In this paper we used an own custom dataset that contains 14 persons. All 14 persons in dataset are in normal walk. Dataset was recorded in outdoor environment. Creation of dataset was conducted using a ZED long-range stereo camera. All of mentioned persons in dataset walking without any accessories (e.g. backpack) and they captured with 90 degrees’ angle. Experimental test was conducted using Matlab. For every of mentioned 14 persons there is a six GEI and BGEI images as well as height and step length values for each person. We used Bag of features with VocabularySize of 500 (default value in Matlab). Mentioned value corresponds to k in k-means clustering algorithm that was used on extracted feature descriptors. We also defined PointSelection as a Detector. PointSelection is a selection method for picking point location. From GEI and BGEI images feature points were selected using Speeded-Up Robust Features (SURF) algorithm. People identification was realized using Classification process. Classification was realized using Support Vector Machine (SVM) algorithm. After features was obtained dimensionality reduction was conducted using Principal Component Analysis (PCA). Also Cross-validation was used, i.e. five-fold Cross-validation. That means that an original dataset is partitioning in a way that there is a subset to train algorithm and remaining data for testing. In case of k-fold Cross-validation data are partitioned in k randomly chosen subsets or folds where subsets are approximately equal size. One subset is used to validate the model 564 A. RAMAKIĆ, Z. BUNDALO, Ž. VIDOVIĆ trained using the remaining subsets. Process is repeated k times. Fig. 4. shows a steps during a features extraction and classification. Fig. 4 Steps during a features extraction Steps shown in Fig. 4 can be described as fallows. When depth images are available silhouettes of a person can be extracted and also height of a person and step length of a person can be estimated. Before that, it’s necessary to do person segmentation from depth image. Silhouettes of a person can be also extracted from RGB images. After person segmentation from depth image, silhouettes of a person can be extracted and height of a person and step length of a person can be estimated. If silhouettes of a person are Feature Extraction for Person Gait Recognition Applications 565 available, GEI or BGEI (depending on what is being created) can be created. When GEI or BGEI images are available for each person, features can be extracted from them. Features from GEI or BGEI images and height of a person and step length of a person are than classified using SVM algorithm. In final step there are results of classification. 5. RESULTS Experimental research has been done using dataset with 14 people in gait. Methods that were tested are GEI, BGEI, GEI along with height and step length of a person (GEI- Height-Step Integration), BGEI along with height and step length of a person (BGEI- Height-Step Integration) and only height of a person as a feature. Step length of a person was not tested as a stand-alone feature because it is not too reliable for people identification when it is only feature for identification. When height of person was used it means that only values of height were classified with SVM classifier. In case of GEI or BGEI only extracted features were classified using SVM classifier. When BGEI and GEI were used along with height and step length of a person that means that height and step length values were added as additional features alongside features from GEI and BGEI and that together classified with SVM classifier. Results of classification are shown in Table 1. Table 1 Classification results Methods Results with SVM Classifier Height feature 83.3% BGEI 85.7% GEI 97.6% BGEI + Height and Step Length (BGEI-Height-Step Integration) 94.0% GEI + Height and Step Length (GEI-Height-Step Integration) 100.0% Height as a method for a people identification has lowest result and that is expected because many people have same or similar height. BGEI has better result than height of a person but lower compared to GEI. GEI as a method for people gait recognition has good overall result of 97.6%. In cases when height and step length of a person were used along with GEI and BGEI overall results of identification were improved. In case when BGEI was used along with height and step length of a person (BGEI- Height-Step Integration) result is 94% in regards to result of 85.7% when BGEI was used stand-alone as a method for people gait identification. Also, in case of GEI, result of GEI-Height-Step Integration was better in regards to using a GEI as a stand-alone method. Result in case when only GEI was used is 97.6% while in GEI-Height-Step Integration is 100%. Fig. 5 shows obtained results for all used method while in Fig. 6 is shown comparison between used methods. 566 A. RAMAKIĆ, Z. BUNDALO, Ž. VIDOVIĆ Fig. 5 Obtained results for used methods Fig. 6 Comparison between used methods 6. CONCLUSION Gait recognition is interesting way for identification. People may be identified with different methods such as fingerprint, retina and iris recognition, facial recognition, voice recognition etc. Most of above mentioned methods ask for some interaction with a person during identification process. Gait recognition methods ordinarily do not need any interaction with a person during a process of identification and that is the main advantage of these type of methods. There are two approaches that deal with gait recognition, model-based approach and appearance-based approach. In this paper we presented some additional features that can be used along with well-known appearance-based gait recognition methods such as GEI and BGEI. Mentioned additional features are height and step length of a person. Feature Extraction for Person Gait Recognition Applications 567 Experimental research was point out that integration of features some well-known gait recognition methods, such as GEI and BGEI, and additional features such as height and step length of a person significantly improves accuracy of identification in regards to stand-alone use of appearance-based gait recognition methods. Experimental results show that in case when GEI-Height-Step Integration was used people identification result is approximately 100% for used own custom dataset. REFERENCES [1] J. Han and B. Bhanu, "Individual recognition using gait energy image",” IEEE Тrans. Pattern Anal. Mach. Intell., vol. 28, no. 2, pp. 316–322, Feb. 2006. [2] S. Sivapalan, D. Chen, S. Denman, S. Sridharan and C. Fookes, "The backfilled gei-a cross-capture modality gait feature for frontal and side-view gait recognition", in Proceedings of the International Conference Digital Image Computing Techniques and Applications (DICTA), IEEE, 2012, pp. 1–8. [3] S. Sivapalan, D. Chen, S. Denman, S. Sridharan and C. Fookes, "Gait energy volumes and frontal gait recognition using depth images", in Proceedings of the International Joint Conference on Biometrics (IJCB), IEEE, 2011, pp. 1–6. [4] M. Hofmann, S. Bachmann and G. Rigoll, "2.5 d gait biometrics using the depth gradient histogram energy image", in Proceedings of the 5th International Conference Biometrics: Theory, Applications and Systems (BTAS), IEEE, 2012, pp. 399–403. [5] P. Arora and S. Srivastava, "Gait recognition using gait Gaussian image", in Proceedings of the 2nd International Conference Signal Processing and Integrated Networks (SPIN), IEEE, 2015, pp. 791–794. [6] Y. Iwashita, K. Uchino, and R. Kurazume, "Gait-based person identification robust to changes in appearance", Sensors, vol. 13, no. 6, pp. 7884–7901, June 2013. [7] A. Ramakić, D. Sušanj, K. Lenac and Z. Bundalo, "Depth-based real-time gait recognition", J. Circuits, Syst. Comput., vol. 29, no. 16, p. 2050266, 2020. [8] K. Lenac, D. Sušanj, A. Ramakić and D. Pinčić, "Extending appearance based gait recognition with depth data", Appl. Sci., vol. 9, no. 24, p. 5529, Dec. 2019. [9] A. Ramakić, Z. Bundalo and D. Bundalo, "A method for human gait recognition from video streams using silhouette, height and step length", J. Circuits, Syst. Comput., vol. 29, no. 7, p. 2050101, June 2020. [10] P. Chattopadhyay, A. Roy, S. Sural and J. Mukhopadhyay, "Pose depth volume extraction from rgb-d streams for frontal gait recognition", J. Vis. Commun. Image Represent., vol. 25, no. 1, pp. 53–63, Jan. 2014. [11] K. Bashir, T. Xiang and S. Gong, "Gait recognition using gait entropy image", in Proceedings of the 3rd International Conference on Imaging for Crime Detection and Prevention, 2009, pp. 1–6. [12] J. Portillo-Portillo, R. Leyva, V. Sanchez, G. Sanchez-Perez, H. Perez-Meana, J. Olivares-Mercado, K. Toscano-Medina and M. Nakano-Miyatake, "A view-invariant gait recognition algorithm based on a joint-direct linear discriminant analysis", Appl. Intell., vol. 48, no. 5, pp. 1200–1217, May 2018. [13] A. O. Lishani, L. Boubchir, E. Khalifa and A. Bouridane, "Human gait recognition based on haralick features", Signal, Image Video Process., vol. 11, no. 6, pp. 1123–1130, Sep. 2017. [14] M. Rudek, N.M. Silva, J.P. Steinmetz and A. Jahnen, "A data-mining based method for the gait pattern analysis", “Facta Univ. Mech. Eng., vol. 13, no. 3, pp. 205-215, 2015. [15] S. Zheng, J. Zhang, K. Huang, R. He and T. Tan, "Robust view transformation model for gait recognition", in Proceedings of the International Conference on Image Processing (ICIP), IEEE, 2011, pp. 2073–2076. [16] S. Yu, D. Tan and T. Tan, "A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition", in Proceedings of the 18th International Conference on Pattern Recognition (ICPR), vol. 4, IEEE, 2006, pp. 441–444. [17] "The institute of automation, Chinese academy of sciences (casia)", link: http://www.cbsr.ia.ac.cn/ english/Gait%20Databases.asp, (accessed: 25.03.2021.) [18] M. Hofmann, J. Geiger, S. Bachmann, B. Schuller and G. Rigoll, "The tum gait from audio, image and depth (gaid) database: Multimodal recognition of subjects and traits", J. Vis. Commun. Image Represent., vol. 25, no. 1, pp. 195–206, Jan. 2014.