ARID ZONE JOURNAL OF ENGINEERING, TECHNOLOGY & ENVIRONMENT AZOJETE March 2020. Vol. 16(1):48-57 Published by the Faculty of Engineering, University of Maiduguri, Maiduguri, Nigeria. Print ISSN: 1596-2490, Electronic ISSN: 2545-5818 www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 48 ORIGINAL RESEARCH ARTICLE REAL-TIME DETECTION OF ABANDONED OBJECT USING CENTROID DIFFERENCE METHOD Y. A. Samaila1*, H. Rabiu2 and I. Mustapha1 (1Deparment of Electrical and Electronics Engineering, University of Maiduguri, Maiduguri, Nigeria 2Department of Electrical Engineering, Bayero University, Kano, Nigeria) * Corresponding author’s email address: yausamaila002@gmail.com 1.0 Introduction An abandoned object is a stationary object that has not been in a scene before. To detect the abandoned objects, find the static regions in the scene and determine whether they correspond to abandoned or not. Recently, several Bomb explosions occurred in public areas. For instance, on 20th September 2015, at the Ajilari area of Maiduguri, Nigeria, an abandoned polythene Bag containing an explosive kept at a Mosque entrance exploded, where several people were killed and many others sustained injury. The preventive action is imperative and wide scale deployment of surveillance systems are in demand. In spite of the sophisticated surveillance systems deployed in many places today, the problem still persists. Such limitations are connected with human involvement in the systems. Therefore, it is challenging for man to continuously monitor a scene round the clock, thus, the manual surveillance fails. Semi- automatic surveillance using close circuit Television (CCTV) also seems ineffective due to the ARTICLE INFORMATION ABSTRACT An abandoned object is one that remains stationary for an extended period. Such object might contain explosives and if left on purpose could cause death and injuries to people especially in crowded places. Abandoned objects need to be detected on time to prevent what might endanger people’s lives and health. Various methods have been developed to detect abandoned objects. The most reliable one is the vision-based method which automatically detects the abandoned object using image processing. The efficiency of the method was tested and evaluated on the customized datasets as well as the i-Lids advanced video surveillance system database. The Self -organizing Background Subtraction (SOBS) method overrides other methods in terms of its detection accuracy and simplicity of implementation, but fails for dynamic background scenarios. This work presents a real time vision-based object detection method using the centroid difference to improve on the accuracy of the detection and to tackle challenges of dynamic background of the SOBS method. Matlab Image processing toolbox was used to achieve this goal. The strategy is basically decomposed into two; foreground detection and stationary foreground object (SFO) detection. Gaussian Mixture Model is used for detecting the presence of newly introduced object into a scene (foreground detection), while the blob tracking approach based on frame counting is used to determine whether the detected foreground object is static/ abandoned or not. The results show that the detection accuracy of 83% was obtained which outperform the SOBS method with 67% accuracy. Future research should focus on tracking the person that abandoned the object for onward prosecution. © 2020 Faculty of Engineering, University of Maiduguri, Nigeria. All rights reserved. Submitted 21 March, 2019 Revised 27 September, 2019 Accepted 03 October, 2019 Keywords: Stationary Foreground Object Abandoned Object Gaussian Mixture Model Self -Organizing Background Subtraction Centroid Difference. http://www.azojete.com.ng Arid Zone Journal of Engineering, Technology and Environment, March, 2020; Vol. 16(1) 48-57. ISSN 1596-2490; e-ISSN 2545-5818; www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 49 fact that man cannot effectively monitor many scenes at a time without possible distraction. To ease this problem and improve scene monitoring systems, computer-based object detection and monitoring system is proposed. A computer-based surveillance system detects the left/abandoned object and activate alarm to alert security personnel act on it. Several methods have been devised to detect foreground object. These include frame differencing, background subtraction, optical flow, single Gaussian, Gaussian Mixture Model (Samaila et al., 2019). Background subtraction identifies the presence of moving object by considering the difference between a fixed reference frame and a current frame (Risha and Chempak, 2016). The frame differencing method identifies the presence of moving object by considering the difference between two consecutive frames. Frame differencing is subdivided into three parts. Initial step is the selection of perfect reference or background. The second step is the arithmetic subtraction operation and the third step is the selection of a suitable threshold. Reference image can be selected as a frame which is temporarily adjacent to an image from a dynamic sequence. This method lacks in obtaining the complete contour of the object (Risha and Chempak, 2016). This could arise due to suitable threshold selection that will be applied throughout the video frame, as each frame has its unique feature. A three (3) frame differencing is used to lessen this shortcoming (Shaikh et al., 2014). Interestingly, a hybrid approach co- opting background subtraction and frame differencing detects moving object a lot effectively and precisely (Alex and Wahi, 2014). For scenes that are having dynamic background (Tree branches, Bushes, Water Surface or Flags), a generalization based on a mixture of Gaussians can be used to model such variations (changes cannot be modeled using One Gaussian distribution per pixel). Stauffer and Grimson (1999) proposed a Gaussian Mixture Model (GMM) method to deal with scenarios consisting dynamic backgrounds, illumination challenges, camouflage among others. Tavakkoli, (2009) came up with a method that allows the background model to be a mixture of several Gaussians (typically between 3 and 5). Each pixel is labeled as foreground or background based on its probability. Furthermore, for the stationary foreground object (SFO) detection; tracking of foreground and dual foreground comparison are some of the renowned methods. In the tracking of foreground method, blobs statistics such as object size/area, centroid position, were used in some studies by Miguel and Martinez (2008), Bayona et al. (2009), Utikari and Uke (2014) and Singh et al. (2009) to determine if an object is static or not. Dual foreground comparison (DFC) strategy proposed by Porikli et al. (2008) tried to identify the SFOs by comparing two binary foreground masks at pixel level. These masks were obtained from two background models constructed with different learning rates. The models were constructed using multiple Gaussians. The main objective of the research is to developed a database and as well use it in developing an algorithm for detecting abandoned object. 2. Methodology The abandoned object detection method used in this work comprises of three stages. The block diagram representation of the abandoned object detection method is as shown in Figure 1. Figure 1: Block Diagram Representation of the Abandoned Object Detection Method file:///C:/Users/user/Downloads/azojete143/www.azojete.com.ng Samaila et al: Real-time detection of abandoned object using centroid difference method. AZOJETE,16(1):48-57. ISSN 1596-2490; e-ISSN 2545-5818, www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 50 The custom database was developed in addition to the publicly available database to implement the technique. red, green blue (RGB) partitioned video frames were converted to gray scale for easier and faster processing. Foreground detection: Gaussian mixture method was used to detect foreground objects in the gray scale video frames. Stationary Foreground Object (SFO) detection: Centroid difference tracking method was used to track the stationary foreground object(s) across successive frames. Figure 2 gives the flow chart of the GMM algorithm, from the design and the development of the database stage to the stationary object detection stage. Figure 2: Flow Chart of the GMM Algorithm 2.1. Design and Development of Database Two different databases were considered in this work. The Advanced Video Surveillance System (AVSS) which is a publicly available database, contains objects (of interest) with confined shapes and sizes. The custom database takes care of the shortcomings of the AVSS database as it considers objects of different shapes and sizes. The properties of the abandoned object detection database is shown in Table 1. http://www.azojete.com.ng Arid Zone Journal of Engineering, Technology and Environment, March, 2020; Vol. 16(1) 48-57. ISSN 1596-2490; e-ISSN 2545-5818; www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 51 Table 1: Properties of the Abandoned Object Detection Database. Sequence(S) No. of Frames Object of interest Size (Pixels) Customized Sequence 1(S1) 297 Polythene Bag 1390 Customized Sequence 2(S2) 660 Big Bag 1968 Customized Sequence 3(S3) 318 can 691 Customized Sequence 4(S4) 2071 Big Bag 2021 Customized Sequence 5(S5) 1676 Small Bag 1553 The Video (database) shown in Table 1 was partitioned into frames and converted from RGB to grayscale image in order to speed up the processing. Background model was developed using Gaussian mixture model in order to detect the foreground object. 2.2. The Gaussian Mixture Model Foreground Detection For Video scenes with dynamic background (tree branches, bushes, water surfaces or flags), a generalization based on a mixture of Gaussians can be used to model such variations (changes cannot be modeled using One Gaussian distribution per pixel). The use of a Gaussian mixture model (GMM) was first introduced by Stauffer and Grimson (2000) to deal with dynamic background scenarios. This method allows the background model to be a mixture of several Gaussians (typically between 3 and 5). Each pixel is labeled as foreground or background based on its probability. A pixel-based method was presented which models each pixel (regarded as background) into a mixture of Gaussians. The number of Gaussians K is typically set from 3 to 5 (Stauffer and Grimson, 2000). In addition, each Gaussian has its own weight to represent the portion of the data accounted for from corresponding distribution. The probability that a pixel regards a value x at a certain time t, Xt is given as follows; P Xt = j=1 k ω j,t ∗ ղ Xt ,m j,t , j,t�� (1) where K is the number of Gaussian distributions, ω j,t is the weight estimation of the jth Gaussian in the mixture at time t, mj,t and Σ j,t are the mean value and covariance matrix respectively, of the jth Gaussian in the mixture at time t, and η is a Gaussian pdf (probability density function) defined in (2); ղ Xt. ,m,Ʃ = 1 2π k 2 Ʃ 1 2 exp − 1 2 Xt −mt T Ʃ−1 Xt −mt (2) Table 2 shows the modified GMM Parameter values used which was originally adopted in another work (Power and Schoones, 2002). Table 2: Parameter Values for the modified GMM Symbol Value Related Parameter K 5 25 frames/seconds A 0.005 X ε [0,255] T 0.7 ωinit 0.05 Num Training Frames 50 @ 25fps X [0-255] file:///C:/Users/user/Downloads/azojete143/www.azojete.com.ng Samaila et al: Real-time detection of abandoned object using centroid difference method. AZOJETE,16(1):48-57. ISSN 1596-2490; e-ISSN 2545-5818, www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 52 where: K= Number of Gaussians T= Minimum Background Ratio X=pixel value σinit=Initial Variance α= Learning Rate ῳinit= initial weight NumTrainingFrames= Number of Training Frames The salient point of the GMM stage is given here. 1. The method terminates upon reaching convergence. The parameters are used to fit on the data point (each pixel is denoted as a data point). 2. Upon fitting the GMM to the data, the component or surface (K) with the highest probability claims the pixel (that particular incoming pixel). 3. The process is repeated for all incoming pixel of the current frame as compared to the background model. The detected foreground object is used as an input to the next stage to determine whether it is stationary/abandoned or not. 2.3 Stationary Foreground Object (SFO) Detection Using Centroid Difference Tracking Method The centroid is one of the easiest region properties used for determining whether a foreground object is stationary or otherwise. It comprises of two (2) axis-x and y. Centroid r�=1A (r,c)εR r� , c�=1A (r,c)εR c� , (3) Where r= row; c= column; R=region; A=area. Note: For a foreground blob to be considered static, it has to satisfy two conditions; The Centroid difference must be less than the set threshold [5 5]. The said difference should be maintained for at least 25 consecutive frames Blob counting (tracking) across several frames is employed to track the stationary objects in the sequence. The tracking process is explained in section 2.4. 2.4. Tracking of Foreground Using the Centroid Difference Algorithm The basic steps are: The variables were declared and initialized Counter(c)= 0, threshold (Th)= [5, 5], No. of consecutive frames = 25 The centroid and area of blobs in each frame were obtained and stored in an array. The maximum area and its index as well as its corresponding centroid (maximum centroid) were determined for each frame. The maximum centroid entries were labeled as k1(x1, y1) for the 1st frame, k2(x2, y2) for the second, up to kn (xn, yn) for the last frame. If k1-k2 2.0 for human. Alarm was raised for static non-human objects http://www.azojete.com.ng Arid Zone Journal of Engineering, Technology and Environment, March, 2020; Vol. 16(1) 48-57. ISSN 1596-2490; e-ISSN 2545-5818; www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 53 3. Results and Discussion 3.1 Experimental Results for Foreground Detection The grayscale images in Table 1 were used for detecting the foreground objects. Two detection strategies namely the GMM (proposed) and the Self-organizing Background Subtraction (SOBS) methods were considered. The SOBS method has background which automatically organizes itself (Yadav and Jahagirdar, 2015). The GMM uses the first 50 frames to develop a background model. The grayscale of sequence 1(S1) for frame #50, frame #120 and frame #297 of sequence 1(S1) is shown in Figure 3(a),3(b) and 3(c) respectively. Since no new object was introduced, a zero foreground object was observed in frame #50. Two foreground objects were detected in frame #120. Figure 4(a), 4(b) and 4(c) shows the SOBS foregrounds of frame #50, frame #120 and frame #297 of sequence 1(S1). The objects were large in Figure 4(b) because they are closer to the camera. As the detected foreground objects go farther away from the camera position, the objects diminish in size . Morphological opening operation with a square structural element of radius ‘10’ was carried out on the detected foreground objects to refine them so as to have a better and improved output devoid of noise. Similarly, the GMM foregrounds of S1 for frame #50,frame #120 and frame #297 is shown in Figure 5(a),5(b) and 5(c) respectively. Figure 3: Gray Frames of S1: (a) Frame #50 (b) Frame #120 (c) Frame #297 Figure 4: SOBS Foregrounds of S1: (a) Frame #50 (b)Frame #120 (c) Frame #297 Figure 5: GMM Foregrounds of S1(a) FG Frame #50 (b)Frame #120 (c)Frame # 297 A comparison between the SOBS method and the proposed GMM method in terms of detected foreground, show that the GMM as depicted in Figure 5(c) gives a good detection of the foreground at the point of abandonment of the object of interest (polythene bag in this case) as compared to the SOBS as portrayed in Figure 4(c). In Figure 5(c) the polythene bag is larger 1281 pixels’ as compared to Figure 4(c) which is 681 pixels. The reason is that the GMM method overcomes variation in illumination and background challenges better than the SOBS method. The larger object of interest in the detected foreground at the point of abandonment informed our decision for choosing the GMM in the detection of foreground. Hence the larger the object the better the detection. file:///C:/Users/user/Downloads/azojete143/www.azojete.com.ng Samaila et al: Real-time detection of abandoned object using centroid difference method. AZOJETE,16(1):48-57. ISSN 1596-2490; e-ISSN 2545-5818, www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 54 The results of the foreground detection stage were used to determine the SFO (abandoned object). 3.2 Experimental Results for Stationary Foreground Detection (SFO) Executing the centroid difference tracking algorithm on the detected foreground yielded some results. Figures 6(a) shows the plot of maximum centroid y against maximum centroid x. There is an indication of abandoned object at points (409,350 and 409,351) of S1 denoted by an ellipse in Figure 6(b) shows the abandoned objects detected at frame # 275 of S1. (b) Figure 6:(a) Plot of Maximum Centroid y against Maximum Centroid x (b) Abandoned objects detected at Frame # 275 of S1 Figure 7(a) shows the plot of maximum centroid y against maximum centroid x, indicating abandoned object at points (126,151and 127,155) of S4denoted by an ellipse. Figure 7(b) Abandoned Object Detected at frame #800 of S4. (b) Figure 7:(a) Plot of Maximum Centroid y against Maximum Centroid x, (b) Abandoned Object Detected at Frame #800 of S4. Figure 7(b) entails human having an aspect ratio of 2.3 denoted with green bounding box was distinguished from other objects (aspect ratio <2). The essence is to reduce or eliminate false detection, as static humans should not be declared as abandoned objects even if they are stationary. The larger the size of the blob (detected foreground object), the better the detection as shown in Figures 4 and 5 for the SOBS and GMM respectively. Objects as small as a Tea Cup was detected by the proposed method which is not realizable in the SOBS method. The proposed (b) http://www.azojete.com.ng Arid Zone Journal of Engineering, Technology and Environment, March, 2020; Vol. 16(1) 48-57. ISSN 1596-2490; e-ISSN 2545-5818; www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 55 method handles dynamic background (swaying trees and flowers) as it considered them as part of the background, thus minimizing wrong detection present in the reference method (SOBS). The GMM method also overcomes the challenge of camouflage as the background and foreground objects in the custom database have similar appearance, which literally makes detection difficult in the other method. The proposed algorithm is equally unique from the SOBS and other similar methods as it distinguishes human from other objects using the aspect ratio, as only non-human abandoned objects are declared abandoned. The system fails to detect multiple abandoned objects simultaneously, as it only detects one abandoned object at a time. It also fails for scenarios/sequences with low contrast, especially for sequences recorded at night. 3.3 Performance Metrics Algorithm performance was assessed using the F1 weighted harmonic mean of ‘recall’ ‘precision ‘and ‘accuracy’. Recall is the fraction of abandoned objects which were in fact abandoned. Precision is the fraction of positive classifications which are correct. Accuracy is the fraction of correctly classified objects over the number of objects classified. When presented with a sequence, each algorithm or system yields a number of entities (Table 3) such as: (a) True detection/True Positive (system alarms in response to a genuine alarm event) (b) False detection/False Positive (system alarms without the presence of a genuine alarm event) (c) Missed detection/False Negative (genuine alarm events not resulting in a system alarm) (d) Human detection (genuine non-alarm events that arises as a result of static human) The system performance of the two algorithms (SOBS and GMM) were compared as depicted in Table 3. Table 3: Performance Comparison of The SOBS and GMM Methods Sequence (S) True Detection False Detection Missed Detection Human Detection SOBS GMM SOBS GMM SOBS GMM SOBS GMM S1 1 1 0 0 0 0 0 0 S2 1 1 0 0 0 0 0 0 S3 0 1 0 0 1 0 0 0 S4 1 1 1 0 0 0 0 1 S5 1 1 1 0 1 1 0 1 Total 4 5 2 0 2 1 0 2 Key: a= True detection b=False detection c=missed detection d=Human detection Recall, r = a a+c Precision, p = a a+b (4) Recall, r SOBS = 4 6 = 0.67 Precision, pSOBS = 4 6 = 0.67 Recall, r GMM = 5 6 = 0.83 Precision, pGMM = 5 5 = 1 F1 = ∝+1 rp r+∝p . (5) Where α is the ‘recall bias’; a weighting of recall relative to precision declared in each i-LIDS scenario definition. The weighting to be used for the i-LIDS AVSS is 35. The algorithm demonstrating the highest F1 score for each sequence will be deemed the better one. file:///C:/Users/user/Downloads/azojete143/www.azojete.com.ng Samaila et al: Real-time detection of abandoned object using centroid difference method. AZOJETE,16(1):48-57. ISSN 1596-2490; e-ISSN 2545-5818, www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 56 Thus, the F1 score for the two algorithms will be F1SOBS = 35+ 1 0.67 ∗ 0.67 0.67 + 35 ∗ 0.67 ≈ 0.7 F1GMM = 35+ 1 0.83 ∗ 1 0.83 + 35 ∗ 1 ≈ 0.8 Accuracy = True detection True detection + Missed detection (6) Accuracy SOBS = 4 6 = 0.667 = 67% Accuracy GMM = 5 6 = 0.833 = 83% Table 3 showed that the FI score for the proposed method was the better option, since it has higher score (0.8) as compared to the SOBS algorithm (0.7). The higher the FI score, the better the detection. The accuracy of abandoned object detection on GMM approach is 83%, whereas the SOBS approach has 67% accuracy as shown from results of equation 6. 4. Conclusion In this work, a vision-based abandoned object detection system that runs in real-time is presented. The system uses a custom database that was developed to take care of the shortcomings of the publicly available database which uses objects of confined shape and size. The proposed system (GMM) handles background challenges better than the SOBS method and was able to detect object as small as a Teacup. Classification of abandoned objects into human (not declared abandoned) and non-human (declared abandoned) is achieved which was not presented in other existing methods. The experimental results show that the GMM outperforms the state-of-the-art SOBS in terms of F1 score and detection accuracy. Further research should focused on detecting multiple objects, tracking and identifying humans that carried and dropped the abandoned objects behind in the scene. References Alex, DS. and Wahi, A. 2014. Background subtraction frame difference algorithm for moving object detection and extraction. Journal of Theoretical and Applied Information Technology, 60(3): 623-628. Bayona, A., Miguel, JCS. and Martinez, J. 2009. Comparative evaluation of stationary foreground object detection algorithms based on background subtraction techniques- Video Processing and Understanding Lab. Escuela Politécnica Superior, Universidad Autónoma de Madrid, SPAIN. Advanced Video and Signal Based Surveillance, 25-30. Miguel, JCS. and Martinez, JM. 2008. Robust unattended and stolen object detection by fusing simple algorithms. Advanced Video and Signal Based Surveillance,. AVSS’08. IEEE Fifth International Conference on, IEEE, 18–25. Porikli, F., Ivanov, Y. and Haga, T. 2008. Robust abandoned object detection using dual foregrounds. EURASIP Journal on Advances in Signal Processing, 30: 1-11. Power, PW. and Schoones, JA. 2002. Understanding Background Mixture Models for Foreground Segmentation. Proceedings on Image and Vision Computing New Zealand University of Auckland, Auckland, New Zealand 26–28th November, 267-271. Risha, K. and Chempak, KA. 2016. A Novel method of detecting moving object in video. International Conference on Emerging Trends in Engineering, Science and Technology (ICETEST - 2015). Procedia Technology, 24:1055 – 1060. http://www.azojete.com.ng Arid Zone Journal of Engineering, Technology and Environment, March, 2020; Vol. 16(1) 48-57. ISSN 1596-2490; e-ISSN 2545-5818; www.azojete.com.ng Corresponding author’s e-mail address: yausamaila002@gmail.com 57 Shaikh, SH., Saeed, K. and Chaki, N. 2014. Moving object detection using background Subtraction. Springer Briefs in Computer Science, 2-5. Stauffer, C. and Grimson, WEL. 1999. Adaptive background mixture models for real-time tracking’, in: Computer Vision and Pattern Recognition, 1999. IEEE Computer Society Conference, IEEE, 2:246-252. Stauffer, C. and Grimson, WEL. 2000. Learning Patterns of Activity Using Real-Time Tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8): 747-757. Tavakkoli, A. 2009. Novelty Detection: An Approach to Foreground Detection in Videos, University of Nevada, Reno USA, pp. 123-150. Utikar, AA. and Uke NJ. 2014. Abandoned Object Detection in Educational Institutes using Video Surveillance, International Journal of Advanced Research in Computer and Communication Engineering, 3(10): 8219-8222. Singh, R., Vishwakarma, S., Agrawal, A. and Tiwari, M. 2010. Unusual activity detection for video surveillance, Proceedings of the First International Conference on Intelligent Interactive Technologies and Multimedia, ACM, 297–305. Samaila, YA., Lawan, MB., Tijjani, I., Sarki, MA., Gujja, MM. and Tijjani, I. 2019. A Review on Foreground and Stationary Foreground Object Detection Techniques. Journal of Multidisciplinary Engineering Science and Technology, 6 (4): 9768-9775. Yadav, P. and Jahagirdar, A. 2015. A Static Object Detection in Image Sequences by Self Organizing Background Subtraction. International Research Journal of Engineering and Technology, 2 (7): 72-76. file:///C:/Users/user/Downloads/azojete143/www.azojete.com.ng