204 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) ISSN (Print) 2313-4410, ISSN (Online) 2313-4402 Β© Global Society of Scientific Research and Researchers http://asrjetsjournal.org/ Real-time Road Obstacle Detection Using Association and Symmetry Recognition Khalid Zebbaraa*, Abdenbi Mazoulb, Mohamed EL Ansaric a,b,cLaboratory Computer Systems and Vision, Department of Computer Science, Faculty of Sciences, Ibn Zohr University, Agadir, Morocco aEmail: zebbara.khalid@gmail.com bEmail: mazoul.abdenbi@gmail.com cEmail: m.elansari@uiz.ac.ma Abstract This paper presents a fast road obstacle detection system based on association and symmetry. This approach consists to exploit the edges extracted from consecutive images acquired by a stereo sensor embedded in a moving vehicle. The algorithm contains three main components: edges detection, association detection and symmetry calculation. The edges detection is achieved by using the canny operator and point corner to extract all possible edges of different objects at the image. The association technique is used to exploit relationship between the edges of two consecutives images by combining it with the moment operator. The symmetry is used as road obstacle validation; the road obstacles like vehicle and pedestrian have a vertical symmetry. The proposed approach has been tested on different images. The provided results demonstrate the effectiveness of the proposed method. Keywords: Obstacle detection; Vehicle detection; intelligent vehicle; edges detection; Association. 1. Introduction An intelligent vehicle (IV) can achieve road obstacle detection by knowing its environment. Obstacle and Vehicle Detection play a basic role. In fact, an intelligent vehicle must be able to detect vehicles and potential obstacles on its path. Advanced driver-assistance systems intend to understand the environment of the vehicle contributing to traffic safety. ------------------------------------------------------------------------ * Corresponding author. http://asrjetsjournal.org/ American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 50, No 1, pp 204-213 205 It has been considered important that intelligent vehicles identify obstacles around a host vehicle and estimate their positions and velocities precisely. In this context, many systems have been de-signed to deal with obstacle detection in various environments. Radars [1,2], laser range finder [3,4], stereovision [5,6,7,8,9,10] and multisensory fusion are used on structured roads. Several approaches to obstacle detection based on the localization of specific patterns (features such as shape, symmetry, or edges). In [11,12] the stereo matching is used in many applications, like obstacle detection, 3D-reconstruction, autonomous vehicles and augmented reality The vision-based obstacle detection for the outdoor here we provide a brief review of the state of the art in vision-based obstacle detection. The vision-based obstacle detection for environment can be classified into monocular and multi-camera methods. In Monocular vision-based methods we find some techniques like optical flow was used for robotics obstacle detection in [13] and Appearance- based method [14] applied only appearance or color feature to discriminate the obstacles. Recently, some researches on 3-D reconstruction from single still image were presented to detect obstacle [15,16,17]. However these methods have weak points in estimating an obstacles position, velocity, and pose, and this has been considered one of the most challenging tasks in computer vision for a long time. The V-disparity and G- disparity image [24,25,26,27], was designed to detect obstacles by estimating the disparity of the ground plane automatically. In this paper, we focus on edges Association and symmetry obstacle detection. That is, detecting the road obstacles ahead of the vehicle using a stereo-camera in real time. This paper describes a new detection vehicle approach based on edges detection and association [28,29]. This approach is composed of extracting the interest edges and possible shapes features, extracting possible vehicle from association image in the successive frame. Our approach is able to detect most of the obstacles in the road scene by using the association between the two consecutive frames by combining it with the symmetry operator to detect the vehicles on this scene. This approach provides a good and robust representation of the geometric content of road scenes and it can detect and locate the road vehicles. The remainder of the paper is organized as follows: After reviewing the introduction in the section 1. The improved vehicle detection method is given in Section 2. Then some experimental results will be shown in Section 3 to demonstrate the advantages of our system. Finally, conclusions and discussions of this study will be given. 2. The vehicle detection Process This section describes the steps of the proposed method for vehicle detection on road scene. The principle of the new approach is to exploit the link between the two consecutive frames in the temp. This exploitation consists of finding the association between the contours of two frames in order to build the association image to extract all useful information for vehicles detection. The following notations will be used in the rest of the article. Ikβˆ’1, Ik denote the images of the frame fkβˆ’1 and fk acquired at time kβˆ’1 and k. Dep(𝐂𝐂𝐀𝐀𝐒𝐒 ) is the edges displacement value along both x and y abscises. 2.1. Edges detection (point corner/ canny) American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 50, No 1, pp 204-213 206 In this work, we are interested in using edge points and point corner for extracting significant features from the consecutives images. The Canny edge detector and The Shi corner detectors [18,6] are regarded as one of the best edge detectors and point corner currently in use. It provides continuous edge curves, which are essential to the proposed detection method. Consequently, we use the Canny operator for edge (points and curves) detection from the consecutives frames. Using the Canny detector in the current work allows the detection of more edge points for obstacles detection and especially a vehicle that has a well-known shape compared to other forms such as trees etc ... The reason we used the Harris operator to detect corner points is that a vehicle has a similar shape of a rectangle. Subsequently, the existence of corner points will be stronger. After calculating the point’s corners, a simple thresholding was performed to remove small items from other objects that have a different shape to the shape of a vehicle, points corners in a vehicle are much more compared to trees or features of the road. The last step is to find all the contours that pass through these corner points in order to limit the calculation of the association and the moment of the edges for the two consecutive images. 2.2. Association between edge points of consecutive images and The moment computation 2.2.1. Association between edge points As before mentioned; the main idea of the proposed approach is to exploit the relationship between consecutive frames. We propose to use the association to achieve this goal [28,29].This subsection describes the method used to find the association between edge points of consecutive frames association of the images Ikβˆ’1 and Ik . Let us consider two edge points Pkβˆ’1 and Qkβˆ’1 belonging to a curve Ci kβˆ’1 in the image Ikβˆ’1 and their corresponding ones Pk and Qk belonging to a curve Cj k in the image Ik (see Figure. 1). The associate point to point Pkβˆ’1is defined as the point belonging to the curve Ci k whith the same y-coordinate as Pkβˆ’1. Two associate points are two edge points belonging to two corresponding curves of two consecutive images of the same sequence and having the same y-coordinate. From Figure 1, we remark that point Qk constitutes the associate point of Pkβˆ’1. Figure 1: Ik-1 and Ik represent successive images at t0 and t1. The point Pk-1 in the image Ik-1 constitutes the associate point of the point Qk in the image Ik. The points Pk and Pk-1 are in red color. The points Qk and Qk-1 are in green color. Qk , Pk belong to the same contour Ck in the image Ik and their associates respectively must also be belonged to the same contour Ck-1 in the image Ik-1. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 50, No 1, pp 204-213 207 For each edge point in image Ikβˆ’1 we look for its associate one, if it exists, the image Ik. The association technique consists of finding for each edge curve Ci kβˆ’1 in the set Skβˆ’1 its corresponding edge curve Cj k in the set Sk , if it exists. Let Ass(Ci kβˆ’1) ={aen}n=1,..,Ni be the set of edge points aen, belonging to the image Ik , which represent the associates of the edge points of the edge curve Ci kβˆ’1. Ni is the number of associations found for the edge curve Ci kβˆ’1. If Mi represents the number of edge points in Ci kβˆ’1, Ni <= Mi because there are edge points in image Ikβˆ’1 for which there is no associate in image Ik . If there is no error in the association process, all the edge points belonging to the set Ass(Ci kβˆ’1) should belong to one edge curve, which is the corresponding curve to Ci kβˆ’1. Unluckily, during the process of the association we found some mistakes. Therefore, the edge point’s aem may belong to different curves in Sk. We find the match of Ci kβˆ’1 by looking for the curve Cj k, which contains the maximum number of edge points in Ass (Ci kβˆ’1). We apply the same method to all the edge curves in Skβˆ’1 to find their corresponding ones in kS . 2.2.2. The edges moment computation In the previous part we saw how to calculate the association that will allow us to find each one of the contours at time t0 and his associate at time t1, and we will reinforce this association by the use of the technique WTA which was used in the sparse match process [27]. This technique allows us to avoid erroneous calculations in the association. After the calculation of the association and to avoid any ambiguity concerning the shape of the contours at time t0 and t1; that is to say; we can find the edge curve Ci k-1 and his associate Ci k have not the same form. This is why we are going to add the imperial moment in checking the shape of the contours to achieve good results. The moment computation gives us some rudimentary characteristics of a contour that can be used to compare two contours in the consecutive frames. However, the moments resulting from that computation are not the best parameters for such comparisons in most practical cases. In particular, one would often like to use normalized moments (so that objects of the same shape but dissimilar sizes give similar values). Similarly, the simple moments of the previous section depend on the coordinate system chosen, which means that objects are not matched correctly if they are rotated [30]. Assuming that the edges of a road obstacle at time t0 remain unchangeable at time t1, so we get the following formula: οΏ½ π‘€π‘€οΏ½πΆπΆπ‘˜π‘˜βˆ’1𝑖𝑖 οΏ½ = π‘€π‘€β€²οΏ½πΆπΆπ‘˜π‘˜π‘–π‘– οΏ½ π΄π΄π΄π΄π΄π΄οΏ½πΆπΆπ‘˜π‘˜βˆ’1𝑖𝑖 οΏ½ = π΄π΄π΄π΄π΄π΄οΏ½πΆπΆπ‘˜π‘˜π‘–π‘–οΏ½ (1) π‘€π‘€οΏ½πΆπΆπ‘˜π‘˜βˆ’1𝑖𝑖 οΏ½= {mu}u=1,.,7 / values of moments for the edge curve Ci kβˆ’1 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 50, No 1, pp 204-213 208 π‘€π‘€β€²οΏ½πΆπΆπ‘˜π‘˜π‘–π‘–οΏ½= {m’u}u=1,.,7 values of moments for the edge curve Ci k . And π΄π΄π΄π΄π΄π΄οΏ½πΆπΆπ‘˜π‘˜βˆ’1𝑖𝑖 οΏ½=π΄π΄π΄π΄π΄π΄οΏ½πΆπΆπ‘˜π‘˜π‘–π‘–οΏ½ . From the previous formula (1), the correct associating of each edge between the two images Ik-1 and IK at time t0 and t1 is obtained. Assuming that the general shape of an obstacle remains unchangeable. That is to say, the edges of the Sk-1 scene and their associates in the Sk scene (their correspondents) have the same geometric shape. The edges of the same road obstacle in the two moments are similar. From this information and to know the edges of the same road obstacle we will calculate the displacement of each edge from the previous formula (1). Let Dep(πΆπΆπ‘˜π‘˜π‘–π‘– ) =(dx,dy) be the displacement value of the edges Ci k-1 in the scene Sk. The movement of the object consists of the movement of its own edges. From this assumption all the edges of the same obstacle have the same value of displacement in both directions of abscissa x and y. (see Figure 2) Figure 2: Ik-1 and Ik represent the displacement value of the edges Ci k-1 in the scene Sk. There are cases where two obstacles are even moving so the symmetry technique will help us to differentiate between the edges of the two obstacles. 2.2.3. ymmetry detection In general, symmetry is based on the shape of an obstacle. In our case road obstacles like vehicles or pedestrians have a vertical symmetry [31]. The road obstacle Detection algorithm is based on the following considerations: a vehicle and pedestrians are generally symmetric, characterized by a rectangular bounding box which satisfies specific aspect ratio constraints, this aspect ratio is fixed before a lot of experience in our method in our case we fixed it at (w:20,L:20) pixels. First an area of interest is identified on the basis of road position in the previous section. This area is searched for possible vertical symmetries; just vertical symmetries are considered. Once the width and position of the symmetrical area have been detected, a new search begins, aimed at the detection of the two bottom corners of a rectangular bounding box. American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 50, No 1, pp 204-213 209 3. Results and Analyses To evaluate the performance of the proposed obstacle detection system, tests were carried out under different images. The system including a hardware used for the experiments is a HP Intel(R) Core(TM) i5 running under Linux Ubuntu is able to process approximately 10ms. First, the images had a size of 512x512 pixels are used. Table1 shows the processing time of the road obstacle detection, the overall average processing time for one frame is 10ms. Table 1: Processing time of the road obstacle detection approach. Average processing time (ms) Association calculation 3 Moment calculation 4 Symmetry process 3 Total 10 (a) (b) (c) Figure 3: Results of the proposed algorithm: (a) Real images at the instant t0 and t1, (b) edges detection. (c) Association image. The results of the proposed obstacle detection approach are depicted in Figures 3, 4 and 5. The detection rate is high, and our approach proves to be reliable and be able to detect most road obstacles in road environments. The Figures 3, 4 and 5 shows some representative detection results. The bounding box superimposed on the original images shows the final detection results. From these results, we can see that the bounding box on the image can American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 50, No 1, pp 204-213 210 effectively describe the road obstacles. Figure 4: (left) Edges detection satisfied formula (1). (Right) bounding box detection Figure 5: Symmetry detection; (left) initial image. (Center) symmetry map, (right) vehicle detection. The following table summarizes some statistics concerning the different stages of our approach; first, we computed the possible contours found in the two frames of images at time t0 and t1. The number of contours of the first frame t0 is 197 and in the second is 180. After getting all the contours, the next step is the association between the two sets of contours in the two frames of images and of course the number of associated contours becomes lower than the initial value found by the canny operator. So our approach is able to find 170 associated contours between the two images. To reinforce the proposed approach we will add the imperial Moment factor to determine the good result. In this approach we have applied the imperial moment of the order 7 that is to say we have 7 level of comparison between the contours obviously in the combination with the association. We got 160 contours so a very high rate 94%. Table 2 justifies clearly the performance of our method. Table 2: statistics for each stage of our approach Frame k-1 Frame k Number of edges detected 197 180 Number of Association edges without the moment factor 170 Number of Association edges with the moment factor (Formula 1) 160 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 50, No 1, pp 204-213 211 The following figure shows the moment values for the edges of two images at time t0 and t1. We performed a 7- level comparison to find the correct association values and to eliminate the false calculations found during the association process (figure 6). Figure 6: The moment values for edges of two images at t0 and t1 time. 4. Conclusion In this paper, we have developed a real time method that detects road obstacles (vehicles, pedestrians); our algorithm is proposed to detect road obstacles by using consecutive images which are obtained from cameras installed at a moving vehicle. The proposed obstacle detection algorithm can be used for the development of driver assistance system and autonomous vehicle systems. Firstly, the edges detection process will be carried out to obtain all edges presented the obstacles. Secondly, the association technique used to retrieve all objects found it in the scene. Then the symmetry is used for obstacles validation. The obtained results are perfect and satisfactory. Among the limits of our approach is that the validation phase is not use it except the symmetry operator. That’s why we will to apply vehicle validation phase to differentiate between vehicles and pedestrians. we will consider using the adabor filter to obtain a perfect detection of vehicles. References [1]. Aaqib Khalid, Tariq Umer, Muhammad Khalil Afzal, Sheraz Anjum, Hafiz Muhammad Asif : Autonomous data driven surveillance and rectification system using in-vehicle sensors for intelligent transportation systems (ITS) Computer Networks, Volume 139, 5 July 2018, Pages 109-118. [2]. Dylan Horne, Daniel J. Findley, Daniel G. Coble, Thomas J. Rickabaugh, James B. Martin :Evaluation of radar vehicle detection at four quadrant gate rail crossings, Journal of Rail Transport Planning & Management, Volume 6, Issue 2, September 2016, Pages 149-162 [3]. Kirchner, A., Ameling, C.: Integrated obstacle and road traking using a laser scan-ner. In Intelligent Vehicles, USA, Oct. (2000). [4]. Parent, M., Crisostomo, M.: Collision avoidance for automated urban vehicles. In Intelligent Vehicles, Tokyo, Japan, June (2001). https://www.sciencedirect.com/science/article/pii/S2210970616300063 https://www.sciencedirect.com/science/article/pii/S2210970616300063 https://www.sciencedirect.com/science/journal/22109706 https://www.sciencedirect.com/science/journal/22109706 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 50, No 1, pp 204-213 212 [5]. Tongtong Li, Changying Liu, Yang Liu, Tianhao Wang, Dapeng Yang : Binocular stereo vision calibration based on alternate adjustment algorithm Optik, Volume 173, November 2018, Pages 13-20. [6]. M. Dehnavi, M. Eshghi : Cost and power efficient FPGA based stereo vision system using directional graph transform, Journal of Visual Communication and Image Representation, Volume 56, October 2018, Pages 106-115 [7]. Stefan Gehrig, Nicolai Schneider, Reto Stalder, Uwe Franke : Stereo vision during adverse weather β€” Using priors to increase robustness in real-time stereo vision, Image and Vision Computing, Volume 68, December 2017, Pages 28-39 [8]. Xuanchen Zhang, Yuntao Song, Yang Yang, Hongtao Pan :Stereo vision based autonomous robot calibration : Robotics and Autonomous Systems, Volume 93, July 2017, Pages 43-51 [9]. J. C. RodrΓ­guez-QuiΓ±onez, O. Sergiyenko, W. Flores-Fuentes, M. Rivas-lopez, P. Mercorelli : Improve a 3D distance measurement accuracy in stereo vision systems using optimization methods’ approach, Opto-Electronics Review, Volume 25, Issue 1, May 2017, Pages 24-32. [10]. Hattori, H., Maki, A.: Stereo without depth search and metric calibration, Re-search & Development center, TOSHIBA Corporation.Kawasaki212-8582, Japan. IEEE (2000). [11]. Borja Bovcon, Rok Mandeljc, Janez PerΕ‘, Matej ristan :Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation, Robotics and Autonomous Systems, Volume 104, June 2018, Pages 1-13. [12]. Xuerui Dai : HybridNet: A fast vehicle detection system for autonomous driving, Signal Processing: Image Communication, Volume 70, February 2018, Pages 79-88 [13]. Coombs, D., Herman, M., Hong, T. H., Nashman, M.: Real-time obstacle avoidance using central ow divergence and peripheral ow. IEEE Transactions on Robotics and Automation, 14(1): 4959 , (1998). [14]. Ulrich, I., Nourbakhsh, I.: Appearance-based obstacle detection with monocular color vision. In: Proceedings of the 17th National Conference on Arti cial Intel-ligence and 12th Conference on Innovative Applications of Arti cial Intelligence. Austin, USA: 866871, AAAI Press, (2000). [15]. Saxena, A., Chung S. H., Ng, A. Y.: 3-D depth reconstruction from a single still image. International Journal of Computer Vision, 76(1): 5369 , (2008). [16]. Klarquist, W. N., Geisler, W. S.: Maximum likelihood depth from defocus for active vision. In: Proceedings of the InternationalConferen ce on Intelligent Robots and Systems. Washington D. C., USA: 3743797 , IEEE, (1995). [17]. Rajagopalan, A. N., Chaudhuri, S., Mudenagudi, U.: Depth estimation and image restoration using defocused stereo pairs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(11): 15211525, (2004). [18]. Bellutta, P., Manduchi, R., Matthies, L., Owens, K., Rankin, A.: Terrain percep-tion for DEMO III. In: Proceedings of IEEE Conference on Intelligent Vehicles Symposium. Dearborn,USA: 38, IEEE, (2000). [19]. Rankin, A., Huertas, A., Matthies, L.: Evaluation of stereo vision obstacle detection algorithms for o - road autonomous navigation. AUVSI Unmanned Systems North America. Pasadena, USA: Jet Propulsion Laboratory, (2005). [20]. Thrun, S., Montemerlo, M., Dahlkamp, H., Stavens, D., Aron, A., Diebel, J.: Stan-ley, the robot that https://www.sciencedirect.com/science/journal/00304026 https://www.sciencedirect.com/science/article/pii/S1047320318302116 https://www.sciencedirect.com/science/article/pii/S1047320318302116 https://www.sciencedirect.com/science/journal/10473203 https://www.sciencedirect.com/science/article/pii/S0262885617301166 https://www.sciencedirect.com/science/article/pii/S0262885617301166 https://www.sciencedirect.com/science/journal/02628856 https://www.sciencedirect.com/science/article/pii/S0921889016307308 https://www.sciencedirect.com/science/article/pii/S0921889016307308 https://www.sciencedirect.com/science/journal/09218890 https://www.sciencedirect.com/science/article/pii/S1230340217300100 https://www.sciencedirect.com/science/article/pii/S1230340217300100 https://www.sciencedirect.com/science/journal/12303402 https://www.sciencedirect.com/science/journal/09218890 American Scientific Research Journal for Engineering, Technology, and Sciences (ASRJETS) (2018) Volume 50, No 1, pp 204-213 213 won the DARPA grand challenge. Journal of Robotics Systems, 23(9):661692, (2006). [21]. Konolige, K., Agrawal, M., Bolles, R. C., Cowan, C., Fischler M., Gerkey, B.: Outdoor mapping and navigation using stereo vision. In: Proceedings of the 10th International Symposium on Experimental Robotics. Rio de Janeiro, Brazil: 179190, Springer, (2006). [22]. Manduchi, R., Castano, A., Talukder, A., Matthies, L.: Obstacle detection and ter-rain classi cation for autonomous o -road navigation. Autonomous Robots, 18(1): 81102, (2005). [23]. Matthies, L., Maimone, M., Johnson, A., Cheng, Y., Willson R., Villalpando, C.: Computer vision on Mars. International Journal of Computer Vision, 2007, 75(1): 6792 [24]. Broggi, A., Cara , C., Fedriga, R. I., Grisleri, P.: Obstacle detection with stereo vision for o -road vehicle navigation.In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego, USA: 6572, IEEE, (2005). [25]. Yifei Wang, Yuan Gao, Alin Achim, Naim Dahnoun: Robust obstacle detection based on a novel disparity calculation method and G-disparity Computer Vision and Image Understanding, Volume 123, June 2014, Pages 23-40 [26]. Cara C., Cattani, S., Grisleri, P.: O -road path and obstacle detection using de-cision networks and stereo vision. IEEE Transactions on Intelligent Transportation Systems, 8(4): 607618, (2007). [27]. Soquet, N., Aubert, D., Hautiere, N.: Road Segmentation Supervised by an Ex-tended V-Disparity Algorithm for Autonomous Navigation. In: Intelligent Vehicles Symposium, IEEE (2007). [28]. M. El-Ansari, S. Mousset, and A. Bensrhair, β€œA new stereo matching approach for real-time road obstacle detection for situations with deteriorated visibility,” in Proc. IEEE Intelligent Vehicle Symposium, Eindhoven University of Technology, Eindhoven, The Netherlands, June 4-6 2008. [29]. Ilyas El Jaafari Mohamed El Ansari Lahcen Koutti Abdenbi Mazoul Ayoub El lahyani β€œ Fast spatio- temporal stereo matching for advanced driver assistance systems”, Neurocomputing Volume 194, 19 June 2016, Pages 24-33. [30]. Hu M.K. "Visual pattern recognition by moment invariants." IRE Transactions on Information Theory.Vol.8(2),pp.179-187,1962 [31]. GwenaΓ«lle Toulminet, Massimo Bertozzi, StΓ©phane Mousset, Abdelaziz Bensrhair, and Alberto Broggi, Senior Member, IEEE β€œVehicle Detection by Means of Stereo Vision-Based Obstacles Features Extraction and Monocular Pattern Analysis” https://www.sciencedirect.com/science/journal/10773142 https://www.sciencedirect.com/science/journal/10773142 https://www.sciencedirect.com/science/journal/09252312 https://www.sciencedirect.com/science/journal/09252312/194/supp/C Real-time Road Obstacle Detection Using Association and Symmetry Recognition