FACTA UNIVERSITATIS Series: Mechanical Engineering Vol. 17, N o 3, 2019, pp. 357 - 364 https://doi.org/10.22190/FUME190507041B © 2019 by University of Niš, Serbia | Creative Commons License: CC BY-NC-ND Original scientific paper INTELLIGENT MACHINE VISION BASED RAILWAY INFRASTRUCTURE INSPECTION AND MONITORING USING UAV Milan Banić 1 , Aleksandar Miltenović 1 , Milan Pavlović 2 , Ivan Ćirić 1 1 Faculty of Mechanical Engineering, University of Niš, Serbia 2 College of Applied Technical Sciences, Niš, Serbia Abstract. Traditionally, railway inspection and monitoring are considered a crucial aspect of the system and are done by human inspectors. Rapid progress of the machine vision-based systems enables automated and autonomous rail track detection and railway infrastructure monitoring and inspection with flexibility and ease of use. In recent years, several prototypes of vision based inspection system have been proposed, where most have various vision sensors mounted on locomotives or wagons. This paper explores the usage of the UAVs (drones) in railways and computer vision based monitoring of railway infrastructure. Employing drones for such monitoring systems enables more robust and reliable visual inspection while providing a cost effective and accurate means for monitoring of the tracks. By means of a camera placed on a drone the images of the rail tracks and the railway infrastructure are taken. On these images, the edge and feature extraction methods are applied to determine the rails. The preliminary obtained results are promising. Key Words: Computer Vision, UAV, Drone Imagery, Edge Detection, Railway Infrastructure, Intelligent Systems 1. INTRODUCTION With the growing industrial development and population, the role of railways in transportation is going to be crucial in upcoming years. One of the main goals of modern railway transport is to increase its quality, as well its effectiveness and capacity while maintaining a very high level of safety. On the other hand, in recent years, with advance of technology solutions in the field of UAV (Unmanned aerial vehicle), the new possibilities of using UAVs for civil purposes are opened. UAVs are commonly used in other industries such as oil and Received May 07, 2019 / Accepted August 15, 2019 Corresponding author: Ivan Ćirić Faculty of Mechanical Engineering, University of Niš, Aleksandra Medvedeva 14, 18000 Niš, Serbia E-mail: ivan.ciric@masfak.ni.ac.rs 358 M. BANIĆ, A. MILTENOVIĆ, M. PAVLOVIĆ, I. ĆIRIĆ utilities to inspect their structures, such as pylons and oil rigs. The imagery / video collected using an UAV is up to date and provides oblique information about the railway infrastructure unlike satellite imagery. SESAR European Outlook Study (2016) [1] defines that the infrastructures such as railway may be monitored and kept secure by using drones. Prediction is that railway inspection will be carried out with Long range surveying (primarily BVLOS). BVLOS are fly drones that can be used beyond the visual line of sight and represent future next step for drone industry. Europe market potential is that by 2035 up to 400 000 drones will be in use with the majority flying beyond visual line of sight; out of this number around 180 000 could be used for mapping and surveying where one of the key markets is railway. This will have a great impact in a great number of industry sectors and in railway also. Drones could benefit the mobility sector through railway inspection in which approximately 200 000 kilometers on a bi-monthly basis could be monitored. Kim [2] developed the Structure inspection system to detect and calculate cracks in structure using UAV and digital image processing technique. He acquired the digital image in order to evaluate UAV applicability and performance and field application of the crack detecting program after targeting the bridges. Comparing with the measured values, it is verified that the accuracy above a certain level is secured. Smith [3] developed a software package capable to process image that can be used to determine the location of the roads and railroads on the image as well as to detect intersections and generate specific trajectories along them. The road detection algorithm presented was proven to have a run speed of 1.25 Hz onboard a single board computer, a high classification accuracy of 96.6%, and that it produces very accurate trajectories along roads. Mathe [4] focused on lightweight UAVs to detect events in railways such as missing indicators or cabling. He developed visual servicing technique and performed a comparison of several object detection approaches. Gemert [5] investigates the combination of small UAV with automatic object recognition techniques. They record animal conservation in case of animal detection and animal counting. Karakose [6] proposed a computer-based visual rail condition monitoring in which camera is placed on top of the train and neighbor rail images are taken. On these images, the edge and feature extraction methods are applied to determine the rails. The results obtained are given at the end of the study. Experimental results show that the proposed method gives effective results. Flammini [7] studied preliminarily evaluate drone capabilities in a railway monitoring framework including structural faults and security threat detection as well as investigation of the consequences of natural hazards and intentional attacks. He proved that drone-based sensors can be advantageously integrated in existing security surveillance systems by appropriate sensor integration platforms, middleware and frameworks. Mittal [8] proved that deep learning models can be used in detect of track defects like sinking, loose ballasts and railway assets like switches and signals. Sigha [9] investigated the possibilities of computer vision-based monitoring with UAV imagery. The UAV camera provides high quality images that contain large information for monitoring and analysis. Inspection by drone does not require dedicated track for inspection hence it does not affect smooth running of trains but the aerial images with the camera oriented downward do not give convergence view of track. Intelligent Machine Vision Based Railway Infrastructure Inspection and Monitoring Using UAV 359 Jianfang [10] proposed multiple hypothesis tracking (MHT) algorithm to track an object (target) in a series of image sequences. To obtain target tracking from UAV aerial image sequences, three steps must be performed: break each track set into tracklet (track data subset) at a specific time, estimate association cost of each track set and merge trajectory fragments into a longer iteration. Increased use of UAVs can provide railway engineers with higher levels of quality information to allow them to make the best possible decisions about the future of our structures assets. As well as being cost effective, this innovation can reduce the need for possessions, track access and roped access, reducing safety risk. As well as delivering a more comprehensive 360 degree view of the structures, in daylight and showing all the defects clearly, the UAV images can be stitched together with photogrammetry to create high quality 2D elevations, 3D models and also cloud point surveys (avoiding the need for a separate dimensional survey). Aerial inspections cannot fully replace an engineer with a hammer and some degree of tactile inspection is still needed. However, the drone imagery powered by intelligent computer vision algorithms can enable detection of the areas of concern and their targeting. Having in mind drone imagery, a novel software algorithm used for railway infrastructure inspection and monitoring according to the railway demands and safety procedures needs to be developed. A novel fail safe and reliable system for rail track detection and inspection on railway mainlines as well as detection and monitoring of some specific railway infrastructure elements should be integrated into railway information system. 2. INTELLIGENT COMPUTER VISION METHODOLOGY An image can be defined by a set of regions that are connected and non-overlapping, so that the pixels in each partitioned region possess an identical set of properties or attributes [11- 18]. These sets of properties of the image may include gray levels, contrast, spectral values, textural properties, etc. which can be result of variations in reflectance, illumination, color, shade, texture, orientation and depth of scene surfaces. Abrupt changes in gray level can be used for partition an image. The principal areas of interest within this category are the detection of lines and edges in an image. Typical image processing module consists of four main parts, image acquisition, image segmentation, image understanding and application specific feature extraction (Fig. 1). The vision module developed in this paper consists of the same four parts, but each part is specific due to railway application and drone imagery. Image segmentation consists of the main image segmentation algorithm, pre-processing and post-processing of segmented image. The goal was to determine rail tracks in an image acquired from a drone. Image understanding is done in order to detect irregularities of the rail tracks, but also different railway infrastructure elements detection through classification is to be done in the future work. The core of the image understanding is neural network classifier that detects rail tracks and other railway infrastructure elements. 360 M. BANIĆ, A. MILTENOVIĆ, M. PAVLOVIĆ, I. ĆIRIĆ Fig. 1 Computer vision block diagram Since focus of this preliminary research was on the rail tracks, an edge detection algorithm was logical choice for the basis of the image processing research presented in this paper. Edge, in the physical sense, indicates discontinuities in physical, photometrical and geometrical properties of an object. Edges are represented in image by changes in the image intensity function. The difference between edges and lines is reflected in that edge essentially defines boundaries between two distinctly different regions, while a line may be inside of the single uniformly homogeneous region. In the process of edge detection, a significant change in pixel intensity is used for identifying and locating sharp discontinuities in an image, and thus detection of edges. The change in intensity level is measured by the gradient of the image so, since an image is a two-dimensional function, its gradient is a vector [17,18]: [ ] [ ⁄ ⁄ ] (1) The magnitude of the gradient may be computed as follows [13]: [ ] ⁄ [ ] | | | | (2) [ ] {| | | |} Eqs. (2) represent three possible ways for the computation of the magnitude of the gradient. However, the direction of the gradient is (θ is measured with respect to x-axis): ( ⁄ ) (3) Gradient operators calculate the change in gray level intensities as well as the direction of change. This calculation is performed by the difference in values of the neighboring pixels, i.e. the derivate along the x-axis and the derivate along y-axis. The gradients in a two-dimensional image are approximated by [17,18]: (4) Intelligent Machine Vision Based Railway Infrastructure Inspection and Monitoring Using UAV 361 For obtaining the x-direction gradient and y-direction gradient, gradient operators require two masks. However, these two gradients are combined in order to obtain a vector quantity whose magnitude represents the strength of the edge gradient at a point in the image and angle represents the gradient angle. The goal of ideal edge detector is to detect an edge point precisely in the sense that a true edge point in an image should not be missed, but a false edge point should not be wrong detected. However, since quality of detection is dependent on lighting conditions, the presence of objects of similar intensities, density of edges in the scene, and noise, different operator-based edge detectors are used for different purposes. Fig. 2 Block diagram of feature-based classification Feature-based classification (Fig. 2) is most common for image understanding. From segmented objects we can extract many features, like height, width, eccentricity color, or same shape descriptors that can help us in classification [14, 15]. The contour-based approach is not as popular as the texture-based approach because of the complexity of detecting extended contours. For classification of the segmented contours mean distance k-nearest neighbor (k- NN) machine learning approach is proposed, where the average distance between the query and the samples is calculated and the class is assigned taking in account this distance [16]. Because of the simplicity, effectiveness and implementation of the k-NN based classification, k-NN has been viewed as one of the top 10 algorithms in data mining. The main idea of using mean distance k-NN is to assign the query sample to the class whose mean distance to the query sample is smaller, instead of assigning it to the most represented class. 3. EXPERIMENTAL RESULTS AND DISCUSSION The edge detector presented in this paper is based on a single derivative Canny edge detector. The Canny edge detector is based on operator that uses multi-stage algorithm, which includes: detection of the edge with a low error rate, the edge point detected should be well localized (located edges must be as close as possible to the true edges) and a single edge point response (detector should return only one point for each true edge point). The Canny edge detection algorithm consists of the following basic steps [11, 12, 17-18]: 1. Smooth the input image with a Gaussian filter in order to remove the noise; 2. Calculate the gradient magnitude and angle images; 3. Apply non-maximum suppression to the gradient magnitude image; 4. Use double threshold to determine potential edges; 5. Connectivity analysis to finalize detection by suppressing all the other edges that are weak and not connected to strong edges. 362 M. BANIĆ, A. MILTENOVIĆ, M. PAVLOVIĆ, I. ĆIRIĆ Fig. 3 Original image acquired by drone, adaptive Canny edge detection processing results and rail tracks detected The images were extracted from a 19201080 AVI file created by camera mounted on a DJI Phantom III Advanced drone. The video file was created at 24 FPS with a video bitrate of 40 Mbps. For Canny edge detector, all frames were extracted from the video file. The Canny edge detection algorithm was applied to acquired images, while simple adaptive parameter tuning previously developed by authors [11, 12] was used. Small edges that are not part of the rail track are removed by the simple filter. In order to determine whether the detected edges are rail tracks or not, the edges were classified using k-NN Mean Euclidean distances. Training data set was formed from 500 frames of video and 3671 edges manually labeled edges, where 4 features were extracted (height, width, eccentricity and linear approximation error). Intelligent Machine Vision Based Railway Infrastructure Inspection and Monitoring Using UAV 363 Rail track image processing results with developed k-NN classifier with k=7 and detected rail tracks are shown in Fig. 3 for 5 video frames. Developed system was tested for various height and speed of a drone and various light and weather conditions. Developed algorithm showed good results in detecting of rail tracks and various railway infrastructure elements (Fig. 3). However, despite the good preliminary results, in order to improve robustness and adaptability, tools from artificial intelligence domain can be additionally used in future work for determining of adaptive parameters for object detection, object classification and distance estimation. 4. CONCLUSIONS Competitiveness, efficiency and operational reliability of European railway infrastructure can be achieved through the development of innovative solutions for measuring and monitoring of railway assets based on machine vision technologies. The primary goal of increasing the quality of rail freight as well its effectiveness and capacity, is in line with European transport strategy 2011-2021 (Roadmap to a Single European Transport Area - Towards a competitive and resource efficient transport system). Initiatives proposed have a goal to shift 30% of road freight over 300 km to other transport modes such as rail or waterborne transport by 2030, and more than 50% by 2050. In order to achieve such vision, developing of new infrastructure will be necessary, as well as increasing of efficiency throughout the existing infrastructure. For increasing quality and safety of railway transport, many monitoring systems can be used but, because of the infrastructure, their usage is limited. In this paper, an advanced drone imagery system for autonomous inspection and monitoring of rail tracks is presented. The advanced image processing algorithms were suggested, implemented and tested. Canny edge detector has shown promising results for rail tracks detection, while some intelligent algorithms have great potential in adaptive adjustment of parameters of conventional image processing algorithms. Based on detected rail tracks, some problem or fault can be detected and therefore a quality and safety can be increased. The presented system is one step forward towards total railway automation. Shift2Rail Multi -annual Action Plan [Shift2Rail 2015] outlines numerous advantages of railway automation which include the benefits for quality of service (due to better punctuality), increase of capacity (10 – 50%), reduced system costs (20% energy saving), reduction of operation costs (50% reduction of cost for drivers) and overall efficiency increase of 10 %. All of these benefits will result in a system cost reduction in the three-digit million Euro range, as well as great customer benefits [Shift2Rail 2015]. The railway automation is of highest priority for the future of European rail transportation and it is one of innovations which will lead to turnaround necessary for achievement of transport strategy goals. Acknowledgements: This research has been done in the framework of Horizon 2020 Shift2Rail project "Smart Automation of Rail Transport - SMART". The authors would like to thank the Serbian Railway Infrastructure for issuing permit and providing operational assistance for testing. 364 M. BANIĆ, A. MILTENOVIĆ, M. PAVLOVIĆ, I. ĆIRIĆ REFERENCES 1. 2016, SESAR European Drones Outlook Study, accessed on 10.10.2018. 2. Kim, J.W., Kim, S.B., Park, J.C., Nam, J.W., 2015, Development of crack detection system with Unmanned Aerial Vehicles and digital image processing, Proc. The 2015 World Congress on Advances in Structural Engineering and Mechanics (ASEM15), South Korea. 3. Smith, E.M., 2016, A collection of computer vision algorithms capable of detecting linear infrastructure for the purpose of UAV control, MSc Thesis, Virginia Tech, USA, 101 p. 4. Mathe, K., Busoniu, L., Barabas, L., Iuga, C.I., Miclea, L., Braband, J., 2016, Vision-based control of a quadrotor for an object inspection scenario, Proc. 2016 International Conference on Unmanned Aircraft Systems (ICUAS). 5. Gemert, J., Verschoor, C.R., Mettes, P., Epema, K., Koh, L. P., Wich, S., 2014, Nature conservation drones for automatic localization and counting of animals, ECCV 2014: Computer Vision - ECCV 2014 Workshops, pp. 255-270. 6. Karakose, M., Yaman, O., Baygin, M., Murat, K., Akin, E., 2017, A new computer vision based method for rail track detection and fault diagnosis in railways, International Journal of Mechanical Engineering and Robotics Research, 6(1), pp. 22-27. 7. Flammini, F., Pragliola, C., Smarra, G., 2016, Railway infrastructure monitoring by drones, Proc. 2016 International Conference on Electrical Systems for Aircraft, Railway, Ship Propulsion and Road Vehicles & International Transportation Electrification Conference (ESARS-ITEC). 8. Mittal, S., Rao, D., 2017, Vision based railway track monitoring using deep learning, https://arxiv.org/abs/1711.06423. (last access: 03.03.2019) 9. Singha, A. K., Swarupa, A., Agarwalb, A., Singha, D., 2018, Vision based rail track extraction and monitoring through drone imagery, ICT Express. 10. Jianfang, L., Hao, Z., Jingli, G., 2017, A novel fast target tracking method for UAV aerial image, Open Physics, 15(1), pp. 420–426. 11. Pavlović, M., Pavlović, N.T., Pavlović, V., 2016, Methods for detection of obstacles on railway level crossings,Proc. 17th Scientific-Expert Conference on Railways RAILCON ‘16, pp. 121-124. 12. Pavlović, M., Nikolić, V., Ćirić I., Ćirić, M., 2017, Application of thermal imaging systems for object detection, Proc. 13th International Conference on Accomplishments in Mechanical and Industrial Engineering, Banja Luka, Republic of Srpska, pp. 653-662. 13. Ćirić, I., Ćojbašić, Ž., Nikolić, V., Antić, D., 2013, Computationally intelligent system for thermal vision people detection and tracking in robotic applications, Proc. 11th International Conference on Telecommunication in Modern Satellite, Cable and Broadcasting Services (TELSIKS), pp. 587-590. 14. Ćirić, I., Ćojbašić, Ž., Nikolić, V., Igić, T., Turnšek, B., 2014, Intelligent optimal control of thermal vision-based person-following robot platform, Thermal Science, 18(3), pp. 957-966. 15. Ćirić, I., Ćojbašić, Ž., Ristić-Durrant, D., Nikolić, V., Ćirić, M., Simonović, M., Pavlović, I., 2016, Thermal vision based intelligent system for human detection and tracking in mobile robot control system, Thermal Science, 20(5), pp. S1553-S1559. 16. García-Ordás, M.T., Alegre, E., García-Olalla, O., García-Ordás, D., 2013, Evaluation of different metrics for shape based image retrieval using a new contour points descriptor. In: Brisaboa N., Pedreira O., Zezula P. (eds) Similarity Search and Applications. SISAP 2013. Lecture Notes in Computer Science, vol 8199. Springer, Berlin, Heidelberg 17. Acharya, T., Ray, A.K., 2005, Image processing: principles and applications, John Wiley & Sons Inc., New Jersey, USA 18. Nadernejad, E., Sharifzadeh, S., Hassanpour, H., 2008, Edge detection techniques: evaluations and comparisons, Applied Mathematical Sciences, 2(31), pp. 1507 – 1520.