Microsoft Word - ETASR_V13_N3_pp10765-10768 Engineering, Technology & Applied Science Research Vol. 13, No. 3, 2023, 10765-10768 10765 www.etasr.com Doan Van: Application of Advanced Deep Convolutional Neural Networks for the Recognition of Road … Application of Advanced Deep Convolutional Neural Networks for the Recognition of Road Surface Anomalies Dong Doan Van Science and Technology Application for Sustainable Development Research Group, Ho Chi Minh City University of Transport, Vietnam dongdv@ut.edu.vn (corresponding author) Received: 28 March 2023 | Revised: 8 April 2023 | Accepted: 14 April 2023 Licensed under a CC-BY 4.0 license | Copyright (c) by the authors | DOI: https://doi.org/10.48084/etasr.5890 ABSTRACT The detection of road surface anomalies is a crucial task for modern traffic monitoring systems. In this paper, we used the YOLOv8 network,- a state-of-the-art convolutional neural network architecture, for real-time object recognition and to automatically identify potholes, cracks, and patches on the road surface. We created a custom dataset of 1044 road surface images in Vietnam, each of which was annotated with pavement anomalies, and the YOLOv8 network was trained with this dataset. The results show that the model achieved an accuracy of 0.56 mAP at a threshold of 0.5, indicating its potential for practical application. Keywords-road surface anomalies; convolutional neural networks; digital image processing; transportation I. INTRODUCTION Identifying anomalies on road surfaces such as potholes, cracks, and bumps is an important factor in creating conditions for road maintenance, providing a better driving experience, and reducing the risk of accidents (collisions, falls, etc.) [1-5]. Analyzing data related to the condition of streets promptly can help make better decisions about transportation spending [4]. The anomalies on the road surface are repaired when they are reported by citizens or when a major incident occurs. However, a real-time reaction system that automatically detects various anomalies on urban and national roads does not exist. Systems for identifying anomalies on roads can be divided into three categories: vision-based, sensor-based, and 3D reconstruction methods [6]. The sensor-based method mostly uses sensor data to identify road anomalies. Authors in [5] compared the Decision Tree (DT) and Support Vector Machine (SVM) algorithms for classifying abnormalities on the road using data measured from acceleration and gyro sensors. Authors in [7] used inertial sensor datasets collected in different contexts to detect and classify abnormalities on road surfaces (e.g. dirt roads, cobblestones, and asphalt roads). Based on the reported results, the proposed Convolutional Neural Network (CNN) model achieved the best performance with an accuracy of 93.17%. Authors in [8] developed a hybrid method combining threshold-based signal processing techniques and machine learning algorithms to form a near real-time road anomaly detection system. On the other hand, the technique that uses 3D reconstruction to anticipate the shape of the road anomalies and evaluate their volume through stereo-vision technology is considered the most precise of the three methods. However, this method is more costly and difficult to identify when potholes are filled with water or dirt than other approaches. For instance, authors in [9] developed a pixel-level road surface anomaly detection approach based on stereo vision and deep learning. Specifically, the vehicle-mounted photography system was used to capture both parallel and oblique photos to generate a 3D pavement point-cloud model. Stereo-vision technology was employed in the 3D reconstruction phase to process the input images. Point-cloud calibration relied on a PCA algorithm, and various orthoimages, including color, depth, and overlapped images, were generated during the 3D data-processing phase. To identify pavement cracks and potholes in the orthoimages, a modified U-net deep-learning technology was utilized for segmentation. Their approach achieved significant results: 0.9632 precision, 0.9552 recall, and 0.9592 F1 score. The vision-based method uses images to identify the presence of abnormalities through image processing algorithms. The advantage of this method is that it does not require direct access to the location of the abnormalities on the road, making it easy to detect multiple objects at the same time through traffic monitoring cameras or cameras on mobile devices. For example, authors in [10] proposed a real-time automatic pavement crack and pothole recognition system using a mobile device. The proposed system achieved only 0.7 precision, recall, accuracy, and F1 score. Recently, deep learning techniques have gained widespread application in diverse fields [11-14]. These methods have also been employed in the identification of road surface anomalies, leveraging their strengths such as accurate detection and the ability to handle Engineering, Technology & Applied Science Research Vol. 13, No. 3, 2023, 10765-10768 10766 www.etasr.com Doan Van: Application of Advanced Deep Convolutional Neural Networks for the Recognition of Road … intricate data. For example, authors in [1] discussed a deep learning algorithm for detecting potholes on road surfaces. The algorithm employed a CNN with 9 layers. However, the method is not suitable for real-time applications because it cannot be used for online video processing. Authors in [2] developed a system based on YOLOv2 [15] network to detect potholes on roads. However, their system can only run offline and cannot be used in real-time applications. Additionally, the system's accuracy only reached 82.5%. Authors in [16] proposed a lightweight CNN model based on a modified MobileNetv2 [17] that can operate on edge devices. The proposed system is capable of performing pixel-wise crack detection on streets. The common drawback of the image processing methods mentioned above is that they cannot meet real-time operational criteria. Therefore, in this article, we propose a method that used YOLOv8, an advanced CNN model capable of real-time object detection with high accuracy. The proposed method can detect abnormalities on the road surface such as potholes, cracks, and road patches. The technical contributions of this paper can be summarized as: To the best of our knowledge, this paper is the first that applies the state-of-the-art YOLOv8 architecture to road anomaly detection. The proposed method can detect in real time various types of anomalies such as potholes, cracks, and patches. The empirical results suggest that the proposed method can be applied in practical settings with suitable modifications. II. ROAD ANOMALIES DATASET We constructed a road surface dataset consisting of 1044 images collected from random roads in Vietnam. Figure 1 shows some examples from the dataset. The data were taken at different times and weather conditions, resulting in a wide range of lighting and shadow conditions. This is the biggest challenge for abnormal road detection methods based on image processing. The dataset has a total of 1044 images and was divided into 967 images for training of the network and 77 images for model evaluation. Each image was labeled with three kinds of anomalies namely potholes, cracks, and road patches. The instance distribution is presented in Figure 2. III. METHODOLOGY A. YOLOv8 Network Released at the beginning of 2023, YOLOv8 is the latest generation of the YOLO network, in particular, and is currently the most efficient model in tasks such as classification, detection, and segmentation of objects [18]. In object detection tasks, YOLOv8 can achieve superior results and faster processing times than other models thanks to its combination of optimization techniques and improvements. Specifically, the YOLOv8m model achieved a 50.2% mAP score on the COCO dataset, which is higher than its predecessors [18]. Also, the model requires fewer parameters than the others. Fig. 1. Examples in the dataset. Fig. 2. Distribution of road anomalies in the dataset. YOLOv8 utilizes advanced techniques in the object detection field such as decoupled head and anchor-free detection. Also, novel ideas, such as mosaic stopping strategy that skips mosaic augmentation for the last 10 epochs, were introduced. The modifications compared to YOLOv5 [18, 19] are: The C3 module was replaced with the C2 module 5. Engineering, Technology & Applied Science Research Vol. 13, No. 3, 2023, 10765-10768 10767 www.etasr.com Doan Van: Application of Advanced Deep Convolutional Neural Networks for the Recognition of Road … The first 6×x6 Conv was replaced with a 3×3 Conv in the Backbone. The first 1×1 Conv` was replaced with a 3×3 Conv in the Bottleneck. Decoupled head was used and the objectness branch was deleted. This technique separates the classification and regression tasks into two separate subnetworks, each with its own set of parameters [11, 20]. Fig. 3. Evaluation results of proposed model. Fig. 4. Visualization of the results in real-time application. B. Experimental Environment The proposed YOLOv8 model was trained for 200 iterations with a batch size of 8. Due to the relatively small sizes of the objects of interest, the input image size increased from 640×640 to 1280×1280. All experiments were run on a computer with the following configuration: GPU Nvidia RTX3050 4 GB VRAM CPU AMD Ryzen 5 5600H, 3.3 Hz 16 Gb RAM IV. RESULTS AND DISCUSSION The performance of the anomaly detection model on the road is shown in Figure 3. Specifically, the model’s precision achieved 84%, and the average accuracy was 56.8% at a confidence threshold of 0.5. The recall criterion, which indicates the ability to detect all objects present in the image, achieved 60% score. The visualized results of the evaluation set can be observed in Figure 4. The results show that the model is capable of detecting anomalies such as potholes, cracks, and patches on the road with high accuracy. Engineering, Technology & Applied Science Research Vol. 13, No. 3, 2023, 10765-10768 10768 www.etasr.com Doan Van: Application of Advanced Deep Convolutional Neural Networks for the Recognition of Road … V. CONCLUSION In this paper, the utilization of the advanced YOLOv8 convolutional neural network architecture to address the issue of detecting road anomalies was discussed. The study shows that the proposed model's performance is promising, with MAP of 0.56 at the threshold of 0.5, suggesting that it can be applied in practical settings with suitable modifications. We intend to gather additional road data and further finetune the model to achieve better results in the future. REFERENCES [1] V. Pereira, S. Tamura, S. Hayamizu, and H. Fukai, "A Deep Learning- Based Approach for Road Pothole Detection in Timor Leste," in 2018 IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI), Singapore, Jul. 2018, pp. 279–284, https://doi.org/10.1109/SOLI.2018.8476795. [2] K. E. An, S. W. Lee, S.-K. Ryu, and D. Seo, "Detecting a pothole using deep convolutional neural network models for an adaptive shock observing in a vehicle driving," in 2018 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA, Jan. 2018, https://doi.org/10.1109/ICCE.2018.8326142. [3] J. M. Celaya-Padilla et al., "Speed Bump Detection Using Accelerometric Features: A Genetic Algorithm Approach," Sensors, vol. 18, no. 2, Feb. 2018, Art. no. 443, https://doi.org/10.3390/s18020443. [4] F. Seraj, B. J. van der Zwaag, A. Dilo, T. Luarasi, and P. Havinga, "RoADS: A Road Pavement Monitoring System for Anomaly Detection Using Smart Phones," in Big Data Analytics in the Social and Ubiquitous Context, 2016, pp. 128–146, https://doi.org/10.1007/978-3- 319-29009-6_7. [5] A. Basavaraju, J. Du, F. Zhou, and J. Ji, "A Machine Learning Approach to Road Surface Anomaly Assessment Using Smartphone Sensors," IEEE Sensors Journal, vol. 20, no. 5, pp. 2635–2647, Mar. 2020, https://doi.org/10.1109/JSEN.2019.2952857. [6] Y.-M. Kim, Y.-G. Kim, S.-Y. Son, S.-Y. Lim, B.-Y. Choi, and D.-H. Choi, "Review of Recent Automated Pothole-Detection Methods," Applied Sciences, vol. 12, no. 11, Jan. 2022, Art. no. 5320, https://doi.org/10.3390/app12115320. [7] J. Menegazzo and A. von Wangenheim, "Road surface type classification based on inertial sensors and machine learning," Computing, vol. 103, no. 10, pp. 2143–2170, Oct. 2021, https://doi.org/ 10.1007/s00607-021-00914-0. [8] S. Sattar, S. Li, and M. Chapman, "Developing a near real-time road surface anomaly detection approach for road surface monitoring," Measurement, vol. 185, Nov. 2021, Art. no. 109990, https://doi.org/ 10.1016/j.measurement.2021.109990. [9] J. Guan, X. Yang, L. Ding, X. Cheng, V. C. S. Lee, and C. Jin, "Automated pixel-level pavement distress detection based on stereo vision and deep learning," Automation in Construction, vol. 129, Sep. 2021, Art. no. 103788, https://doi.org/10.1016/j.autcon.2021.103788. [10] A. Tedeschi and F. Benedetto, "A real-time automatic pavement crack and pothole recognition system for mobile Android-based devices," Advanced Engineering Informatics, vol. 32, pp. 11–25, Apr. 2017, https://doi.org/10.1016/j.aei.2016.12.004. [11] H. D. Quy, N. N. Son, and H. P. H. Anh, "DeYOLOv3: An Optimal Mass Detector for Advanced Breast Cancer Diagnostics," in Computational Intelligence Methods for Green Technology and Sustainable Development, 2023, pp. 325–335, https://doi.org/ 10.1007/978-3-031-19694-2_29. [12] V. T. H. Tuyet, N. T. Binh, and D. T. Tin, "Improving the Curvelet Saliency and Deep Convolutional Neural Networks for Diabetic Retinopathy Classification in Fundus Images," Engineering, Technology & Applied Science Research, vol. 12, no. 1, pp. 8204–8209, Feb. 2022, https://doi.org/10.48084/etasr.4679. [13] D. Patil and S. Jadhav, "Road Segmentation in High-Resolution Images Using Deep Residual Networks," Engineering, Technology & Applied Science Research, vol. 12, no. 6, pp. 9654–9660, Dec. 2022, https://doi.org/10.48084/etasr.5247. [14] N. C. Kundur and P. B. Mallikarjuna, "Deep Convolutional Neural Network Architecture for Plant Seedling Classification," Engineering, Technology & Applied Science Research, vol. 12, no. 6, pp. 9464–9470, Dec. 2022, https://doi.org/10.48084/etasr.5282. [15] J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, Jul. 2017, pp. 6517–6525, https://doi.org/ 10.1109/CVPR.2017.690. [16] G. Doğan and B. Ergen, "A new mobile convolutional neural network- based approach for pixel-wise road surface crack detection," Measurement, vol. 195, May 2022, Art. no. 111119¸https://doi.org/ 10.1016/j.measurement.2022.111119. [17] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks." arXiv, Mar. 21, 2019, https://doi.org/10.48550/arXiv.1801.04381. [18] G. Jocher, A. Chaurasia, and J. Qiu, "YOLO by Ultralytics." Jan. 2023, [Online]. Available: https://github.com/ultralytics/ultralytics. [19] G. Jocher et al., "ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation." Zenodo, Aug. 22, 2022, https://doi.org/10.5281/ zenodo.7347926. [20] Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, "YOLOX: Exceeding YOLO Series in 2021." arXiv, Aug. 05, 2021, https://doi.org/10.48550/ arXiv.2107.08430.