Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 10, No. 3, 2024 125 Vehicle Detection and Tracking Techniques Based on Deep Learning in Road Traffic Surveillance Xiaolong Liu1, a, Nelson C. Rodelas2, * 1Graduate School, University of the East, Manila, 1008, Metro Manila, Philippines 2Graduate School, University of the East, Manila, 1008, Metro Manila, Philippines aliu.xiaolong@ue.edu.ph, * Corresponding author: nelson.rodelas@ue.edu.ph Abstract: This article discusses the application of deep learning in vehicle detection and tracking technology, elaborating on the basic concepts of deep learning and its advantages in vehicle target detection. Deep learning models such as Convolutional Neural Networks (CNNs) overcome the reliance on manual feature engineering by automatically learning image features. The article focuses on two deep learning detection frameworks, Faster R-CNN and YOLO. The former combines region proposal networks with region classification networks to achieve end-to-end optimization, while the latter transforms the detection task into a regression problem, enabling real-time detection in a single forward pass. Regarding vehicle tracking, the article explores the challenges of multi-object tracking such as occlusion, cross-movement, and the tracking requirements of different vehicle types. Deep learning applications in this field, such as the DeepSORT and Tracktor algorithms, combine CNNs, RNNs, and traditional tracking methods to achieve feature learning, historical state modeling, and probabilistic reasoning. Performance evaluation is discussed in terms of metrics like IoU, precision, recall, and F1 Score, comparing and analyzing the performance of different algorithms in vehicle detection and tracking tasks. Lastly, the article discusses the balance between real-time and accuracy in deep learning-based vehicle detection and tracking technology in road traffic monitoring, as well as its significant role in traffic accident warning and management. Keywords: Deep learning; tracktor algorithm; vehicle detection. 1. Introduction With the rapid growth of urban traffic, traditional vehicle detection and tracking technology face challenges in dealing with large-scale data and complex scenarios. Based on the advantages of deep learning in feature learning and pattern recognition, deep learning-based vehicle detection and tracking technology, along with trajectory prediction methods based on deep learning, have shown outstanding performance in long-term, multimodal motion, and vehicle-road interaction scenarios. [1] This provides new solutions for vehicle detection and tracking, and this paper aims to explore the principles, methods, and application prospects of this technology. 2. Application of Deep Learning in Vehicle Detection 2.1. Basic Concepts of Deep Learning In deep learning, neural networks are computational models that mimic the structure and function of the biological nervous system. They consist of a large number of artificial neurons interconnected, forming a network structure where information is passed layer by layer to achieve learning and decision-making. The basic unit of a neural network simulates some functions of biological neurons; each neuron receives a set of input signals (from the previous layer's neurons' outputs or directly from input data), processes them through weighted summation followed by a nonlinear activation function (such as ReLU, sigmoid, tanh, etc.) to produce a single output signal. In CNNs for vehicle detection, a neuron may correspond to features of a small pixel area in an image. Typically, several convolutional layers are used to extract image features, followed by pooling layers to reduce dimensionality, and finally connected to fully connected layers or specific detection heads (like YOLO's fully convolutional structure) for predicting vehicle categories and positions. Typical neural network structures include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and their variants or combinations. The input layer receives raw data and transforms it into a format understandable by the network for subsequent processing. For CNNs, the input layer usually takes image data, such as an RGB color image, representing it as a three-dimensional tensor (height, width, number of channels), where each pixel's value corresponds to its red, green, blue channel intensity. In RNNs, the hidden layer contains recurrent units (like LSTM or GRU), which extract features from data through multiple computations and learning, with the output layer performing classification or regression predictions, capturing long-term dependencies in time series data. The working principle of neural networks is based on the backpropagation algorithm, a gradient-based optimization method used to compute gradients of all learnable parameters in the neural network, indicating how small changes in parameters affect the loss function. Through training data, network parameters are adjusted continuously to minimize the error between network output and actual results. Backpropagation decomposes the partial derivatives (gradients) of the loss function for each parameter using the chain rule, comprising gradients of activation functions, weights, biases, etc. Traditional machine learning methods usually require manual feature design, where "features" refer to key information extracted from raw data, effectively reflecting the core attributes or patterns of data, facilitating subsequent model training and prediction. In contrast, deep learning can automatically learn feature representations of data through networks. This feature learning process is done layer by layer, 126 significantly reducing reliance on manual feature engineering compared to traditional machine learning, with each layer gradually increasing the data's abstraction level. By constructing deep learning models tailored for vehicle detection tasks, specifically addressing the recognition and tracking needs of vehicles in various environmental conditions, customizing designs for challenges like vehicle morphology, background complexity, lighting changes, perspective shifts, etc., using convolutional kernels for sliding window scanning of images to extract local features such as vehicle edges, shapes, colors, textures, etc., can achieve automatic vehicle recognition and tracking. Based on Convolutional Neural Networks (CNNs), vehicle detection models utilizing convolutional kernels for sliding window scanning of images to extract local features like vehicle edges, shapes, colors, textures, etc., can effectively extract vehicle features from images and accurately recognize and locate them. 2.2. Deep Learning Applications in Vehicle Object Detection Traditional methods of object detection mainly involve techniques based on feature engineering and machine learning, such as Haar features, Histogram of Oriented Gradients (HOG) features, and Support Vector Machines (SVM). Haar features, proposed by Paul Viola and Michael Jones, are simple and efficient visual features that essentially compute contrast on image subregions, typically manifested as rectangular structures including single rectangles, differential adjacent rectangles, and more complex multi- rectangle combinations. Haar features are often combined with cascade classifiers and the AdaBoost algorithm. Cascade classifiers are composed of multiple weak classifiers connected in series, with each classifier responsible for screening out a portion of non-target regions, thereby reducing the computational burden on subsequent classifiers. Histogram of Oriented Gradients (HOG) features, introduced by Navneet Dalal and Bill Triggs, are feature descriptors based on local gradient direction histograms. HOG features have shown excellent performance in tasks like pedestrian detection and vehicle detection, particularly becoming one of the mainstream features in early computer vision competitions like the PASCAL VOC challenge. While these methods demonstrate effectiveness in specific scenarios, their ability to detect objects in complex environments and small targets is challenged as the field of computer vision evolves. In contrast, deep learning-based object detection methods are better equipped to address these challenges. Among them, Faster R-CNN (Faster Region-based Convolutional Neural Network) stands as a classic object detection framework proposed by Ross Girshick and others. It combines a Region Proposal Network (RPN) and a Region-based Convolutional Neural Network (RCNN) to optimize the entire model from input image to final detection output as a whole, reducing error accumulation between feature extraction and classification decisions seen in traditional methods, achieving end-to-end object detection. The RPN generates candidate regions, while the RCNN, as the first stage of Faster R-CNN, is responsible for classifying and locating these candidate regions. Its core task is to automatically generate candidate regions (Regions of Interests, RoIs) that may contain target objects, reducing the computational burden of subsequent processing and thus balancing the accuracy and efficiency of object detection. Aditionally, YOLO (You Only Look Once) is another popular object detection algorithm that innovatively treats the entire object detection task as a single regression problem， YOLOv4 is a widely used object detection technology that boasts high accuracy and fast inference speed[2], unlike traditional two-stage or multi-stage methods. Its core idea is to transform the object detection problem into a regression problem, requiring only one forward pass on the input image to directly predict the class and bounding box of the target at the image level, achieving fast, real-time object detection. The YOLO algorithm segments the image into grids, with each grid cell responsible for predicting whether there is an object in its coverage area, including predicting the presence of objects, their positions, and categories, as well as multiple bounding boxes (3 in YOLOv3) containing 5 coordinate values (center coordinates, width, height, and confidence) and multiple class probabilities, thus enabling real-time object detection capabilities. 3. Based on Deep Learning Techniques for Vehicle Tracking 3.1. Overview of Vehicle Tracking Problem Vehicle re-identification technology falls within the realm of urban intelligent transportation and has garnered widespread attention due to its ability to identify vehicles based solely on their appearance.[3] Multiple-object tracking algorithms face challenges in complex traffic scenarios where vehicles frequently experience occlusion and intersecting motion trajectories. For instance, when a car is driving in front of a bus, the car may be partially or fully occluded by the bus, causing its visual features such as color and shape to become blurry or even invisible from the camera's perspective. This variation in target appearance increases the difficulty of tracking. Different types of vehicles (e.g., cars, trucks, bicycles) vary significantly in size and motion characteristics. For example, small cars, large buses, trucks, motorcycles, and bicycles differ in size, shape, speed, acceleration, etc. Tracking algorithms need to adapt well and generalize effectively to handle different types of vehicles, regardless of their size, changing shapes, or complex motion patterns. Real-time processing is an indispensable feature of multiple-object tracking algorithms in practical applications, yet it poses a significant challenge. These algorithms must process large-scale data and achieve real-time tracking within a short timeframe, updating target state information promptly to make timely decisions and avoid safety risks or monitoring failures caused by processing delays. Hence, multiple-object tracking algorithms need to swiftly perform a series of operations including object detection, association, and state updating on each frame of the image. They must output the latest positions, speeds, orientations, etc., of all targets in milliseconds, enabling decision-making modules to plan driving paths and avoid collision risks. Deep learning-based vehicle tracking technology utilizes the powerful representation capabilities of Convolutional Neural Networks (CNNs) in deep learning. It designs specific feature extraction network structures to address the challenges and requirements of multiple-object tracking algorithms. By designing feature extraction networks suitable for complex scenes, such as CNN-based feature extraction modules, it can effectively capture spatial and temporal information of targets, automatically extracting hierarchical and abstract features from input images, covering spatial 127 layouts, textures, colors, etc., of targets as well as their dynamic features evolving over time. Furthermore, it adopts target matching and trajectory prediction techniques, combining tools like Recurrent Neural Networks (RNNs) or Long Short-Term Memory networks (LSTMs) for sequence modeling. These networks can capture the temporal dependencies of target motion, predict future positions, speeds, directions, etc., based on historical state information, enabling continuous tracking and prediction of target motion trajectories, thereby enhancing tracking stability and robustness. For instance, in a highway monitoring scenario, a car gradually accelerates and changes lanes over several frames of images. An LSTM-based trajectory prediction model can learn the speed trend of the car from past frames and the lane-changing pattern. Even if other vehicles temporarily obstruct the line of sight, the prediction model can maintain effective tracking of the target car based on the established motion model. Combining traditional target association and state estimation algorithms such as Kalman Filters or graph-based tracking methods with deep learning frameworks for probabilistic reasoning and optimal state estimation can effectively address issues like target occlusion and intersecting motion, improving the accuracy and efficiency of multiple-object tracking algorithms. For example, when two cars traveling in the same direction at a crossroad slow down as they approach, then almost simultaneously enter different turning lanes causing severe visual intersection, the multiple- object tracking system will construct a graph model representing potential correlations between targets. It combines each target's observation information (e.g., detection box position, size) with the state prediction provided by Kalman Filters to determine the true identity of each detection box in the current frame via Maximum A Posteriori (MAP) inference, as well as their correct trajectories after the intersecting motion. 3.2. Application of Deep Learning in Vehicle Tracking In the combination of deep learning and tracking algorithms, DeepSORT is a deep learning-based online multi- object tracking method. Its core idea is to integrate the representation power of deep learning with traditional techniques like Kalman filtering and the Hungarian algorithm to achieve robust target tracking. DeepSORT combines the characteristics of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), enabling feature extraction and historical state modeling of targets. Specifically, it utilizes pre-trained CNNs (e.g., ResNet) to extract appearance features of targets (strong invariance to lighting, pose, partial occlusion, etc.). In each frame, DeepSORT first uses the CNN network to extract features of targets, then models and predicts the motion trajectory of targets using the RNN network. It uses a metric learning network to calculate the similarity between newly detected targets and existing trajectories, combined with the Hungarian algorithm for data association, thereby achieving accurate tracking and state prediction of multiple targets. DeepSORT maintains a historical feature queue for each target, so when a target briefly disappears and reappears, it can compare the current detected target features with the historical feature queue, leveraging static appearance features extracted by deep learning instead of dynamic sequence modeling through RNNs. Apart from DeepSORT, there are other deep learning-based multi-object tracking methods like Tracktor. Tracktor is an innovative multi-object tracking algorithm that combines the feature learning ability of deep learning with traditional tracking optimization strategies. Its effectiveness depends on whether the model can extract task-relevant and discriminative information from raw data (such as images or video frames). The Tracktor algorithm achieves stable tracking and state estimation of targets through effective feature representation and target matching strategies. By carefully designing network structures and training strategies, Tracktor ensures that learned features have good invariance to factors like lighting changes, viewpoint variations, and partial occlusions, thereby improving tracking robustness. It combines the feature learning ability of deep learning with tracking algorithm optimization strategies, making it suitable for addressing target tracking challenges in complex scenes, although there may be differences in specific implementations and technical paths. 4. Performance Evaluation and Comparison of Vehicle Detection and Tracking Techniques 4.1. Detection Accuracy Evaluation Metrics The Intersection over Union (IoU) metric is widely used in the computer vision field to measure the degree of overlap between the output of object detection algorithms and the ground truth bounding boxes. It is a key metric for assessing model prediction accuracy, localization accuracy, and overall object detection performance. Specifically, IoU is calculated by dividing the intersection area of the ground truth bounding box and the detected bounding box by their union area, given as IoU = (Intersection Area) / (Union Area). The IoU value ranges from 0 to 1, where a value closer to 1 indicates a higher degree of match between the detection result and the ground truth. As the IoU value decreases, the overlap between the predicted bounding box and the ground truth bounding box decreases, leading to reduced localization accuracy. Predictions with IoU below a certain threshold (e.g., 0.5) may be considered as false positives or false negatives. In addition to the IoU metric, there are other commonly used detection accuracy evaluation metrics such as precision (measuring the proportion of true positives among all samples classified as positive by the model), recall (measuring the proportion of actual positive samples that are successfully detected by the model), and F1 Score (the harmonic mean of precision and recall, providing a single value to comprehensively reflect the model's performance in terms of both precision and recall). Precision focuses on the reliability of the model's predictions, indicating the proportion of correctly identified positive samples among the detected positives. Recall emphasizes the comprehensiveness of the model in finding targets, representing the proportion of all true positive samples that are detected. F1 Score balances precision and recall, providing a comprehensive evaluation metric that considers both the accuracy and completeness of the algorithm. In practical applications such as vehicle detection and tracking technologies, the IoU metric is commonly used to evaluate the accuracy and robustness of object detection algorithms. It quantifies the overlap between the model's predicted bounding boxes and the actual ground truth bounding boxes, providing an objective and consistent 128 standard for assessing the performance of different object detection algorithms. By calculating the IoU metric, the degree of match between the detection results and the ground truth can be determined, enabling performance evaluation and comparison of algorithms. 4.2. Performance Comparison Analysis of Different Algorithms For vehicle detection tasks, commonly used deep learning methods include Faster R-CNN, YOLO (You Only Look Once), and SSD (Single Shot Multibox Detector).[4] Faster R-CNN excels in accuracy, especially for detecting small objects and in complex scenarios. Its two-stage design allows for multiple iterations of optimization within the network, thus improving localization accuracy. YOLO boasts excellent real-time performance and ease of deployment, making it suitable for large-scale object detection tasks like vehicle detection on highways, where it often exhibits good performance. SSD balances accuracy and speed, with its multi-scale detection mechanism making it adaptable to different sizes of vehicles, especially addressing common distance and perspective variations encountered in vehicle detection tasks. For vehicle tracking tasks, commonly used deep learning methods include DeepSORT (Deep Simple Online and Realtime Tracking) and Tracktor. For instance, pre-trained models like YOLO or SSD can be used for initial vehicle detection in images, followed by feeding the detected vehicle regions into specialized CNN models such as ResNet or MobileNet for feature extraction. The DeepSORT algorithm combines convolutional neural networks (CNNs) and recurrent neural networks (RNNs), where RNNs can memorize feature information of the same vehicle across past frames, forming a "trajectory embedding" to achieve accurate tracking through feature extraction and historical state modeling. The Tracktor algorithm typically extends existing detectors (e.g., Faster R-CNN, Mask R-CNN) by designing effective feature representations and target matching strategies to improve tracking accuracy and robustness. When conducting a comparative analysis of different deep learning methods for vehicle detection and tracking tasks, it's essential to consider the accuracy of detection and tracking. This involves evaluating whether the algorithms can accurately detect and track vehicle targets, with high accuracy indicating precise localization of vehicle positions, reduced false positives (misidentifying non-vehicles as vehicles), and false negatives (failing to detect actual vehicles), thus avoiding misidentifications and missed detections. Secondly, real-time performance is crucial for applications like autonomous driving and video surveillance, requiring algorithms to complete detection within milliseconds to ensure real-time synchronization with video streams, i.e., whether the algorithms can rapidly and efficiently perform object detection and tracking in real-time video streams. Robustness of the algorithm should also be considered, assessing its performance across various environmental conditions such as urban roads, highways, rural roads, tunnels, nighttime, rainy or snowy weather, etc., i.e., the algorithm's adaptability to different scenes, lighting conditions, and target scales. 5. Application in Road Traffic Monitoring 5.1. Balancing Real-Time and Accuracy A real-time monitoring system refers to the use of cameras, radars, sensors, and other devices deployed at key locations such as roads, intersections, tunnels, and bridges. One of the most fundamental requirements for a real-time monitoring system is the real-time detection and tracking of vehicles.[5] This means that the system needs to accurately identify and track vehicles that appear on the road within a short period. It requires almost real-time capture and analysis of every frame in the video stream to promptly identify and track newly appeared vehicles, update the status of existing vehicles, and respond quickly to abnormal situations. The vehicle detection and tracking in real-time monitoring systems require high real-time capabilities. Therefore, detection models that can accurately identify vehicle boundaries, types, and even vehicle attributes such as license plate numbers and colors should be selected or developed. These models should be able to complete target detection and tracking tasks quickly and timely, reducing false positives (misidentifying non-vehicles as vehicles) and missed detections (failing to detect actual vehicles). Real-time monitoring systems also need to consider accuracy issues. In continuous video streams, the system must ensure accurate detection and tracking results for vehicles. Even in complex situations such as vehicle occlusion, deformation, rapid movement, or changes in lighting conditions, there should be no ID switching or tracking loss, avoiding false positives or missed detections. 5.2. Traffic Accident Warning and Management Traffic accident warning and management systems are crucial components of modern intelligent transportation systems. Deep learning-based traffic accident prediction methods involve training and learning deep learning models using road traffic data. Detailed information such as the location, time, type, and severity of traffic accidents that occurred in the recent past is provided as learning samples to the model. Dynamic information such as vehicle position, speed, acceleration, and direction is collected through GPS, onboard sensors, and other devices to analyze vehicle behavior patterns. This analysis can effectively identify the possibility of traffic accidents occurring and simultaneously annotate accident information in real-time on the geographical information system (GIS) of the traffic management department, visually displaying accident locations and surrounding traffic conditions. These models can utilize various data sources such as historical traffic data, road conditions, and vehicle motion trajectories, undergo preprocessing operations like cleaning, formatting, and spatiotemporal alignment to form structured, standardized datasets for training and prediction by deep learning models. By learning the patterns and trends of traffic accidents, these models can predict potential traffic accidents. Predictions can be global (for an entire city or specific area) or local (for specific road sections or intersections). Setting reasonable warning thresholds, when the model predicts a risk exceeding the threshold, triggers a traffic accident warning. The warning information may include prediction time, location, risk level, potential impact range, etc. Upon receiving the warning, rescue vehicles such as ambulances 129 and fire trucks can quickly reach the scene based on the system's optimal route, ensuring timely medical treatment for the injured. The traffic management department should promptly initiate emergency plans, mobilize rescue forces, manage traffic flow, and provide road condition information to reduce accident impacts and casualties, thereby optimizing traffic signal timing, enhancing law enforcement supervision, improving road infrastructure, conducting safety education, etc., to reduce the probability of traffic accidents at the source. 6. Conclusion Deep learning has brought revolutionary progress to vehicle detection and tracking technology. Through automatic feature learning and advanced representation capabilities, it can overcome the limitations of traditional methods in complex environments, significantly improving detection accuracy and tracking stability. In practical applications, deep learning models such as Faster R-CNN, YOLO, DeepSORT, and Tracktor provide strong support for road traffic monitoring systems with their excellent performance and real-time capabilities. These technologies not only enable precise detection and tracking of vehicles but also effectively predict traffic accidents, providing decision support for traffic management departments. In the future, with the continuous development and optimization of deep learning technology, the level of intelligence in vehicle detection and tracking is expected to further improve, playing a greater role in building a safer and more efficient traffic environment. References [1] Yang, R., & Zhang, G. (2024). A Review of Intelligent Vehicle Trajectory Prediction Based on Deep Learning. Automotive Abstracts, 2024(02), 1-9. DOI: 10.19822/j.cnki.1671- 6329.20230085. [2] Fang, H. (2023). Research on Key Technologies of Vehicle Identification System Based on Deep Learning (Doctoral dissertation). Huazhong University of Science and Technology. DOI: 10.27157/d.cnki.ghzku.2021.005408. [3] Hu, Z. (2023). Research on Cross-Camera Vehicle Re- identification Technology Based on Deep Learning (Doctoral dissertation). Guizhou University. DOI: 10.27047/d.cnki.ggudu.2023.000071. [4] Cao, C. (2022). Research on Vehicle Object Detection Based on Deep Learning (Master's thesis). Nanjing University of Information Science & Technology. DOI: 10.27248/d.cnki.gnjqc.2021.000628. [5] Wang, Y. (2022). Design of Road Vehicle Detection System Based on Deep Learning (Master's thesis). Wuhan University. DOI: 10.27379/d.cnki.gwhdu.2022.000356.