Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 15, No. 1, 2025 198 A Review of Deep Learning‐Based Steel Surface Defect Detection Feiyu Chen1, Lei Fu1, *, Yingqian Zhang2, Jiaqi Li3, Qian Zhang1, Shihao Bi1 1School of Mechanical Engineering, Sichuan University of Science and Engineering, China 2School of Civil Engineering, Sichuan University of Science and Engineering, China 3School of Information Science, Beijing Forestry University, China * Corresponding author Abstract: With the rapid development of deep learning technologies, the detection of steel surface defects has made significant progress, especially in automated industrial production environments. This paper reviews the current state of research on steel surface defect detection, focusing on deep learning-based methods. We discuss traditional and modern techniques, including Convolutional Neural Networks (CNN), YOLO, and other object detection frameworks. We also highlight the challenges faced in this field, such as data dependency, small target detection, multi-scale recognition, and the need for model generalization and industrial deployment. The review emphasizes the necessity of overcoming the limitations of existing models, such as accuracy and efficiency, to achieve real-time, reliable defect detection in complex industrial settings. Keywords: Deep learning technologies; CNN, object detection; steel surface defect detection. 1. Introduction With the continuous advancement of industrial manufacturing, steel has become a critical structural material extensively used in construction, transportation, energy, and machinery manufacturing. The surface quality of steel directly influences the performance, safety, and service life of downstream products. Against the backdrop of rapid industrialization, China’s steel industry has grown steadily, maintaining the world’s leading production capacity. In 2021, China’s crude steel output reached 1.065 billion tonnes—56.7% of global production—a year‑on‑year increase of 7%. As steel output continues to rise, various application scenarios demand ever more stringent quality control, particularly with regard to surface integrity. However, during production, processing, and transportation, steel surfaces are susceptible to a variety of defects—such as scratches, indentations, oxide scale, and cracks—due to equipment wear, process instability, and environmental factors. These surface defects not only degrade appearance and functional performance but also pose potential structural safety hazards, leading to significant economic losses and safety incidents. Consequently, developing highly efficient, accurate, and automated techniques for steel surface defect detection has become an urgent challenge in the realm of smart manufacturing. Traditional methods for detecting steel surface defects primarily include manual visual inspection and classic machine‑vision techniques based on hand‑crafted image processing. Manual inspection relies on the experience of operators, resulting in low throughput, inconsistent accuracy, and difficulty meeting the real‑time requirements of high‑speed production lines. Conventional image‑processing approaches can achieve a degree of automation but typically lack robustness against complex texture backgrounds, diverse defect morphologies, and variable lighting conditions. They often require bespoke feature‑extraction and classification rules for each defect type, limiting both their generalizability and scalability. In recent years, rapid advances in artificial intelligence— particularly the widespread application of deep learning in image recognition and object detection—have opened new avenues for steel surface defect detection. Deep learning models can learn hierarchical feature representations directly from raw images in an end‑to‑end manner, offering superior feature‑extraction capabilities and greater generalization. They have demonstrated high‑precision identification of multiple defect types even in complex environments. Therefore, designing and developing deep‑learning‑based models for steel surface defect detection not only promises to overcome the performance bottlenecks of traditional methods but also to provide intelligent support for quality control in industrial production. 2. Types of Steel Surface Defects 2.1. Common Steel Surface Defects During steel production—spanning smelting, rolling, cooling, handling, and subsequent processing—various physical, chemical, and mechanical factors can induce a range of surface defects. These imperfections not only compromise the surface integrity but also adversely affect mechanical properties, corrosion resistance, and downstream manufacturability. Based on their origin and morphology, the most frequently encountered steel surface defects include scratches, cracks, oxide scale, pitting, and delamination. 2.1.1. Scratch Scratches are among the most prevalent linear defects on steel surfaces, appearing as slender grooves oriented along a particular direction. They typically arise from foreign‑object abrasion, mechanical contact, roll‑mark damage, or friction during transport and handling. Scratches disrupt surface continuity and degrade appearance; in severe cases, they act as stress concentrators that can precipitate premature material failure. 2.1.2. Crack Cracks are highly critical defects characterized by fine linear or irregular fissures on or just below the surface. Their 199 formation is closely linked to residual internal stresses, uneven cooling rates, compositional inhomogeneities, or thermal‑treatment anomalies. Cracks significantly undermine structural strength and are a primary cause of fatigue failure and catastrophic fracture. 2.1.3. Oxide Scale Oxide scale forms when hot‑rolled or heat‑treated steel reacts with atmospheric oxygen, creating a layer of oxides on the surface. This layer—typically gray‑black or bluish in color—varies in thickness. Although acid‑pickling or abrasive blasting can remove scale, uneven distribution or strong adhesion to the substrate can impair subsequent surface treatments such as coating, painting, or welding. 2.1.4. Pitting Pitting manifests as localized corrosion, producing dense or isolated small cavities resembling pinholes or etch spots. It often results from prolonged exposure to humid, saline, or acidic environments, or from corrosion of inclusions. While pits are small, they breach protective films and can expand corrosion, thereby reducing service life and aesthetic quality. 2.1.5. Delamination (Peeling) Delamination, or peeling, refers to the partial separation or blistering of metal layers from the steel surface, sometimes resulting in large areas of flaking that expose the underlying substrate. It is commonly caused by laminar defects during metallurgical processing, oxide inclusions, roll‑stick phenomena, or improper cooling. Delamination not only mars surface finish but also promotes crack initiation in stress‑concentrated regions, compromising structural integrity. 2.2. Steel Surface Defect Datasets In computer vision, datasets play an indispensable role: they not only provide the foundational data for model training and evaluation but also largely determine an algorithm’s accuracy and generalization capability. In the context of steel surface defect detection, having high‑quality and diverse datasets is especially critical for effectively training deep learning models and enhancing their performance. The NEU dataset, released by the College of Materials Science and Engineering at Northeastern University, comprises 1800 grayscale images at 200 × 200 pixels, covering six representative defect types: cracks, inclusions, patches, pitting, rolled‑in scale, and scratches. It is available in two variants: NEU‑CLS for classification tasks, which contains 300 unlabeled images per defect category; and NEU‑DET for object detection tasks, which includes bounding‑box annotations to assess both localization and classification performance. The following figure shows an example from the NEU dataset. Figure 1. Examples of NEU dataset The Severstal dataset, provided by Severstal Steel Corporation, contains four types of strip surface defects: holes, cracks, scale, and delamination. The dataset is divided into 12,568 training images and 5,506 test images, each with a resolution of 1600 × 256 pixels. Every image is annotated either at the pixel level or with bounding boxes, making the dataset suitable for both detection and segmentation tasks. It enables a comprehensive evaluation of a model’s ability to recognize surface defects of varying scales and forms on large steel strips. The following figure shows an example from the Severstal dataset. Figure 2. Examples of Severstal dataset 200 3. Research Status and Development Trends at Home and Abroad 3.1. Object Detection The fundamental principle of object detection algorithms is to learn feature representations of targets, extract features from input images, and ultimately determine the location and category of targets within the image. In the feature extraction stage, commonly used methods include handcrafted features and deep learning-based features. Handcrafted features such as Haar features[1] and SIFT features[2] rely on human prior knowledge[3], and perform well in simple scenarios. In contrast, deep learning-based features automatically learn target representations through training convolutional neural networks, offering better robustness[4] and generalization capability[5] in complex environments. Currently, the classic deep learning-based object detection methods are mainly divided into two categories: Two-stage[6] and One-stage[7] object detection methods. The Two-stage object detection method consists of two main steps. First, the algorithm generates a series of proposals after the image is input. Then, these proposals are classified through a convolutional neural network to output the final results. In the One-stage object detection method, the task is treated as a standard regression task, where the input image directly yields the predicted results. The detection process is illustrated in Figure 3. Input Backbone Neck Dense Prediction Spare Prediction One-Stage Detector Two-Stage Detector Figure 3. Flowchart of One-stage and Two-stage Detection Deep learning-based object detection algorithms have undergone significant development over time, leading to the emergence of many novel and efficient object detection algorithms, as well as optimized versions based on existing methods. The R-CNN series, as one of the representative algorithms of Two-stage detection, achieves object detection by generating candidate regions and classifying them. This series includes algorithms such as R-CNN, Fast R-CNN[8], Mask R-CNN[9], and Faster R-CNN[10]. The YOLO series, on the other hand, represents One-stage detection algorithms. By inputting the entire image into the network and predicting bounding boxes and categories, YOLO achieves real-time object detection capabilities. SSD (Single Shot MultiBox Detector)[11] is another One-stage algorithm known for its speed and high precision, making it suitable for object detection tasks across various scales. CenterNet[12], also a One-stage detection algorithm, predicts the center points of objects directly in the image, enabling fast and accurate detection, particularly excelling in detecting sparse or dense objects. In 2016, Dai Jifeng et al[13] from Tsinghua University proposed the R-FCN algorithm, which employs position- sensitive score maps to evaluate class probabilities. This approach significantly improves detection speed while maintaining high localization accuracy. In 2019, Zhao Qijie and Tao Sheng et al[14] from Peking University introduced the M2Det algorithm, a multi-scale, multi-feature object detection method that enhances detection accuracy and speed by incorporating a pyramid feature extractor and multi-stage detectors. In 2021, Yu Changqian and Xiao Bin et al[15] from Huazhong University of Science and Technology proposed the Lite-HRNet, a network that improves detection accuracy and stability through multi-branch parallel processing of features at different resolutions. With the continuous improvement in computational power and ongoing advancements in deep learning, object detection technology is undergoing rapid development and innovation, exhibiting explosive growth. 3.2. Current Research Status of Surface Defect Detection Surface defects[16] are among the most critical factors affecting the quality of mechanical components. Traditional manual visual inspection methods are prone to subjective bias and human experience, often resulting in significant errors and failing to meet current detection standards or the real-time requirements of automated production lines. As industrial automation continues to advance, conventional inspection methods have increasingly revealed their limitations—such as being labor-intensive, inefficient, and heavily reliant on manual judgment. These limitations make traditional techniques inadequate for the high-efficiency, high-precision, and high-stability demands of modern industrial production. With the advent of Industry 4.0[17], researchers have proposed various surface defect detection methods to meet the high demands of automated production lines in terms of detection accuracy and real-time performance. In particular, the rapid development of computer vision and deep learning technologies has offered new solutions for surface defect detection. Automated inspection methods based on image processing and machine learning have gradually become mainstream in industrial manufacturing and demonstrate great potential for widespread application. For instance, BaiT[18] utilized MobileNetV3 as the backbone of YOLOv4 to extract image features and applied depthwise separable convolution in the PANet layer of YOLOv4, achieving network lightweighting and enabling real-time inspection of railway surfaces. Similarly, Min Y[19] optimized the detection 201 of rail surface defects based on morphological processing and the fundamental principles of machine vision, using directional chain code tracking to extract defect features. 3.3. Research Status of Surface Defect Detection in Steel Materials Steel Materials surface defect detection plays a vital role in industrial manufacturing, as it directly impacts the structural integrity and quality of welded products. Traditional detection methods include penetrant testing, magnetic particle testing[20], and ultrasonic testing. While these techniques can achieve high accuracy in defect identification, they suffer from limitations in terms of detection speed, operational complexity, and dependency on operator expertise. For example, although penetrant testing is effective in identifying surface defects such as cracks and porosity, it requires the test surface to be clean, oil-free, oxide-free, and uncoated. Otherwise, the penetrant may not properly seep into the defect, leading to inaccurate detection results. Magnetic particle testing is commonly used for weld defect inspection, but it relies heavily on visual interpretation to determine the location and size of defects. As a result, the operator’s skill and experience have a significant impact on the outcome. Small or subtle defects may be easily overlooked, leading to false negatives or missed detections. To overcome these challenges, researchers have increasingly explored the application of computer vision and deep learning techniques in Steel Materials surface defect detection. These technologies offer the potential to enhance detection accuracy and efficiency, while minimizing human error and subjectivity. Currently, the basic approach to defect classification involves designing machine learning classifiers that learn the mapping function from input features to defect categories. Commonly used algorithms include the Back Propagation (BP) neural network[21], Support Vector Machine (SVM)[22], and Principal Component Analysis (PCA)[23]. Convolutional Neural Networks (CNNs)[24] are deep learning models capable of automatically extracting image features such as color, contours, and textures, thereby avoiding complex manual feature engineering. CNNs exhibit strong robustness in applications requiring distortion invariance. A typical CNN consists of convolutional layers, downsampling (pooling) layers, and fully connected layers[25]. Although deep learning techniques have demonstrated high accuracy and efficiency in surface defect detection for steel materials, most current research efforts focus on single detection methods or specific feature extraction strategies. These approaches still exhibit limitations in handling complex features and small targets. For instance, when detecting fine cracks or other small-scale defects, existing deep learning models may fail to effectively capture subtle features, resulting in suboptimal detection accuracy. In addition, the adaptability and robustness of current models under complex industrial environments require further improvement—especially in the presence of noise, varying lighting conditions, or background interference, where model performance often degrades significantly. Therefore, enhancing the performance of deep learning models in complex scenarios and improving their ability to identify small targets and intricate defects has become a critical challenge in the field of steel surface defect detection. Improving existing models to achieve higher detection accuracy and efficiency in real-world industrial applications is an urgent task in the advancement of steel defect detection technology. 4. Limitations 4.1. Data Dependency and Annotation Cost The performance of deep learning models in steel surface defect detection heavily relies on large-scale, high-quality labeled datasets. However, publicly available datasets such as NEU-DET and Severstal suffer from limitations in terms of sample size, defect types, and imaging conditions, making them insufficient to fully represent the complex and diverse defects encountered in real industrial scenarios. Moreover, data annotation often requires pixel-level precision by domain experts, which is time-consuming and labor-intensive, significantly increasing the cost of data acquisition. Although approaches such as transfer learning and semi-supervised learning have been proposed to alleviate the reliance on large labeled datasets, these methods still face practical challenges, and publicly available weakly or semi-labeled datasets remain scarce, thus limiting the generalization ability and practical usability of defect detection models. 4.2. Challenges in Small Object and Multi- Scale Detection Steel surface defects vary significantly in size and morphology. Particularly, tiny defects such as fine cracks and pitting occupy only small regions in the image and provide limited feature information, posing significant detection challenges. Conventional detectors such as Faster R-CNN, SSD, and YOLO often suffer from missed detections or inaccurate localization when handling small objects. Although multi-scale feature fusion techniques like Feature Pyramid Networks (FPN) have been introduced to enhance the detection performance for small objects, real-world industrial environments introduce additional complexity— such as background clutter, lighting variation, and noise interference—that demand more robust and optimized model structures and feature extraction strategies to achieve high- precision multi-scale defect detection. 4.3. Generalization and Deployment Challenges in Industrial Scenarios While deep learning models may demonstrate high detection accuracy in controlled laboratory settings, their generalization ability and adaptability in real industrial applications remain limited. Differences in image data distributions across production lines, equipment, and environmental conditions can lead to performance degradation. Additionally, deep models are often computationally intensive, making them difficult to deploy on-site where real-time performance and hardware resource constraints must be considered. Although techniques such as model lightweighting, pruning, and quantization have been developed to reduce computational complexity, achieving efficient and stable deployment in industrial settings while maintaining detection accuracy still requires further research and optimization. 5. Conclusion In summary, deep learning-based steel surface defect detection has made significant strides, evolving from traditional hand-crafted feature + classifier methods to end- to-end CNN-based detection and segmentation frameworks, 202 and further to innovations involving lightweight networks, cross-scale feature fusion, and semi-supervised learning. Despite these advancements, key challenges remain, including the scarcity of high-quality and diverse datasets, precise localization of small-scale defects, and generalization and real-time deployment in complex industrial environments. Looking forward, with the maturation of weakly supervised and self-supervised learning techniques, the integration of multi-source heterogeneous data, and the design of more efficient network architectures, steel surface defect detection is expected to achieve a better balance among accuracy, speed, and robustness. These advancements will play a crucial role in supporting intelligent manufacturing and quality control systems. References [1] Besnassi M, Neggaz N, Benyettou A. Face detection based on evolutionary Haar filter[J]. Pattern Analysis and Applications, 2020, 23: 309-330. [2] Zhou H, Yuan Y, Shi C. Object tracking using SIFT features and mean shift[J]. Computer vision and image understanding, 2009, 113(3): 345-352. [3] Dotan-Cohen D, Melkman A A, Kasif S. Hierarchical tree snipping: clustering guided by prior knowledge[J]. Bioinformatics, 2007, 23(24): 3335-3342. [4] Han H, Zhu J, Liao S, et al. Moving object detection revisited: Speed and robustness[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 25(6): 910-921. [5] Zhao Q, Principe J C. Support vector machines for SAR automatic target recognition[J]. IEEE Transactions on Aerospace and Electronic Systems, 2001, 37(2): 643-654. [6] Du L, Zhang R, Wang X. Overview of two-stage object detection algorithms[C]//Journal of Physics: Conference Series. IOP Publishing, 2020, 1544(1): 012033. [7] Chen K, Li J, Lin W, et al. Towards accurate one-stage object detection with ap-loss[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 5119-5127. [8] Jiang H, Learned-Miller E. Face detection with the faster R- CNN[C]//2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017). IEEE, 2017: 650-657. [9] He K, Gkioxari G, Dollár P, et al. Mask r-cnn[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2961-2969. [10] Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real- time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): 1137-1149. [11] Liu W, Anguelov D, Erhan D, et al. Ssd: Single shot multibox detector[C]//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14. Springer International Publishing, 2016: 21-37. [12] Shi T, Gong J, Hu J, et al. Feature-enhanced CenterNet for small object detection in remote sensing images[J]. Remote Sensing, 2022, 14(21): 5488. [13] Dai J, Li Y, He K. R-fcn: Object detection via region-based fully convolutional networks[J]. Advances in neural information processing systems, 2016:29. [14] Zhao Q, Sheng T, Wang Y, et al. M2det: A single-shot object detector based on multi-level feature pyramid network[C]//Proceedings of the AAAI conference on artificial intelligence. 2019, 33(01): 9259-9266. [15] Yu C, Xiao B, Gao C, et al. Lite-hrnet: A lightweight high- resolution network[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 10440-10450. [16] S Ravikumar, K Ramachandran, V Sugumaran. Machine learning approach for automated visual inspection of machine components [J]. Expert System swith Appli cations, 2011, 38(4): 3260-3266. [17] Ghobakhloo M. Industry 4.0, digitization, and opportunities for sustainability[J]. Journal of cleaner production, 2020, 252: 119869. [18] TANGBO B, JIALIN G, JIANWEI Y, et al. A Study on Railway Surface Defects Detection Based on Machine Vision[J]. Entropy, 2021,23(11):1437. [19] MIN Y,XIAO B,DANG J, et al. Real time detection system for rail surface defects based on machine vision[J].EURASIP Journal on Image and Video Processing,2018(1):1-11. [20] Eisenmann D J, Enyart D, Lo C, et al. Review of progress in magnetic particle inspection[C]//AIP conference proceedings. American Institute of Physics, 2014, 1581(1): 1505-1510. [21] Zhao Y. Research and application on BP neural network algorithm[C]//2015 International Industrial Informatics and Computer Engineering Conference. Atlantis Press, 2015: 1444- 1447. [22] Guido R, Ferrisi S, Lofaro D, et al. An overview on the advancements of support vector machine models in healthcare applications: a review[J]. Information, 2024, 15(4): 235. [23] Krzyśko M, Nijkamp P, Ratajczak W, et al. Spatio-temporal principal component analysis[J]. Spatial Economic Analysis, 2024, 19(1): 8-29. [24] Zhao X, Wang L, Zhang Y, et al. A review of convolutional neural networks in computer vision[J]. Artificial Intelligence Review, 2024, 57(4): 99. [25] Jiang W, Huang X, Lin W, et al. Power Angle Prediction with Embedded Physics Knowledge Network[C]//2024 3rd Asia Power and Electrical Technology Conference (APET). IEEE, 2024: 333-337.