Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 7, No. 1, 2023 131 Recognition and Extraction of Power Transmission Lines Based on Infrared Image Processing for Line‐ following Robots Wenli Zhang, Qingfeng Gao Key Laboratory of Power Station Energy Transfer, Conversion and System, Ministry of Education, North China, Beijing 102206, China Abstract: To further improve the real-time performance and accuracy of power transmission line maintenance, this paper primarily focuses on the preliminary line recognition and extraction method based on thermal image processing of infrared images collected by line-following robots for thermal fault detection. Firstly, filtering and noise reduction techniques along with enhanced image processing are applied to preprocess the collected infrared images. This effectively addresses the noise and interference from background objects, which can affect the extraction of overheated areas on the lines, while also reducing the computational memory required for subsequent image processing. Subsequently, an improved Canny edge detection algorithm is employed to extract the edges of foreground objects in the images. Additionally, a region-growing algorithm combined with the geometric features of the lines is employed to filter out unwanted thermal sources, enabling the accurate segmentation and extraction of power transmission lines. This forms a solid foundation for subsequent detection and identification of abnormal hotspots on the extracted lines, and holds significance for the inspection and maintenance of existing thermal faults and potential hotspots in power transmission lines. Keywords: Line-following robot, Infrared image, Grayscale transformation, Canny edge detection, Region growing algorithm. 1. Introduction With the continuous increase in the proportion of volatile clean energy connected to the grid, the safety hazards and failure rates of high-voltage transmission lines have been somewhat increased, posing serious challenges to line inspection work. Line-following robots capable of autonomous control, such as walking and image acquisition, can be employed for inspection tasks. The adoption of infrared thermal imaging cameras on line-following robots for detecting faults and safety hazards in power transmission lines has gradually become the mainstream method. Accurate and rapid identification of abnormal hotspots in power transmission lines from the collected infrared images is of crucial significance for realizing intelligent inspection with line-following robots. The main fault diagnosis can be roughly divided into two categories. The first category involves leveraging powerful artificial intelligence techniques. Wang Xuhong et al. [1] improved the traditional SSD framework to achieve automatic detection of abnormal power equipment infrared images through sufficient image data training by various inspection robots, drones, and other intelligent power inspection devices. Reference [2] addressed the real-time issue of deep learning networks and designed a one- dimensional convolutional neural network for fault classification of power transmission lines. Liu Chuanyang et al. [3] comprehensively discussed the research progress of deep learning-based detection methods for power transmission line inspection images collected by drones over the past decade. References [4-5] also utilized powerful neural network models in deep learning for direct fault diagnosis and identification training of infrared images. The major limitation of this method lies in its dependence on the quality and quantity of training data. The second category involves traditional image processing methods. Reference [6] applied particle filtering algorithm to remove three types of common shape noise and differentiated the background and target foreground of power transmission lines based on threshold detection of infrared image grayscale, identifying overhead ground wires and detecting faults directly. However, most current research algorithms do not consider the significant interference caused by complex background environments in the images captured by line- following robots. Sha Weijiao et al. [7] preprocessed aerial images through histogram equalization and image filtering to address the interference of lighting intensity and background on the extraction of power transmission line elements. They employed the LSD algorithm for line edge extraction and connected the extracted line segments using Hough transform to achieve line extraction. Additionally, Reference [8] also employed the LSD line detection method to extract lines from infrared images. There are also numerous studies that adopt the Canny edge detection for line edge extraction [9-10]. It can be observed that most of the existing research using the second method is aimed at aerial images for recognition and detection. However, it is still meaningful to investigate the line detection and inspection algorithms based on improved image processing methods to exploit their unique advantages in terms of higher real-time performance and accuracy when combined with line-following robots. In view of this, this paper proposes the following power transmission line detection algorithm based on infrared image processing for line images obtained by inspection robots. The algorithm mainly consists of the following three parts: image preprocessing based on grayscale features, target edge recognition based on improved Canny edge detection, and interference source filtering based on line features. The overall flowchart of the algorithm is illustrated in Figure 1. 132 Figure 1. Flow chart of power transmission lines thermal fault detection algorithm 2. Image Preprocessing 2.1. Image filtering and denoising When collecting images, line-following robots inevitably introduce salt-and-pepper noise, which refers to black and white dots in the image caused by factors such as infrared sensor degradation, signal line interference, and sensor failures, as well as Gaussian noise caused by electromagnetic interference and thermal noise [9]. The presence of these noises can interfere with and affect subsequent thermal fault detection based on infrared image processing, thereby reducing the accuracy and reliability of the algorithm. In order to extract thermal fault information from power transmission lines and remove potential noise and interference in the image, filtering and denoising techniques are employed. Filtering and denoising can effectively reduce random noise in the image, smooth out image details, and improve the contrast and recognizability of the lines and abnormal overheated areas. This enables more accurate and reliable thermal fault detection algorithms, leading to reliable detection and early warnings of potential thermal faults in power transmission lines. Applying filtering and denoising techniques to the image can improve image quality and enhance the accuracy of hotspot detection, thereby enhancing the system’s sensitivity and visualization of thermal faults, and providing support for the prevention and maintenance of thermal faults in power transmission lines. The effects of using traditional median filtering and Gaussian filtering are demonstrated in Figure 2. Figure 2. Image after noise reduction processing Meanwhile, it is also important to optimize the stability of equipment and signal transmission during the image acquisition and transmission process, in order to minimize the introduction of noise interference. This can effectively enhance the clarity and reliability of infrared images, thus improving the accuracy and reliability of thermal fault detection in power transmission lines. 2.2. Improved Image Enhancement Infrared thermographic images are typically multi-band images, where each pixel contains both thermal and color information. However, in line extraction and detection of abnormal hotspots, our main focus is on the temperature differences in different regions of the image rather than color information. Therefore, converting the image into a grayscale image can retain the temperature information while reducing the computational complexity introduced by color information. Simplifying the processing and reducing the computation, the emphasis is placed on the analysis and processing of temperature information to achieve more effective line extraction and detection of abnormal hotspots. Nonlinear enhancement transformation plays an important role in the detection of thermal faults in power transmission lines based on infrared image processing with line-following robots. By directly filtering out the noise at specific grayscale levels to reduce the interference of background noise in subsequent processing and analysis, the visual effect of the infrared image can be improved. Simultaneously, by analyzing the local features of the target lines in the image, the grayscale levels of the pixels can be adjusted to enhance the contrast of the power transmission line targets in the infrared image, thus improving their visibility and enabling more accurate and reliable detection of thermal faults. The specific workflow is shown in Figure 3. 133 Figure 3. Image Enhancement Process According to the process shown in Figure 3, preprocess the infrared image. The original image and the results are shown in Figure 4 (a) (b) (c) Figure 4. Image after image enhancement processing Traditional image enhancement methods mainly include histogram equalization, which adjusts the grayscale distribution of the image to make the brightness range more uniform, thus enhancing the contrast and visualization of the image [11]. As shown in Figure 4(b), histogram equalization can effectively enhance the contrast of the target lines in the scene, but the drawback is that it barely filters out the noise, and in some cases, it may even amplify the noise. Therefore, in this case, it is necessary to first analyze the grayscale features of the image and directly filter out the noise in the corresponding grayscale level layer. Then, histogram equalization is applied to achieve a more desirable result. 3. Line Edge Detection 3.1. Traditional Canny edge detection algorithm The traditional Canny edge detection algorithm is a classic and widely used computer vision algorithm that is commonly used to detect object edges in digital images. After a preliminary study of various edge detection algorithms, the Canny algorithm is selected as the preliminary edge detection algorithm for line extraction, due to its high accuracy, low error rate, single-edge response, and adjustable parameters [12], which can meet the requirements of preliminary edge detection for lines. Canny edge detection often combines different operators such as Sobel, Prewitt, Roberts, and LoG to jointly perform edge detection [13]. It mainly includes the following steps: first, the image is smoothed using a Gaussian filter; then, the Sobel, Prewitt, or Roberts operator is used to calculate the gradient magnitude and direction of the image; based on the gradient information, non-maximum suppression is performed to refine the edges and reduce noise; finally, non- edges are filtered out by setting high and low thresholds, and an edge linking algorithm is used to connect weak edges to form the final edge image. The LoG operator is commonly used for edge refinement, which involves applying a Gaussian filter to the image and then using the Laplacian operator to extract edges. Therefore, in the Canny edge detection algorithm, the gradient information can be calculated using operators such as Sobel, Prewitt, Roberts, and then the LoG operator is used for edge refinement and thinning, resulting in more accurate and clear edge detection results. The effects of the traditional algorithm are shown in Figure 5. Figure 5. Traditional methods for detecting effectiveness 134 3.2. Improved adaptive threshold adjustment In the scenario of thermal fault detection in power transmission lines using line-following robots and infrared image processing, the thermal faults of the power transmission lines captured in the infrared images typically appear as hotspots or temperature abnormal regions with significant differences from the surrounding environment. The brightness information of these hotspots can be influenced by various factors, such as uneven thermal radiation, background noise, and illumination changes. This leads to unstable brightness differences between the thermal fault regions and the surrounding environment, making it unsuitable to use fixed global thresholds for edge detection. Therefore, adaptive thresholding is employed here, which dynamically determines the appropriate threshold based on the local characteristics of each region. Adaptive thresholding can determine the threshold for each pixel based on local statistical information of the image, such as local mean or local variance. This approach can better preserve the details and edge features of the thermal fault regions, even under uneven lighting and noise influences, effectively differentiating them from the background. By combining adaptive thresholding with the traditional Canny edge detection algorithm, the interference of surrounding background impurities in the image can be filtered out, resulting in better detection and extraction of line edges, thereby improving the efficiency of subsequent thermal fault detection. Experimental verification demonstrates that this improved adaptive thresholding approach achieves satisfactory results for two typical types of edge interference noise, as shown in Figure 6. Scenario 1: Nonlinear interference edge Scenario 2: Linear interference edge Figure 6. Treatment effect Comparing with the traditional adaptive dual-threshold segmentation, it can be observed that the image line edge recognition quality is significantly improved after the improved adaptive thresholding proposed in this study. It also exhibits a remarkable ability to resist strong noise interference from line-shaped artifacts in the image background. 4. Regional Segmentation and Interference Source Filtering By combining the image characteristics of the line targets in the inspection images, the line targets are identified from all the image edge segments. The grayscale and geometric features of the transmission line targets are used as the main methods to recognize the transmission line targets and filter out noise and interference sources. 4.1. Target Line Area Segmentation By combining the line feature information, the line region segmentation is performed using the region-growing algorithm based on the aforementioned edge segment detection results. This is achieved by comparing and analyzing the inspection images of the power transmission lines obtained by the inspection robot. Due to the thermal effects of the current in the power transmission lines, they exhibit a relatively unique grayscale range in the infrared images. On the other hand, other noise or interference sources captured in the same field of view will have more obvious grayscale differences compared to the power transmission lines due to the distance from the infrared components and the actual temperature differences. Therefore, this grayscale feature can be used to denoise the image effectively, filtering out interference sources and noise with significant grayscale differences from the target lines. The specific effects are shown in Figure 7. 135 Figure 7. Two or more references Among all the merged edge line segments, we select those segments that have at least one endpoint within the region of interest, forming the candidate pool for line edges. Since the edges on both sides of the power transmission lines are approximately parallel, we calculate the corresponding slopes based on the coordinate values of the endpoint coordinates of the line segments in the image coordinate system. Then, we convert the slopes to obtain the corresponding inclinations and identify line segments as candidate power transmission lines if the inclination difference is very small and the vertical distance between the line segments is less than a certain pixel size. Finally, we calculate the length of the candidate power transmission line segments and verify the accuracy of the results based on the characteristic information of power transmission lines that traverse the image vertically, completing the final extraction of the power transmission line targets. 4.2. Image Background Interference Removal Based on Connected Domain Analysis In order to improve the accuracy of line region extraction using the region-growing algorithm and filter out other heat interference sources generated during the growth process, it is necessary to remove interference sources from the connected domains obtained by the algorithm. This can be achieved by setting filtering conditions based on the connected domain characteristics of power transmission lines to eliminate non-linear heat sources in the background. The binary image obtained after applying the region-growing algorithm results in white regions representing the connected domains. A search is performed on each white connected domain to analyze the connectivity. Here, interference removal based on geometric features similar to the line grayscale is applied. The infrared thermal imaging component carried by the inspection robot captures line images in a downward vertical direction. The lines in the captured image are either straight or approximately straight, and their minimum bounding rectangles have linear properties. In other words, the connected domains of target power transmission lines usually consist of one or multiple parallel line segments. Therefore, the ratio of the length to the width of a connected domain can be used to determine if it is a straight line segment. Generally, if this ratio is very large, it can be considered as a straight line segment. This method helps filter out interference sources with grayscale similar to the target power transmission lines. However, this filtering method may face different situations, as shown in Figure 8. (a) Scenario 1 (b) Scenario 2 Figure 8. Interference filtering effect Based on Figure 8, the interference removal method based on connected domain analysis has a good effect in filtering out interference with grayscale values similar to the line targets. It can effectively compensate for the shortcomings of line region segmentation based on grayscale features in images. Additionally, for the two different scenarios mentioned above, this method has a good effect in filtering out interference areas that do not overlap with the lines, while retaining the areas that overlap with the lines. Considering that these overlapping areas are mostly hardware components or foreign objects such as plastic bags, which are closely related to the operation of the power lines, it is necessary to retain them to avoid potential safety hazards. Therefore, it can be concluded that this interference removal method can achieve the desired objective effectively. 5. Conclusion In summary, this paper focuses on detecting, recognizing, 136 and segmenting power transmission lines in the infrared images captured by the inspection robot, providing a foundation for subsequent thermal fault diagnosis of power transmission lines based on infrared images. It has to a certain extent enhanced the level of intelligence in online inspection and maintenance tasks performed by the inspection robot. Through analysis and research, the following important conclusions have been drawn: 1) Considering the actual scenarios of capturing infrared images with the inspection robot, the grayscale features of the target lines in the captured images are distinct and differ significantly from most of the background noise. Therefore, it is recommended to first apply grayscale filtering and then perform histogram equalization. This will significantly improve the signal-to-noise ratio of the images and enhance the contrast between the lines and the background. 2) Canny edge detection is applied to the preprocessed images to extract the edges of the power transmission lines. To improve the traditional algorithm, interference edge suppression is incorporated before applying different edge detection operators, effectively filtering out linear interference edges with large slope differences. This enhances the anti-interference capability and accuracy of power transmission line edge extraction. 3) The region-growing algorithm is used to segment and extract the line region, and morphological analysis and selection are performed on the connected domains obtained from the growth process to filter out interference heat sources outside the power transmission lines. This reduces the workload of subsequent thermal fault diagnosis and improves the diagnostic accuracy. Overall, these conclusions contribute to the development of intelligent thermal fault diagnosis of power transmission lines based on infrared images and enhance the functionality of the inspection robot in online inspection and maintenance tasks. References [1] Wang, X., Li, H., Fan, S., et al. (2020). A method for automatic detection of abnormal infrared images of power equipment based on improved SSD. Proceedings of the Chinese Society for Electrical Engineering, 35(S1), 302-310. [2] Subhrajit Mitra, Rajarshi Mukhopadhyay,Paramita Chattopadhyay. PSO driven designing of robust and computation efficient 1D-CNN architecture for transmission line fault detection[J]. Expert Systems with Applications, 2022, 210: 118178. [3] Liu Chuanyang, Wu Yiquan. Research progress of vision detection methods based on deep learning for transmission lines[J/OL]. Proceedings of the CSEE:1-24[2023-07-30]. http://kns.cnki.net/kcms/detail/11.2107.TM.20220831.1053.0 02.html. [4] Vitor H. Ferreira, Rainer Zanghi, Márcio Z. Fortes, et al. Probabilistic transmission line fault diagnosis using autonomous neural models [J]. Electric Power Systems Research, 2020, 185: 106360. [5] Md. Sihab Uddin, Md. Zahid Hossain, Shahriar Rahman Fahim, et al. On the protection of power system: Transmission line fault analysis based on an optimal machine learning approach[J]. Energy Reports, 2022, 8: 10168-10182. [6] Liang Xing，Yan Jubin，Yin Lei. Fault identification of transmission lines based on infrared image[J]. Electrical Measurement & Instrumentation, 2019, 56(24): 99-103. [7] Sha Weiyan, He Ninghui, Ding Pei, et al. Research on transmission line extraction technology based on UAV image processing[J]. Electrical Measurement & Instrumentation, 2022,59(05):158-165. DOI:10.19753/j.issn1001- 1390.2022.05.021. [8] Wang Miao, Du Wei, Sun Hongbo, et al. Transmission Line Fault Diagnosis Method Based on Infrared Image Recognition [J]. Infrared Technology, 2017, 39(04): 383-386. [9] Wang Hao, Wu Gongping, Liu Zhongyun, et al. Power transmission line detection algorithm based on adaptive Canny edge detection [J]. Journal of Yangzhou University(Natural Science Edition), 2020, 23(06): 45-52. [10] Wan Lin, Wu Shijing, Xie Fuqi, et al. Monitoring system of transmission line strain clamps based on image processing [J]. Engineering Journal of Wuhan University, 2020, 53(12): 1106- 1111. [11] Liu Deshuai, Li Yongjun, Luo Debiao. Transmission Line Detection and Fault Location Method Based on Unmanned Aerial Vehicle UAV Technology [J]. Radio Engineering: 1-8. [12] Wang Hao, Wu Gongping, Liu Zhongyun, et al. Power transmission line detection algorithm based on adaptive Canny edge detection [J]. Journal of Yangzhou University(Natural Science Edition), 2020, 23(06): 45-52. [13] Junfeng Jing,Shenjuan Liu,Gang Wang, et al. Recent advances on image edge detection: A comprehensive review[J]. Neurocomputing, 2022, 503: 259-271. [14] Tremeau, A., & Borel, N. (1997). A region growing and merging algorithm to color segmentation. Pattern Recognition, 30(7), 1191–1203. https://doi.org/10.1016/S0031- 3203(96)00147-1. [15] P. Stefanidou-Voziki,N. Sapountzoglou,B. Raison, et al. A review of fault location and classification methods in distribution grids[J]. Electric Power Systems Research, 2022, 209: 108031.