Microsoft Word - 001.docx CHEMICAL ENGINEERING TRANSACTIONS VOL. 66, 2018 A publication of The Italian Association of Chemical Engineering Online at www.aidic.it/cet Guest Editors: Songying Zhao, Yougang Sun, Ye Zhou Copyright © 2018, AIDIC Servizi S.r.l. ISBN 978-88-95608-63-1; ISSN 2283-9216 Multi-target Location of Infrared Image of Power Equipment Qiufeng Fan School of Electronic Information & Electrical Engineering, Anyang Institute of Technology, Anyang 455000, China fqf_god@163.com Purpose: This study focuses on multi-target positioning in infrared images of power equipment. Methods: The FAs T-Match algorithm and the complementation of visible light images and infrared images information to achieve multi-target positioning in infrared images of power equipment. Results: The multi-target positioning in infrared images proposed in this study can be used for infrared automatic diagnosis of power equipment faults, which has good effects. Conclusions: Multi-target positioning in infrared images of power equipment is very important, which can reduce the labor of manual image acquisition and further improve the efficiency of fault positioning of power equipment. 1. Introduction Infrared technology is to use corresponding settings to detect and convert the infrared radiation energy emitted by the power equipment into electrical signals that can be collected and processed by an electrical signal processing system, and to make them to form the surface thermal image of the electrical equipment. Due to the difference in fault part, fault position and fault severity, the power equipment with fault will also show different surface temperature rises and have different temperature distributions. Therefore, the temperature information of the running equipment can be collected through the infrared sensor so as to find the fault or hidden trouble, and assess its seriousness and locate its specific position. Under normal circumstances, the technicians mainly use the hand-held thermal imager to collect pictures, and then carry on the further analysis. In recent years, the application of infrared technology has been improved in the power system, replacing technicians with patrol robots, which may be equipped with charge coupled device (CCD) camera and an infrared thermal imager to inspect the power equipment, but the collected infrared images still need to be processed and analyzed manually. In order to further improve the application effect of infrared technology in power system and get rid of its dependence on manpower, this study deals with the multi-target positioning in infrared images of power equipment, and adopts the improved FAs T-Match algorithm and approximate affine transformation between infrared images and visible light images to first achieve the multi-target positioning in visible light images of power equipment and then the positioning in the infrared images of power equipment. 2. Literature review The images collected by the front end of the infrared image real-time processing platform often require pre- processing of the collected images due to problems of its own noise, background clutter and algorithm complexity. Through preprocessing, the signal-to-noise ratio of the image is greatly improved, which reduces the detection difficulty of the target to some extent. Target detection is the premise of tracking, and high- precision target detection can ensure reliable tracking of subsequent targets. Target detection methods are generally divided into single-frame detection and multi-frame detection. The single-frame detection algorithm detects the target based only on the gray level difference between the target and the background and other characteristic differences, and does not correlate the inter-frame information, which is convenient for hardware implementation. It has high execution efficiency. Scholars proposed a target detection algorithm adapted to environmental changes (Duanggate et al., 2013). This method can accurately detect the target without prior knowledge, but it cannot achieve accurate multi-target detection. For complex background images, a single-frame detection method is difficult to accurately DOI: 10.3303/CET1866144 Please cite this article as: Fan Q., 2018, Multi-target location of infrared image of power equipment, Chemical Engineering Transactions, 66, 859-864 DOI:10.3303/CET1866144 859 distinguish between the background and the target, which can easily lead to high missed detection rates and false alarm rates. Most infrared images in complex backgrounds use multi-frame detection methods. If the association of the inter-frame information is performed after detecting all the objects in one frame, it also belongs to multi-frame detection. According to the processing order of target detection and trajectory tracking, multi-frame detection can be divided into two categories, including a pre-tracking detection algorithm and a pre-detection tracking algorithm. The pre-tracking detection algorithm is mainly divided into two parts: single-frame target detection and multi- frame motion trajectory correlation. There are two kinds of single-frame target detection, which is based on some global features of the image and some local features to classify the target and background, so as to achieve the target detection. The detection methods based on global features mainly include: Chen et al.'s largest interclass variance method (Chen et al., 2013), maximum entropy estimation method, and Cheng et al.'s partial differential equation method (Cheng et al., 2015). Once all suspicious targets have been detected in a single frame, the real target needs to be extracted. Based on prior knowledge, the continuity of the speed and direction of the target in the adjacent frame and the consistency of the motion trajectory are judged. Pipeline filtering, Huff transform, classical likelihood ratio, and neural network algorithms are often used to confirm the true goal. However, the pre-tracking detection algorithm does not highlight the global features of the weakly small target but presents local maximum features. The pre-detection tracking algorithm first tracks all suspicious target motion trajectories and achieves multi- frame signal energy accumulation. According to the characteristics of the short-term movement of the target, the posterior probability of all suspicious trajectories is calculated, the trajectory of the real target is determined, and then the detection and judgment of the target are performed. Han et al. pointed out that typical pre-detection tracking algorithm algorithms include three-dimensional matching filtering, image flow method, and dynamic programming method (Han et al., 2014). It shows certain performance in terms of improving the signal-to-noise ratio of the target image and effectively extracting the target motion trajectory. In general, although the pre-detection tracking algorithm can well adapt to low SNR conditions, it must track all suspicious target trajectories. The amount of calculation is too large. At present, there is no good way to solve the real-time requirements. The pre-tracking detection algorithm has a simple structure. It is mainly divided into single-frame detection and multi-frame trajectory correlation. The real-time property is good. This method is used more often in practical projects. However, the current single-frame detection algorithm cannot adapt to strong background clutter interference images. Therefore, an effective preprocessing algorithm and a reliable single-frame detection algorithm are very important for improving the performance of the pre-tracking detection algorithm. For dynamic targets, positioning based on motion analysis and matching-based techniques are commonly used. Motion-based analysis refers to the method of determining the target area by the difference between the speed of the target and the background in the image sequence and completing the positioning. The match- based technique refers to a method of using a certain image in a target motion video as a template to perform a matching search on other hits to determine the target position (Lin et al., 2013). In matching-based positioning techniques, different matching criteria have different mathematical implications and computational costs. The commonly used matching criteria are the maximum correlation function, the minimum variance function, the maximum matching pixel statistics, and the minimum average absolute difference function. In the maritime, air control and modern defense, infrared weak multi-target tracking is an important technology. The main algorithms for tracking are: template matching, optical flow method, Kalman filtering method, and so on. Template matching method is a simple and easy target tracking method. As the size of the image and the size of the template increase, the required search space also increases dramatically, and the target tracking efficiency is greatly reduced. The optical flow method is based on the assumption that the background gray level does not change with time. When it encounters strong clutter, its tracking effect is unstable. The traditional Kalman filtering method can achieve satisfactory tracking results under linear conditions. However, in reality, most of the target tracking problems do not satisfy the linear condition. Therefore, many scholars have proposed improvements to the limitations of traditional Kalman filtering. It includes Shen et al.'s extended Kalman filter (Shen et al., 2013) and the volumetric Kalman filter. They all approximate the nonlinear system through certain methods. When the system has strong nonlinearity, the estimation error of these methods will be very large, and it is difficult to guarantee the tracking accuracy. At this stage, the nonlinear estimation theory has made a major breakthrough. Many researchers have randomly extracted the samples in the state space and assigned different weights to accumulate posterior probability distributions that approximate the current state. Particle filtering is one of the typical methods. In addition, there are many particle filter derived algorithms, such as Han et al.'s interactive multi-model particle filter (Han et al., 2016), and Korman et al.'s Spherical Particle Filtering (Korman et al., 2013). The tracking accuracy of various particle filter algorithms depends on the number of particles. The greater the number of 860 particles, the higher the precision. As a result, the computational workload and storage space requirements of the algorithm also increase, resulting in difficulty in real-time tracking. In addition, the problem of particle degradation needs to be further addressed. To sum up, previous studies have mainly studied the final methods of the target. However, it is difficult to ensure the target tracking accuracy and tracking real-time performance at the same time, and the target tracking algorithm with a lower algorithm operation amount is promoted. Therefore, based on the FPGA+DSP architecture, the real-time processing method of infrared image is designed and optimized for weak target detection and weak target tracking. 3. Methods FAs T-Match algorithm, proposed by Simon Korman et al. in 2013, can obtain the approximate global optimal position of the target in the image, with good anti-interference performance for changes in illumination and degradation of image quality (such as blur, noise, etc.). The commonly used template matching algorithms include gray correlation method and sequential similarity detection method. But when the template needs translation, rotation, scaling and other affine transformations, calculations will be increased immensely using these algorithms. The FAs T-Match algorithm solves the computational increase by properly discretizing affine transformation parameters and random algorithms. Assuming that the template and visible light images are I1 and I2, respectively, and the sizes of the images are n1n1 and n2n2 (the application of FAs T-Match algorithm is not only limited to square images, but square is used here for convenience of presentation), which are converted into gray-scale images, with a pixel value range of [01]. One of the basic ideas of the FAs T-Match algorithm is that when the image segment is continuously smooth, it will not be necessary to consider all affine transformations when the template is matched, and the computational amount can be reduced by reasonably discretizing the affine transformation parameters. 3.1 Improved FAs T-Match algorithm This study improves the original FAs T-Match algorithm for multi-target positioning in a manner from roughness to precision. Roughness means to first find out the approximate position of the target, and precision means to further search for the precise position based on the rough results. The improved FAs T-Match algorithm still adopts discrete two-dimensional affine transformation parameters and the random algorithm to reduce the computational amount. First, initialize the parameters tx, ty, r2, sx, sy, r1, δ and ε, and calculate the similarity measure dT dT (I1, I2) for each affine transformation using the random algorithm. Then set the threshold and keep all dT (I1, I2) and corresponding transformation parameters within the threshold range. The FAS T-Match algorithm continues to use the Branch-and-Bound Scheme to continue to optimize to obtain an approximate global optimal position. The difference of the proposed method lies in that the transformation matrix is obtained for the preserved transformation parameter, and the template can obtain the positions in the target image through the affine transformation matrix, which are clustered into k (k is the target number) classes using the K-means algorithm. The average value of each cluster can be obtained and framed by the smallest rectangular box. Taking the framed rectangle as a new target image, the original FAs T-Match algorithm is used to find the approximate global optimal region. Therefore, k new target images and k targets can be obtained, the position of the target in the new target image is restored to the original target image, and finally the position of multiple targets in the target image can be obtained. Figure 1: Infrared image target positioning of electric equipment based on FAs T-Match algorithm 3.2 Approximate affine transformation between infrared images and visible light images In this study, the experimental pictures are taken from the pictures collected by the hand-held thermal imager in the substation. It can be seen that the visible light camera and the infrared CCD are fixed in close proximity, which means that there is an approximate affine transformation between the infrared images and the visible 861 light images. In the experiment, 7 groups of infrared images and visible light images are selected at random. The image registration tool of the Matlab Image Processing Toolbox is used to select some corresponding points in the visible light images and infrared images for each group (Figure 2 shows one of the 7 groups) to obtain the approximate affine transformation matrix of the 7 groups. The target position in visible light images (such as the red parallelogram in Figure 3) can be obtained by the FAs T-Match algorithm. Then through 7 sets of different affine transformations, the position of the target in the infrared images can be found (as shown in Figure 4, each color represents a set of affine transformations). It can be seen from the Figure that although the approximate affine transformation parameters between different groups of visible light and infrared images are different, it will not affect the positioning of the target in the infrared images. For the substation patrol robot, the approximate affine transformation parameters between the infrared images and the visible light images taken at different times of each stop point are substantially the same (since the shooting position is relatively fixed). In the present experiment, affine transformation parameters between visible light images and infrared images are determined in advance by an image registration tool and then given as known parameters. The position of the target in the visible light images can be obtained through the improved FAs T-Match algorithm, and then the position of the target in the infrared images can be obtained according to the affine transformation parameters between the given infrared and the visible light images. Figure 2: Corresponding points in infrared images and visible light images Figure 3: Visible light images target Figure 4: Affine transformation results 4. Results and Discussions The main idea of multi-target positioning method in the infrared images of power equipment is to first perform multi-target positioning in the visible light images, and then affine transform the positioning result into the infrared images. Set the threshold in Step 4, save the affine transformation parameters that are smaller than the threshold, and then obtain the positions of the template after affine transformation in the target image (as shown in Figure 6(a)), which are clustered into k classes using K-means algorithm. In this experiment, there are 4 classes, namely N, A, B and C bushing, as shown in Figure 5 (b), (c), (d) and (e). In this study, the average value of a cluster is used as the criterion function of K-means algorithm. The template is an approximate parallelogram in the target image through affine transformation, so it only needs to focus on the coordinates of one point in clustering, or even only on the x-axis coordinates (or y-axis coordinates) of one point. The average value of each cluster is calculated and framed by the smallest rectangular box. In the experiment, the framed rectangles are taken as new target images after expanding10 pixels up and down and left and right. In the new target image, the approximate global optimum position is found by the FAs T-Match algorithm, then the obtained position is returned to the original target image, and finally the position of the target in the infrared images is obtained by the affine transformation parameters between the given infrared images and visible light images. Figure 67(a) shows the multi-target positioning in the visible light images (red box in the figure), and Figure 6(b) shows the multi-target positioning in the infrared images (green box in the figure). 862 Figure 5: Multi-target clustering based on K-means algorithm Figure 6: Multi-target positioning In the experiment, the minimum value of dT (I1, I2) is defined as min Dis, and the given threshold is n×min Dis (n is a constant greater than 1). The value of the threshold (that is, the value of n) has an important influence on the positioning of the target. If the threshold is too small, it may not be able to find all the targets; if the threshold is too large, other interference targets may be generated. The parameter setting in FAs T-Match algorithm affects its running time and accuracy. Where, δ determines the step lengths of the six parameters tx, ty, r2, sx, sy and r1.If its value is too small, the parameter network Nδ will be large (its size is proportional to 1/δ6(n2/n1)2), the running time of the program will be long, and the optimal dT(I1, I2) will be smaller, and vice versa. δ determines the estimated number of sampling points for dT(I1, I2) (its size is proportional to 1/2), if its value is too small, the number of sampling points will be large, the running time of the program will be long, and the credibility of the estimation will be higher, and vice versa. Table 1 shows the running time of the multi- target positioning method in this study under different parameters. Table 1: Running time of the proposed multi-target positioning method under different parameters parameter value 1 2 3 4 5 ε 0.15 0.15 0.15 0.10 0.20 δ 0.25 0.20 0.30 0.25 0.25 t/s 43.9613 213.7969 19.5333 96.7507 25.2506 The multi-target positioning in the infrared images of power equipment presented in this study can be used for automatic infrared fault diagnosis of power equipment. As shown in Figure 7, through this method, 4 bushings of the transformer can be obtained and the highest temperature values in each bushing region can be obtained according to the actual temperature matrix (corresponding to the infrared images), which are 42.9°C, 44.0°C, 45.2°C and 47.3°C, respectively. Taking the lowest temperature as the reference temperature, it is possible to diagnose the common defects in the rightmost bushing according to the temperature difference and the judgment criterion, and then the technicians will be reminded to observe the development of the defects, take the blackout opportunity to overhaul and to arrange the test overhaul to eliminate the defects in a planned way. It is worth noting that, when the method is used for automatic diagnosis of patrol robot, the parameters and threshold values of multi-target positioning in the infrared images of power equipment can be set reasonably according to the specific conditions of each stop point. 863 Figure 7: Infrared diagnosis of electrical equipment fault 5. Conclusions According to the present study, it can be known that the traditional FAs T-Match algorithm template cannot match with infrared images of the power equipment, because the power equipment infrared images itself is a kind of pseudo image with sparse property, while the FAs T-Match algorithm can locate multiple targets of segmented smooth images, but there is only one approximate global optimal result. Therefore, an improved FAs T-Match algorithm is used in this study to achieve multi-target positioning in visible light images and then multi-target positioning in infrared images is performed according to the affine changes between infrared images and visible light images. It can be seen from the test results that the proposed method has a certain effect and a very broad application prospect, which has a positive effect on the further application of infrared images in fault detection of power equipment and improvement of the automation degree of application, providing a reference for the related research. The multi-target positioning in infrared images of power equipment has broad application prospects. In the future researches, the accuracy of positioning shall be further improved, and the existing research results shall be optimized. Acknowledgement [Fund Project] Anyang Institute of Technology 2016 annual school research fund: "YJJ2016001"Research on Temperature Control System of Electrothermal Melting Ice for Wind Generator’s Rotor Blades. Subject Number: YJJ2016001. Project Leader: Fan Qiufeng Reference Chen C.L.P., Li H., Wei Y., Xia T., Tang Y.Y., 2013, A local contrast method for small infrared target detection, IEEE Transactions on Geoscience & Remote Sensing, 52(1), 574-581, DOI: 10.1109/TGRS.2013.2242477 Cheng M.M., Mitra N.J., Huang X., Torr P.H., Hu S.M., 2015, Global contrast based salient region detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3), 569-582, DOI: 10.1109/TPAMI.2014.2345401 Duanggate C., Uyyanonvara B., Makhanov S.S., Barman S., Williamson T., 2013, Object detection with feature stability over scale space, Journal of Visual Communication & Image Representation, 22(4), 345- 352, DOI: 10.1016/j.jvcir.2011.02.004 Han J., Ma Y., Huang J., Mei X., Ma J., 2016, An infrared small target detecting algorithm based on human visual system, IEEE Geoscience & Remote Sensing Letters, 13(3), 452-456, DOI: 10.1109/LGRS.2016.2519144 Han J., Ma Y., Zhou B., Fan F., Liang K., Fang Y., 2014, A robust infrared small target detection algorithm based on human visual system, IEEE Geoscience & Remote Sensing Letters, 11(12), 2168-2172, DOI: 10.1109/LGRS.2014.2323236 Korman S., Reichman D., Tsur G., Avidan S., 2013, Fast-match: fast affine template matching, International Journal of Computer Vision, 121(1), 1-15, DOI: 10.1109/CVPR.2013.302 Lin C.M., Hsueh C.S., 2013, Adaptive ekf-cmac-based multisensor data fusion for maneuvering target, IEEE Transactions on Instrumentation & Measurement, 62(7), 2058-2066, DOI: 10.1109/TIM.2013.2247712 Shen X., Fan J., Wu Y., 2013, Scribble tracker: a matting-based approach for robust tracking, IEEE Transactions on Pattern Analysis & Machine Intelligence, 34(8), 1633-1644, DOI: 10.1109/TPAMI.2011.257 864