Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 3, No. 2, 2023 1 Defect detection algorithm for image reconstruction based on the self-attentive mechanism Han Fu1,2 *, Jie Gong1,2 1 Department of Machinery, Wuhan Textile University, Wuhan, Hubei 430074, China. 2 Hubei Key Laboratory of Digital Textile Equipment, Wuhan Textile University, Wuhan, Hubei 430074, China. * Corresponding author: Han Fu (Email: 838043902@qq.com) Abstract: In this paper, we propose an image reconstruction method for defect detection, which introduces a self-attentive mechanism into the generative adversarial neural network to make it capable of extracting global features in response to the weak ability of the generative adversarial neural network to establish image remote dependencies. To address the problem that the adversarial loss function is not suitable for network training, a pixel-by-pixel loss function is introduced to constrain the image generated by the generator so that the output does not deviate too much from the true value during training. The experimental results show that the trained GAN network can effectively reconstruct the defective areas to separate the defects from the background. Keywords: Defect detection; Image reconstruction; Loss function; Self-attentive mechanism. 1. Introduction Due to various factors, it is inevitable that the produced textile will have some defects, but it is difficult to find the defect location because the textile area is large and the defect area is small. Traditional manual inspection is inefficient and has a high error rate, which cannot meet the demand for high- speed online inspection. Traditional defect detection algorithms [1]-[3] can detect fabric defects quickly and effectively with high accuracy, but they do not work well for fabrics with complex textures. Since AlexNet was proposed, the deep learning method based on Convolutional Neural Network (CNN) has gradually become the mainstream method in the field of defect detection, convolutional neural network can advance the deeper features of the image and can automatically learn the image features, which is more effective in the detection of defects in images. The current fabric defect detection method based on neural network can be divided into defect detection method based on objective detection algorithm and defect detection method based on image segmentation. Song et al [4] proposed a low- latency, low-power, and easily scalable automatic vision inspection system based on edge computing in order to improve the efficiency of fabric blemish detection, which uses the EfficientDet-D0 algorithm as the detection model, transposes the trained model to the NVIDIA Jetson TX2 edge device, and uses TensorRT to optimize the model. Qin et al[5] proposed the Hourglass-MRB network, which adds a new convolutional branch to the residual module to dynamically adjust the network perceptual field size. Yuan et al [6]improved the CenterNet algorithm by using ResNet-50 for feature extraction, soft non-maximum suppression (soft NMS) for expectation to improve detection accuracy, and expansion of the fabric image database by image cropping and rotation to improve the stability and anti-interference capability of the neural network model. Jing [7] et al. used k-means algorithm to modify the a priori framework of YOLOv3 to cluster the defect data, followed by image pyramids to obtain feature mapping at different scales and add detection layers to improve the network structure. The defect algorithm model based on objective detection has a high accuracy in fabric defect recognition and can recognize some defects with complex texture fabric, which has a great improvement in recognition accuracy, efficiency and range, but these methods have a high requirement for training samples and need a large number of defective samples to participate in training, which are difficult to collect in most cases. Defect detection method based on image segmentation is to segment the defects into striking forms to express the shape, location and size of the defects. The method mainly uses the image reconstruction method to reconstruct the defective image to restore the texture of the defective area to the normal fabric texture, and then the reconstructed image and the original fabric image for image segmentation operation to obtain the defective map containing the defect information. The two main methods of image reconstruction are using Auto-Encoder networks or generative adversarial neural networks (GAN). Auto-Encoder is mostly trained with unsupervised training, which means that they are trained using defect-free samples, so that a large number of defective samples do not have to be collected to train the network. Li et al [8]proposed a fault detection method based on superposition denoising automatic coding (SDAE) and improved the model to distinguish between defective and non-defective images by adding Fisher's criterion. Mei[9] proposed a multi-scale convolutional denoising Auto- Encoder network (MSCDAE) to reconstruct images, which uses a multi-scale network to train images with different resolutions, and then uses the reconstruction residuals to generate detection results, compared to traditional unsupervised algorithms, MSCDAE has a large improvement in the model evaluation metrics. Although the self-coding- based fault detection method requires only normal samples for network training, the network can only learn the features of normal samples. When reconstructing defects, the reconstructed image may retain too many defective features due to the similarity between the defects and the original image texture background, resulting in a decrease in detection accuracy; secondly, the self-coding loses more image information during the training process, which is difficult to 2 be used for defect detection of complex patterned fabrics. Compared with the Auto-Encoder, the GAN is capable of extracting image features, its competitive adversarial training method can be adapted to process images with complex patterns, and it can effectively reduce the number of training samples required[10], making it more suitable for processing fabric images with complex textures than objective detection algorithms that require a large number of samples. Hu et al[11] proposed a kind of unsupervised textile defect detection algorithm based on convolutional generative adversarial network (DCGAN), which uses DCGAN to reconstruct defect-free images based on defective sample images. In the paper, an investor (E) similar to an encoder is proposed, firstly, DCGAN is trained, and then the one-dimensional feature vector output from E is sent to the trained generator G to get the reconstructed the whole training process requires only defect-free normal samples, divides the image into blocks, decomposes the image-level reconstruction into block-level reconstruction, and optimizes the reconstructed image using normalization and error maps. Liu et al [12] proposed a GAN- based framework for fabric defect detection, which first trains a GAN to explore the conditional distribution of defects in different texture backgrounds, then, the generated defects are fused to new fabric-specific locations, and finally, a deep semantic segmentation network is trained to achieve fabric defect detection by the generated defect images. In this paper, we propose a method for detecting fabric defects based on image reconstruction with self-attentiveness mechanism, which focuses on reconstructing the fabric defect region into a texture similar to the original background of the fabric, reconstructing the fabric image to obtain a fabric image without defects, and then performing simple image segmentation on the reconstructed image and the original image to obtain a detection result containing the location and size information of the defects. 2. Defect detection method 2.1. Network Structure The network structure of the generator is shown in Figure 1. The generator adopts the "encoder-middle layer-decoder" structure, and the input and output network structures are roughly aligned considering that the inputs and outputs of the model are roughly the same. The network adopts the "Convolutional Layer-Batch Norm-Activation Function" block. The encoder block first uses the convolutional layer for down-sampling, and the activation function uses the Leaky ReLU function, which passes a self-attentive mechanism when the image size reaches 32×32, then passes a down- sampling to size 4×4, and finally passes a self-attentive mechanism. The decoder module activation function uses the ReLU function, which first uses a transposed convolutional layer for downward up-sampling, and then passes a self- attention mechanism when the image size reaches 32×32, and finally reverts to the original size by up-sampling. We added jump connections to the generator, following the general connection of "U-Net"[13]adding jump layers between layer i and n-i, where n is the total number of layers in the generator. Each jump connection simply connects the channel of layer i to layer n-i. For some non-periodic images of textured fabrics with complex textures but little difference between the input and output, a large amount of underlying information needs to be shared between the input and output, and by connecting in this way it is possible to make the network up-sampling layer transfer texture information directly to the down- sampling layer. CNN LeakyReLU CNN Batch CNN Skip 128×128×64 256×256×3 Input Output 64×64×128 32×32×256 16×16×512 8×8×512 4×4×512 128×128×64 4×4×512 8×8×512 16×16×512 32×32×256 64×64×128 256×256×3 LeakyReLU CNN Batch LeakyReLU CNN Batch LeakyReLU CNN Batch LeakyReLU CNN Batch LeakyReLU CNN Batch ReLU CNN Batch ReLU CNN Batch ReLU CNN Batch ReLU CNN Batch ReLU Self-Attention Self-Attention Self-Attention Figure 1. Generator network structure CNN LeakyReLU CNN Batch CNN Batch CNN Batch CNN Batch CNN Input Output S 1 S 1 128×128×64 256×256×3 64×64×128 32×32×256 16×16×512 15×15×512 14×14×1 LeakyReLU LeakyReLU LeakyReLU LeakyReLU Self-Attention Figure 2. Discriminator network structure Our discriminator is the "Patch GAN"[14]. The output value of the discriminator of the original GAN is just an evaluation of the image, while the "Patch GAN" is a fully convolutional form, where the image passes through the convolutional layer and is not fed into the fully connected layer or the activation function, but the input is mapped into an N × N matrix using the convolution. Each point in the N×N matrix represents a small area of the original image, so the "Patch GAN" can focus on more areas, and thus can focus on more subtle differences in the image. 2.2. Loss function In order to enable better training of the network, the loss 3 function is redesigned in this paper. The traditional pixel-by- pixel loss-averaged absolute value error (L1 loss function) [15] is used to constrain the images generated by the generator so that the output results do not deviate too much from the true value. Compared with the pixel-by-pixel loss-average squared loss function (L2 loss function), the L1 loss function is chosen as the pixel-by-pixel loss function of the network in this paper because of the advantages of fast convergence and gradient stability. the expression of the L1 function is as follows: 1 , , 1 1 1 W H GT pred i j i j i j L I I W H = = = −   (1) where, GTI and predI denote the real image and the generated image, respectively, and W and H denote the width and height of the image. GAN against loss can be expressed as follows: ~ ( ) ( ( )) 1 dataCG x p xL E D G x= −（ ） (2) ~ ( ) ~ ( )( ( ) 1) ( ( ( )) dataCD y p y x pdata xL E D y E D G x= − + (3) where LCG and LCG denotes the GAN generator and discriminator adversarial loss. So, the loss function of the generator is as follows: 1G CGL L L= + (4) The discriminator loss function is as follows: D CDL L= (5) Where λ is the weight parameter to balance the weight of the generator GAN adversarial loss function with L1. 3. Experiment 3.1. Data set of experiments The experimental data sets were obtained from two different texture defect databases D1 and D2 in the fabric defects database of WTU. D1 (plain tissue; weft density of 30/cm; warp density of 32/cm; line density of 18.23 tex; acquisition resolution of 0.089 mm/pixel) and D2 (plain tissue; weft density of 37.5/cm; line density of 41/cm; line density of 36.34 tex; acquisition resolution of 0.089 mm/pixel). (37.5 weft/cm; 41 dense/cm; 36.34 tex; 0.089 mm/pixel) and D2 database (plain; 37.5 weft/cm; 41 dense/cm; 36.34 tex; 0.089 mm/pixel). 3.2. Model Evaluation Metrics In this paper, three model metrics, Recall, Precision and F1-Measure, are used to evaluate the performance of the model. They are defined as follows: Re 100%P P P TP TP FN =  + (6) Pr 100%P P P TP TP FP =  + (7) 2 Pr Re 1 100% Pr Re F   =  + (8) where Re, Pr and F1 represent recall, precision and weighted summed average, respectively. TPp is the proportion of correctly segmented defect areas in the foreground, FPp is the proportion of incorrectly segmented defect areas in the background, and FNp indicates the proportion of undetected defect areas in the defect areas. F1 evaluates recall and precision. 3.3. Training details All tests were performed on a GPU-equipped computer with the configuration shown in Table 1. Table 1. Computer system configuration System Memory GPU CPU Framework Windows 10 64G NVIDIA GTX- 1080Ti Intel Core i7- 8700K @ 3.70GHz Pytorch, CUDA10.1, CUDNN7.6 In the experiments of this paper, we experiment the Adam optimizer for training, where β1=0.5 and β2=0.999. The initial learning rate is set to 0.0002, and the learning rate is gradually decayed to 0 in positive proportion to the last 200 epochs of training. 3.4. Experimental results Holes Oil Stains Thread (a)Original image (b)Detectio n result Figure 3. D1 sample detection results Holes Oil Stains Thread (a)Original image (b)Detectio n result Figure 4. D2 sample detection results From Figure 3 and Figure 4, it can be seen that the proposed method can effectively detect all kinds of defects on both D1 and D2 databases. From Table 2, we can see that the detection accuracy and recall rate of this method are good, and the F1- Measure reaches 87%. Table 2. Computer system configuration Indicators Recall Precision F1-Measure Value 0.78 0.99 0.87 4. Conclusion In this paper, we propose a defect detection algorithm for image reconstruction based on the self-attentive mechanism, which introduces the self-attentive mechanism into the Pix2pix network to make it capable of extracting global features, and then introduces the L1 loss function in the training to constrain the images generated by the generator so that the output results will not deviate too much from the true values. Through the experiments, it is proved that the 4 proposed defect detection method is well performed on the fabric defects database of WTU. Acknowledgment Project supported by the Natural Science Foundation of Digital Textile Equipment Key Laboratory Open Fund DTL2021006. References [1] Jia, L., et al. "Fabric defect inspection based on lattice segmentation and Gabor filtering. "Neurocomputing 238.2017:84-102. [2] Ismail, N., J. M. Zain, and T. Hai. "Fabric authenticity method using fast Fourier transformation detection." Electrical, Control and Computer Engineering (INECCE), 2011 International Conference on IEEE, 2011. [3] Cohen, F. S., and Z. Fan. "Automated inspection of textile fabrics using textural models." IEEE Transactions on Pattern Analysis and Machine Intelligence 13.8(1991):803-808. [4] Song, S., et al. "EfficientDet for fabric defect detection based on edge computing." Journal of Engineered Fibers and Fabrics 16.3(2021):1014-1026. [5] Qin Runtian, Li Yu, and Fan Yujie."Research on Fabric Defect Detection Based on Multi-branch Residual Network." Journal of Physics: Conference Series 1907.1(2021). [6] He Yuan, Huang Xin Yue,and Hock Tay Francis Eng."Fabric Defect Detection based on Improved Object as Point." International Journal of Computer Science and Information Technology 13.3(2021). [7] Jing, J., et al. "Fabric defect detection using the improved YOLOv3 model." Journal of Engineered Fibers and Fabrics 15.1(2020):155892502090826. [8] Li, Y, W. Zhao , and J. Pan . "Deformable Patterned Fabric Defect Detection With Fisher Criterion-Based Deep Learning." IEEE Transactions on Automation Science and Engineering (2017). [9] Mei, Hyz . "An Unsupervised-Learning-Based Approach for Automated Defect Inspection on Textured Surfaces." Fortschritte der Physik 67.6(2018). [10] Huang, R., et al. "Prior-guided GAN-based interactive airplane engine damage image augmentation method." Chinese Journal of Aeronautics 35.10(2022):11. [11] Guanghua Hu, et al. "Unsupervised fabric defect detection based on a deep convolutional generative adversarial network." Textile Research Journal 90.3-4(2020). [12] Liu Juhua, et al. "Multistage GAN for Fabric Defect Detection." IEEE transactions on image processing: a publication of the IEEE Signal Processing Society 29.(2019). [13] Ronneberger, Olaf, P. Fischer, and T. Brox. "U-Net: Convolutional Networks for Biomedical Image Segmentation." International Conference on Medical Image Computing and Computer-Assisted Intervention Springer International Publishing, 2015. [14] Phillip Isola, et al. "Image-to-Image Translation with Conditional Adversarial Networks." CoRR abs/1611.07004. (2016). [15] Zhao, H, et al. "Loss Functions for Image Restoration With Neural Networks." IEEE Transactions on Computational Imaging (2017).