Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 3, No. 3, 2023 102 Diagnostic Study of MBR Membrane Fouling based on CA‐ResNet18 Jingxiao Zeng * College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China * Corresponding author Email: zengjingxiao@outlook.com Abstract: In this paper, we propose a diagnostic model for addressing difficult-to-diagnose factors that influence membrane fouling, using a residual neural network (ResNet) optimized with a coordinate attention mechanism. After pre-processing fouling data from the membrane bioreactor using Principal Component Analysis (PCA) to derive nine categories of fouling factors, we determined the residual neural network structure and optimized it using the Coordinate Attention Mechanism (CA) to enhance feature extraction, improve diagnosis accuracy, and establish a stable and reliable diagnostic model for membrane scaling. Through experimental verification, the ResNet with the addition of the CA attention mechanism outperformed the ResNet with other attention mechanisms and the traditional ResNet in terms of prediction accuracy and convergence speed, achieving the research goal of accurately diagnosing the causes of membrane scaling. Keywords: Residual Neural Network; Coordinate Attention Mechanism; Membrane Fouling; Feature Fusion. 1. Introduction The membrane bioreactor (MBR) is an advanced wastewater treatment technology that integrates membrane separation technology with the activated sludge method, overcoming the limitations of traditional activated sludge treatment and elevating wastewater regeneration treatment to a new level [1,2]. Despite its benefits, the widespread use of MBR technology for wastewater treatment has been limited due to the inevitable generation of membrane fouling, which leads to high energy consumption and costs. Therefore, investigating the causes of MBR membrane fouling in practical applications has become a pressing issue. While early researchers used mathematical modeling to reduce the formation of the "filter cake layer" in membrane modules, membrane fouling is a complex and dynamic process that involves many parameters that are difficult to express as a precise function [3,4]. The advancement of computer technology has led to the enhancement and application of machine learning methods in the field of fault diagnosis, including the diagnosis of membrane scaling. Machine learning-based methods for membrane scaling diagnosis include support vector machines (SVM) and backpropagation (BP) neural networks, among others [5,6]. Despite their widespread use, shallow neural network models such as SVM and BP neural networks have limitations in fault diagnosis due to their shallow structure, complex parameter tuning, and limited computational power [7,8]. Additionally, the shallow neural network diagnostic approach relies on manual data pre- processing with substantial prior knowledge, and the models often lack generalization power for different diagnostic problems. To address the limitations of shallow neural network models, such as manual information processing and incomplete feature extraction from small sample data in complex environments, researchers have proposed convolutional neural networks (CNNs) [9,10]. CNNs offer a promising solution to these issues, providing improved feature extraction capabilities and enhanced generalization power for diverse diagnostic problems. Compared to shallow neural network models, CNNs use a different feature extraction mechanism that involves a small number of network parameters per layer, a proven anti-overfitting strategy, and an efficient convergence speed [11]. These characteristics not only provide a noise reduction and filtering effect but also guarantee the invariance of feature extraction, making CNNs a powerful tool for image and video analysis tasks. The signals monitored in the actual production process are non-smooth signals. The literature [12] applies wavelet transform to separate fault signals from their fluctuating signals, generates grey-scale time-frequency images, and then combines CNNs, thus reducing the false alarm rate of fault diagnosis. Research into building fault diagnosis models based on CNNs is beginning to bear fruit, but as the volume of data continues to expand, the required diagnostic accuracy continues to increase, and the network complexity of deep learning continues to strengthen, problems such as model degradation and overfitting can occur. In response to the shortcomings of CNNs, residual neural networks were created to further increase the depth of the network, but they are often accompanied by problems such as training difficulties when excessive network depth is greatly eliminated through short joins under degenerative phenomena [13]. To overcome the drawback that traditional convolutional operations can only extract single-scale feature information, the literature [14] superimposed convolutional layers of networks at different scales to extract multi-scale feature information, achieving effective fusion of multi-scale feature information within the residual block and taking into account the advantages of cross-layer constant mapping of residual natural networks and multi-scale feature extraction. Although this structure enhances the extraction of features in multiple dimensions, it also causes a large amount of data redundancy and creates an overfitting problem. The literature [15] uses the stacking idea of Inception to propose a stacked convolutional layer, which is inserted into the ResNet network and effectively extracts feature information from faulty data. With the continuous development of deep learning, a simpler and lighter attention mechanism has been proposed. The literature [16] enhances the representational power by introducing an attention 103 mechanism to weigh the feature maps after multiple convolutional operations. The improved attention mechanism enables the network to differentiate the importance of features, increasing the training speed and improving the recognition accuracy of the network. In this paper, we propose a strategy for membrane fouling fault diagnosis that uses an attention mechanism to enhance network performance. Our model combines a modified residual neural network with fault data visualization. Firstly, principal component analysis is used to filter out the main fouling factors, followed by the generation of grayscale images to reduce redundancy while retaining feature information. Additionally, we introduce an attention mechanism module to extract features effectively across multiple dimensions, reduce the number of training parameters in the network, improve the convergence rate, and enhance network performance. 2. Relevant Knowledge 2.1. Residual Natural Network The Residual natural network is a deep neural network structure, proposed by He et al. in 2015 [17]. ResNet adds a residual building block to the convolutional neural network, which theoretically allows the network depth to increase as infinitely as the hardware allows, without performance degradation. It solves the problem of gradient disappearance and gradient explosion during the training of deep neural networks through shortcut connection, allowing the network to be trained more deeply and more easily. The residual structure block is shown in Fig. 1. Input x Output H(x) Conv BN ReLU Conv BN ReLU F(x) x Fig 1. Residual block structure where x is the input to the residual block;  F x is the residual mapping function; and    H x F x x  is the mapping function. The size of the convolution kernel of the convolution layer in the residual block is 3 3 . 2.2. Coordinate Attention Mechanism The Coordinate Attention (CA) mechanism is designed to obtain detailed location information while maintaining channel information [18]. Unlike channel attention, which converts a feature tensor into a single feature vector via 2- dimensional global pooling, the CA mechanism decomposes channel attention into two 1-dimensional feature encodings and aggregates features along two spatial directions, with attention weights calculated based on the spatial location coordinates of features in the input image or feature map. This approach differs from common attentional mechanisms, such as the Squeeze-and-excitation (SE) network, which only considers internal channel information and ignores location information and spatial structure important in visual detection tasks [19]. SKNet (Selective Kernel Networks) requires a large amount of computation, which makes SKNet's computational resources more demanding than some other models. At the same time, the selective multi-scale feature fusion mechanism requires different weights to be applied to different feature maps, which makes SKNet sensitive to the learning rate, and if the learning rate is not set properly, the model will have difficulty converging or even not converging [20]. The specific structure of the CA module is illustrated in Fig. 2. Input Feature Map Coordinate Attention Result AvgPool w AvgPool h Sigmoid Contact Conv Layer Conv Layer h Conv Layer w Fig 2. Structure of the CA attention For the input features, the CA module first uses two 1- dimensional global pooling operations to aggregate features from two spatial directions, vertical and horizontal, to obtain a feature map in two different directions. Based on the height H and width W of the input feature map, an average pooling layer of dimensions  ,1H and  1,W is used to encode each channel along the vertical and horizontal coordinates, respectively, to obtain a feature map incorporating direction-specific information, with the expressions shown in equations (1) and (2).     0 1 ,h c c i w F h f h i W     (1)     0 1 ,w c c j h F w f j w H     (2) where cf denotes the input feature map and c denotes the channel of the input feature vector; h cF denotes the vertical output feature at height h ; w cF denotes the horizontal output feature at width w . Eqs. (1) and (2) aggregate features and generate a pair of positional feature mappings that can capture long-range dependency information along the horizontal space and retain accurate positional information in the vertical space, aiding the model in the precise positioning of the target region. The above two feature maps are then encoded into two attention maps using the coordinate attention generation transformation, for the input feature maps, the outputs correspond to Eqs. (3) and (4) respectively.    h h c h cg C F  (3)    w w c w cg C F  (4) where hC and wC represent 1 1 convolution operations in the vertical and horizontal directions, respectively;  is the sigmoid activation function;  is the nonlinear activation function; h cg and w cg are the horizontally and vertically generated attentional feature maps, respectively. The final output of the CA module combines h cg and w cg into a weight matrix, with Eq. (5).        , , h w c c cOut i j x i j g i g j   (5) The CA module applies simultaneous attention to the input 104 features in both horizontal and vertical directions to achieve precise positioning of the target area. 2.3. CA-ResNet18 Model Structure In this paper, the ResNet18 architecture was chosen to be combined with the CA attention mechanism module. In particular, the CA-ResNet18 diagnostic network structure is generally divided into three parts: the pre-processing part, the feature extraction part, and the output part. The pre- processing part consists of a convolutional layer with a convolutional kernel size of 3 3 , which is used to encode the input image. The feature extraction section of the CA- ResNet18 diagnostic network consists of a residual module, a CA module, a ReLU activation function layer, and a batch normalization (BN) layer. This section is designed to fully extract the location and effective features of the image while effectively suppressing gradient disappearance during backpropagation and alleviating overfitting. The output section includes a fully connected layer and a SoftMax layer. A global pooling layer is connected to the fully connected layer to integrate feature relationships and remove redundant information, improving network stability. The SoftMax layer output’s fault category labels and enhances network robustness. Since the CA-ResNet18 diagnostic network is built on the ResNet18 architecture, the parameter design is similar, and the parameters are shown in Table 1. The finalized CA-ResNet18 diagnostic model structure is illustrated in Figure 3. Table 1. Design of the model parameters Number of layers Name Category Outputs Number of channels 1 image input Image inputs 100×100 1 2 conv Pre-processing layer 62×62 64 4 Basicblock_1 Improved residual module 62×62 64 3 Basicblock_2 Improved residual module 62×62 64 5 Basicblock_3 Improved residual module 31×31 128 6 Basicblock_4 Improved residual module 31×31 128 8 Basicblock_5 Improved residual module 16×16 256 7 Basicblock_6 Improved residual module 16×16 256 9 Basicblock_7 Improved residual module 8×8 512 10 Basicblock_8 Improved residual module 8×8 512 17 GlobalAvgpool Global average pooling 8×8 512 18 fc_1 Fully connected layer 8×8 512 19 fc_2 Fully connected layer 8×8 9 20 softmax Softmax layer 1×1 9 21 class output Classified outputs — — Of these, the network involved a total number of 12,113,152 participants and a trained number of 12,105,344 participants. 3. Experimental Verification and Analysis 3.1. Membrane Fouling Data Acquisition and Definition The data used in this paper was derived from Seong-Hoon Yoon's spreadsheet model [21], which was used to verify the performance of the diagnostic model. PCA was used to determine the main factors of membrane fouling faults (shown in Fig. 3), and the analysis resulted in nine main types of membrane fouling faults (shown in Table 2). PCA CA- ResNet18 STR MLSS COD STR PHMLSS COD Fault Code … F/M ratio COD/MLSS F1 F2 F4 F5 F6 F3 F7 F8 F9 Produced water flow Produced water turbidity Conductivity Transmembrane pressure difference Temperature Silt load Aeration rate Aeration rate Temperature Fig 3. Schematic diagram of PCA principal component analysis In particular, the COD/MLSS ratio has an impact on the fouling of the entire MBR module, with anomalies in this indicator reflecting excessive suspended solids in the MBR membrane module. The growth of nitrifying and denitrifying bacteria in the pool is directly influenced by the water's temperature. Table 2. Membrane fouling mode of the membrane device Fault classification Normal range of indicators Code of membrane fouling COD/MLSS 1.1-1.2 F1 Temperature of water 15-35 ℃ F2 DO in aeration tank 1-3 mg/L F3 DO in membrane tank 4-8 mg/L F4 F/M ratio in anoxic tank 0.5-1.0 g BOD/g MLVSSꞏd F5 F/M ratio in aeration tank 0.08-0.24 g BOD/g MLVSSꞏd F6 MLSS in aeration tank 8-12 g/L F7 SRT 5-25 Day F8 Aeration demand 0.2-0.4 m3/(m2ꞏh) F9 These bacteria are both crucial for the water's ammonia and nitrate concentration, which would, in turn, impacts membrane flow and hydrophilicity/hydrophobicity. Low oxygen dissolution targets can lead to rapid increases in the 105 levels of soluble microbial products (SMP) and extracellular polymeric substances (EPS), which can affect water quality. A high or low F/M ratio will produce flocculation and therefore membrane fouling. Excessive SRT (Sludge retention time) can lead to the viscosity of the effluent being too high to affect the biological reaction and contribute to an increased rate of membrane fouling. High aeration suppresses the biological activity and increases the sludge concentration in the pool. Therefore, the above nine index anomalies were selected for analysis as a fault study. 3.2. Experimental Procedures There are six experimental steps in this project, the main steps are as follows: (1) Data collection: a collection of the required fault sample data, including data for normal and fault states; (2) Fault coding: the collected faults are coded and presented in the form of data markers to facilitate subsequent processing; (3) Data picturization: converting data into image form to facilitate training and testing of deep learning models. Feature extraction can be enhanced using image processing techniques; (4) Image set division: The data set is divided into the training set and the test set according to the ratio of 8:2. The training set is used for the training of the model and the test set is used for the performance evaluation of the model; (5) Model generation: Select the appropriate method to build and optimize the network. Training of the model using the training set and performance testing of the model using the test set; (6) Comparative analysis of results: The model is tested using a test set to compare the predicted and actual results of the model and to evaluate the performance of the model. Based on the evaluation results, the model is adjusted and optimized, and the optimal model is finally obtained. 3.3. Membrane Fouling Diagnostic Performance Analysis Table 3. Comparison of average accuracy Diagnostic models Average accuracy rate CA-CNN 61.63% ResNet18 91.58% SE-ResNet18 93.10% SKNet-ResNet18 96.10% CA-ResNet18 99.25% To evaluate the diagnostic accuracy of the proposed CA- ResNet18 model in membrane fouling fault diagnosis, a comparison experiment was conducted using the membrane fouling fault dataset. In addition to the CA-ResNet18 model, ResNet18, SKNet-ResNet18, SE-ResNet18, and CA-CNN models were also used for comparison. The experiment aimed to assess the performance of these models in accurately diagnosing membrane fouling faults. The results of the experiment were analyzed and compared to determine the best-performing model. Overall, the comparison experiment serves to validate the diagnostic accuracy of the CA- ResNet18 model and demonstrate its superiority in membrane fouling fault diagnosis. It is clear from the average accuracy that CA-ResNet18, compared to CA-CNN, has significantly improved the accuracy of diagnosis by suppressing overfitting while retaining the original neglected details through the Shortcut connection operation. When the attention mechanism was added to ResNet18, there was a significant improvement in diagnostic capability, with SE-ResNet18, SKNet-ResNet18, and CA-ResNet18 improving in accuracy by 1.52%, 4.52%, and 7.67%, respectively, compared to ResNet18. The CA module is superior in improving ResNet18 accuracy compared to ResNet18 with the addition of the SKNet, SE attention mechanism module, thus also demonstrating the applicability of adding coordinate features to membrane fouling diagnosis. Fig 4. CA-ResNet18 Confusion Matrix Heat Map The performance of the CA-ResNet18 diagnostic model was evaluated after 100 training sessions (shown in Figure 4). The results demonstrate that the model performs well on different fault classes and can achieve accurate classification, with high recognition scores in some of the fault classes. These findings demonstrate the stability and usefulness of the CA-ResNet18 model in diagnostic classification problems, particularly in the context of membrane fouling fault diagnosis. Overall, the results highlight the potential of this model for practical applications in the field of water treatment and related industries. Fig 5. Comparison of loss functions In the comparison of the loss functions (shown in Figure 5), the CA-CNN was unable to make an accurate judgment in the process of finding the best due to the network structure. Although SKNet-ResNet18 incorporates the attention 0 20 40 60 80 100 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 L os s F un ct io n Epoch CA-CNN ResNet18 SE-ResNet18 SKNet-ResNet18 CA-ResNet18 106 mechanism but performs poorly in the optimization problem, the loss function has a large oscillation amplitude in the early stage and converges slowly in the late stage, so this diagnostic network performs poorly in the stability of the model compared with SE-ResNet18 and CA-ResNet18, even less than the performance of the ResNet18 model. The CA- ResNet18 diagnostic model has a higher rate of change and good convergence compared with other models, can achieve fast optimization, and is more stable and less volatile than SE- ResNet18 and ResNet18 in the later stages of training, without falling into the local optimum problem, thus proving that CA-ResNet18 has good stability. 4. Conclusion The current membrane fouling diagnosis process relies on empirical methods and data analysis, which are both time- consuming and problematic, with complex features causing diagnostic faults. Therefore, this paper proposes a deep learning-based diagnostic strategy, CA-ResNet.18 The main conclusions of this paper are: 1. Resampling and visualization of fault data to enhance the data and improve its generalization. 2. Through the attention mechanism, the feature extraction channel is extended from one-dimensional to multi- dimensional to strengthen the network's ability to extract feature information. The results in the diagnostic performance experiments show that the fault diagnosis capability of the proposed model in this paper is superior to other network models (ResNet18, SKNet-ResNet18, SE-ResNet18, CA- CNN) with significantly higher accuracy, and thus the experimental results achieve the expected objectives. Thus, it is proved that the CA-ResNet18 proposed in this paper applies to the field of membrane fouling fault diagnosis. References [1] X.J. Du, Y.K. Shi, V. Jegatheesan, Ul Haq, 2020. A review on the mechanism, impacts and control methods of membrane fouling in MBR system. membranes. 10, 24. https:// doi.org/ 10.3390/ membranes10020024. [2] B. Li, C. Delpha, D. Diallo, A. Migan-Dubois, 2021. Application of artificial neural networks to photovoltaic fault detection and diagnosis: A review. Renew. Sust. Energ. Rev. 138, 110512. https://doi.org/10.1016/j.rser.2020.110512. [3] N. AlSawaftah, W. Abuwatfa, N. Darwish, G Husseini, 2021. A comprehensive review on membrane fouling: mathematical modelling, prediction, diagnosis, and mitigation. Water. 13, 1327. https://doi.org/10.3390/w13091327. [4] Y.K. Shi, Z.W. Wang, X.J. Du, B. Gong, V. Jegatheesan, Ul Haq, 2021. Recent advances in the prediction of fouling in membrane bioreactors. Membranes. 11, 381. https://doi.org/ 10.3390/ membranes11060381. [5] W.W. Li, C.Q. Li, T. Wang, Application of machine learning algorithms in MBR simulation under big data platform, Water Pract. Technol. 15 (2020) 1238-1247. https:// doi.org/ 10.2166/ wpt. 2020.095. [6] Y.K. Shi, Z.W. Wang, X.J. Du, G.B. Ling, W.C. Jia, Y.R. Lu, 2022. Research on the membrane fouling diagnosis of MBR membrane module based on ECA-CNN. J. Environ. Chem. Eng. 10, 107649. https://doi.org/10.1016/j.jece.2022.107649. [7] A. Amiruddin, H. Zabiri, S.A.A. Taqvi, L.D. Tufa. Neural network applications in fault diagnosis and detection: an overview of implementations in engineering-related systems, Neural Comput. Appl. 32 (2020) 447-472. https://doi. org/ 10. 1007/ s00521-018-3911-5. [8] L.X. Duan, M.Y. Xie, J.J. Wang, T.B. Bai. Deep learning enabled intelligent fault diagnosis: Overview and applications, J. Intell. Fuzzy Syst. 35(2018), 5771-5784. https:// doi.org/ 10.3233/ jifs-17938. [9] S. Manikandan, K. Duraivelu, Fault diagnosis of various rotating equipment using machine learning approaches - a review, P. I. Mech. Eng. E-J. Pro. 235 (2021) 629-642. https:// doi.org/ 10.1177/0954408920971976. [10] M. Mansouri, M. Trabelsi, H. Nounou, M. Nounou, Deep learning-based fault diagnosis of photovoltaic systems: a comprehensive review and enhancement prospects, IEEE Access. 9 (2021) 126286-126306. https:// doi.org/ 10.1109/ access. 2021.3110947. [11] J.L. Cui, Q.W. Zhong, S.B. Zheng, L.L. Peng, J. Wen, 2022. A lightweight model for bearing fault diagnosis based on gramian angular field and coordinate attention, Machines. 10, 282. https://doi.org/10.3390/machines10040282. [12] S. Guo, T. Yang, W. Gao, C. Zhang, 2018. A novel fault diagnosis method for rotating machinery based on a convolutional neural network. Sensors. 18, 1429. https:// doi.org/ 10.3390/s18051429. [13] M. Wei, Y. Liu, T. Zhang, Z. Wang, J.M. Zhu, 2021. Fault diagnosis of rotating machinery based on improved self- supervised learning method and very few labeled samples. Sensors. 22, 35009734. https://doi.org/10.3390/s22010192. [14] Y. Wang, J. Liang, X.G. Gu, D. Ling, H.W. Yu, Multi-scale attention mechanism residual neural network for fault diagnosis of rolling bearings, P. I. Mech. Eng. C-J. Mec. 236 (2022) 10615-10629. https://doi. org/ 10.1177/ 095440622211 04598. [15] Liu, Y., Li, J., Li, Q., & Wang, Q. Transfer learning with inception ResNet-based model for rolling bearing fault diagnosis, J. Adv. Mech. Des. Syst. 16 (2022) 1-19. https://doi. org/10. 1299/jamdsm.2022jamdsm0023. [16] H.Y. Zhong, Y. Lv, R. Yuan, D. Yang, Bearing fault diagnosis using transfer learning and self-attention ensemble lightweight convolutional neural network, Neurocomputing. 501 (2022) 765-777. https://doi.org/10.1016/j.neucom.2022.06.066. [17] F. He, T. Liu, D. Tao, Why ResNet works? Residuals generalize, IEEE T. Neur. Net. Lear. 31 (2020) 5349-5362. https:// doi.org/10.1109/TNNLS.2020.2966319. [18] C. Xie, H.Y. Zhu, Y.Q Fei. Deep coordinate attention network for single image super-resolution, IET. Image Process, 16 (2022), 273-284. https://doi.org/10.1049/ipr2.12364 [19] H. Wang, J.W. Xu, R.Q. Yan, R.X. Gao. A new intelligent bearing fault diagnosis method using SDP representation and SE-CNN, IEEE T. Instrum. Meas. 69 (2022) 2377-2389. https://doi.org/10.1109/tim.2019.2956332. [20] Z.Y. Cui, Y. Liu, W. Zhao, C. Wang. Learning to transfer attention in multi-level features for rotated ship detection, Neural Comput. Appl. 34 (2022) 19831-19844. https:// doi.org/ 10.1007/ s00521-022-07491-z. [21] S.H. Yoon, Membrane Bioreactor Processes (Advances in Water and Wastewater Transport and Treatment), 1st ed., CRC Press: Boca Raton, FL, USA, 2020.