Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 5, No. 3, 2023 172 PM2.5 Concentration Prediction Method Based on Temporal Attention Mechanism and CNN‐LSTM Zhuohui Zhou1, Xiaofang Liu1, *, Huan Yang1 1School of Computer Science and Engineering, Sichuan University of Science & Engineering, Sichuan 643000, China * Corresponding author: Liu Xiaofang (Email: lxf1969@163.com) Abstract: Accurately predicting PM2.5 concentration can effectively avoid the harm caused by heavy pollution weather to human health. In view of the non-linearity, time series characteristics, and the problem of large multi-step prediction errors in PM2.5 concentration data, a method combining Long Short-term Memory Network and Convolutional Neural Network with Time Pattern Attention mechanism (TPA-CNN-LSTM) is proposed. The method uses historical PM2.5 concentration data, historical meteorological data, and surrounding station data to predict the future 6-hour PM2.5 concentration of air quality monitoring stations. Firstly, CNN is used to obtain the spatial characteristics between multiple stations, secondly, LSTM is added after CNN to extract the temporal changes of non-linear data, and finally, to capture the key features of temporal information, Temporal Pattern Attention mechanism (TPA) is added. TPA can automatically adjust weights based on the input of each time step, and select the most relevant time step for prediction, thereby improving the accuracy of the model. An example analysis is conducted on the measured data of Beijing's air quality stations in 2018, and compared with other mainstream algorithms. The results show that the proposed model has higher prediction accuracy and performance. Keywords: PM2.5 concentration prediction, Separated by commas, Temporal pattern attention mechanism, CNN-LSTM. 1. Introduction The introduction should provide background information (including relevant references) and should indicate the purpose of the manuscript. Cite relevant work by others, including research outside your company. Place your work in perspective by referring to other research papers. Inclusion of statements at the end of the introduction regarding the organization of the manuscript can be helpful to the reader. PM2.5 refers to particulate matter with a diameter of equal to or less than 2.5 micrometers in ambient air. Due to their small size, these particles can penetrate deep into the respiratory system and pose a significant threat to human health. PM2.5 particles often adsorb carcinogenic substances, toxic metals, persistent organic pollutants, and other harmful materials, which can directly affect the lungs when they enter the human body, leading to heavy metal poisoning, increased cancer risk, reproductive harm, and other problems [1]. The fluctuations in PM2.5 concentration have varying degrees of impact on the respiratory system, crop growth, and tourism. Predicting future PM2.5 concentrations in advance can provide valuable health information for travelers and serve as a warning for relevant departments to take measures to improve air quality [2]. This research topic has positive significance for the long-term development of public health, the economy, and the ecological environment. At the same time, it presents significant challenges to the accuracy and stability of the models, making it a profound and meaningful research topic. Against the background of rapid development in machine learning, researchers have conducted many studies on air pollutant concentration prediction based on different algorithms and models. For example, Zhang et al. [3] used the PCA method for data feature extraction, which improved the prediction accuracy and reduced the model's time complexity. They input the extracted data into a BP neural network for prediction. Samal et al. [4] proposed the Multi-Time Convolutional Artificial Neural Network (MTCAN) model, which can perform feature learning and sequence modeling simultaneously, and use a large amount of past observation data for long-term prediction to minimize memory requirements and operating time. Liu et al. [5] optimized the BP neural network using a genetic algorithm and constructed a feature-based PM2.5 concentration prediction model. Li [6] proposed an AC-LSTM model composed of a one- dimensional convolutional neural network, long short-term memory network, and attention mechanism. This model not only uses air pollutant concentration but also adds PM2.5 concentration from neighboring air quality monitoring stations as prediction data. They used CNN and LSTM to extract the spatio-temporal correlation and interdependence of multi-variable time series data, and used attention mechanism to capture the importance of different feature states at different time steps in affecting future PM2.5 concentrations. Liu et al. [7] used historical air pollutant and meteorological data for a region to construct an LSTM prediction model and accurately predict the PM2.5 concentration for 1, 4, 8, and 12 hours in the future. The above studies have made certain improvements in predicting PM2.5 concentration, but these studies often only consider the relevant features of a single station itself or ignore the correlation between different time steps and the influence of meteorological factors on PM2.5 concentration. In fact, PM2.5 concentration is not only affected by local historical conditions but also by the transport of pollutants from surrounding areas and meteorological factors. Therefore, this paper proposes a new type of TPA-CNN-LSTM network, which uses historical pollutant concentrations, surrounding PM2.5 concentrations, and meteorological factors as prediction features. LSTM overcomes the problems of gradient explosion and vanishing in RNN and can extract the time features of long-term prediction. CNN can extract the spatiotemporal features in the input data and the relevant 173 features of PM2.5 concentration between different monitoring stations. The time pattern attention mechanism can capture periodic features in time series data and handle multiple time series inputs, improving the accuracy, adaptability, and practicality of the model. 2. Data and Method 2.1. Data Description The temporal and spatial variations of atmospheric particulate matter (PM) are influenced by multiple factors, including pollution sources, meteorological conditions, and others [14,15]. The variation of PM2.5 concentration is not only affected by the previous atmospheric state and PM2.5 concentration but also related to the PM2.5 concentration in adjacent areas [16,17]. Therefore, historical air pollution data and meteorological data need to be considered in the prediction model. In this study, we selected 35 environmental monitoring stations in Beijing and collected hourly historical air quality concentration and meteorological data from January 1, 2018, to December 31, 2018, with a total of 8674 sets of data. The statistical information of the specific data is shown in Table 1. The locations of the air quality monitoring stations are shown in Figure 1. The experimental data includes concentrations of PM10, PM2.5, SO2, NO2, O3, and CO. The detailed characteristics of the air quality monitoring stations are also listed in Table 1. The meteorological data includes indicators such as atmospheric pressure, temperature, wind speed, humidity, and weather. Data sources include the Beijing Environmental Protection Monitoring Center and the European Center for Medium-Range Weather Forecasting reanalysis data. For individual missing data in the used data, linear interpolation, pre-filling, or post-filling methods were used for filling, and then normalization was carried out. Table 1. Description of dataset data type data name Variable name unit type of variable Air quality data The concentration of PM2.5 PM2.5 μg/m3 numerical The concentration of PM10 PM10 μg/m3 numerical The concentration of NO2 NO2 μg/m3 numerical The concentration of CO CO μg/m3 numerical The concentration of O3 O3 μg/m3 numerical The concentration of SO2 SO2 μg/m3 numerical Climatological data weather Weather - Category (Sunny, Cloudy, Fog, Snow, Rain) temperature Temperature °C numerical pressure Pressure hPa numerical relative humidity Relative humidity % numerical wind speed wind_speed m/s numerical wind direction wind_direction - Category (None, Southwest, Unstable, Southeast, Southwest, Northwest) Figure 1. distribution of air quality monitoring stations in Beijing 2.2. TPA mechanism The proposed model utilizes one-dimensional CNN to learn the temporal pattern information of time series data, referred to as TPA (temporal pattern attention), as a local feature learning method in the network. The TPA mechanism 174 structure is shown in Figure 2. The leftmost arrow in Figure 2 represents the processing of variables, and each row represents a variable's time series data. The time pattern matrix , C i jH of the variable within the convolutional kernel range is obtained by convolution calculation. The scoring function calculates the score of the time pattern matrix and normalizes the score using the sigmoid function to obtain the attention weight  . The context vector vt is obtained by combining the time pattern matrix and attention weight. The context vector vt from the encoder and the hidden state h are concatenated and connected with the hidden state s from the decoder. The output prediction value is calculated using the output layer and softmax function. The following describes in detail the role of the TPA mechanism in the proposed model. Firstly, one-dimensional CNN is used for convolution calculation. k filters are set, and the kernel size is 1k. T represents the range covered by attention, usually set to T=w. The convolution is calculated along the row vector of the hidden state matrix H using the above kernel to extract the time pattern matrix , C i jH of the variable within the convolutional kernel range, as shown in Equation (1).  , , , 1 1 *C i j j T l i t l l C         H H (1) Figure.2 TPA Mechanism Structure Among them, , C i jH represents the temporal pattern matrix of the j-th filter for the attention mechanism of the i-th row in the interval [t- , t-1] through convolution calculation.  , 1i t l  H determines the scope of the convolution calculation. The convolution kernel  ,i T lC   performs convolution calculation on the data within this range. The learned temporal patterns are scored using the following formula:    , TC C i t i tf h H H W h (2) Where C iH represents the i-th row of the time mode matrix CH , th represents the hidden state of the encoder output, and W represents the weight parameter. The attention weights i are:   ,Ci i tsigmoid f  H h (3) Weighted sum each row of HCC to obtain the context vector tv : 1 m C t i i i   v H (4) Among them, n is the number of features of the input variable x. The output value of the TPA-CNN-LSTM model is obtained by adding the attention vector tv linearly mapped with the hidden state th of the time step t, as shown in equation (5).  1t h t v th y v   W W h W (5) In the formula, 1ty   represents the output value,  represents the predicted time length, and hW , hW , and vW represent the weight parameter matrices of the corresponding variables. 2.3. Convolutional neural network Convolutional Neural Network (CNN) have achieved excellent performance in the fields of computer vision and natural language processing, thanks to their powerful feature extraction and recognition capabilities. Typically, one- dimensional convolution is used for processing time data, two-dimensional convolution is used for spatial convolution in images, and three-dimensional convolution is used for spatial convolution in three-dimensional space. In this paper, we believe that the convolution kernel in CNN is a one- dimensional structure. By using CNN, multi-variable time series data such as meteorological data, air pollution data, and data from adjacent stations needed for PM2.5 prediction can be input through different channels to maximize information retention. CNN mainly consists of three modules: it is a feedforward neural network. Its basic structure consists of an input layer, a convolutional layer, a pooling layer, a fully connected layer (FC), and an output layer, as shown in Figure 3. 175 Stride Convolution  kernel Convolution Pooling ... ... Full connection Figure 3. Structure of one-dimensional convolutional neural network CNN has the ability to automatically learn data features and has characteristics such as local connectivity, weight sharing, pooling operations, and multi-layer structure. These features significantly reduce the complexity of the PM2.5 concentration prediction model and reduce overfitting through gradient descent optimization, thus improving generalization ability. The calculation formulas for the convolution layer, pooling layer, and fully connected layer are:  1i i i iH f H b  W (6)  1i iZ subsampling Z  (7)     0 ; ,i aY i L l H W  b (8) In the formula, iH represents a convolutional layer, iW represents a weight vector,  represents convolution operation, ib represents bias parameters, ab represents offset vector,  f  represents activation function, iZ represents upsampling layer, subsamping represents sampling process,  Y  represents output, L represents loss function, and il represents class labels. 2.4. LSTM model The LSTM controls the transmission state through a gate structure, divided into forgetting gates, selective memory gates, and output gates, as shown in Figure 4. Figure 4. LSTM network structure diagram The forgetting gate determines the extent to which the previous unit state is forgotten and can be expressed as:   1,t f t t fx   G W h b (9) In the formula: tG is the state matrix of the forget gate at time t; 1th is the state matrix of the previous time step's state unit; tx is the input value at time t;    is the activation function; fW is the weight matrix of the forget gate; fb is the constant parameter matrix of the forget gate. We choose a memory gate and an activation function to control the range of new information being added. By using the joint effect of the forget gate and the memory gate output, we update the unit state of this unit, which can be expressed as:   1,t s t t sx   i W h b (10) 176   1tanh ,t c t t cx  �C W h b (11) 1t t t t t  �C GC i C (12) where: ti , t �C , tC represent the state matrices of the input gate, the intermediate quantity input, and the tuple state at time t, tanh respectively; are hyperbolic tangent activation functions; sW , cW represent the weight matrices of the input gate and the intermediate quantity output, respectively; sb , cb are the memory gate constant parameter matrices. The output gate controls the degree to which the current unit is filtered and can be expressed as:  1,t o t t ox b    O W h (13)  tanht t th O C (14) Where: tO is the state matrix of the output gate at time t; oW is the weight matrix of the output gate; ob is the output gate constant parameter matrix. 3. TPA-CNN-LSTM model design 3.1. Model Parameter Design The parameters of the prediction model were set as follows: the number of convolutional layers was 1, the number of filters was 16, the convolutional kernel size was 5 5, the learning rate was 0.001, the epoch was 100, the batch size was 32, and the optimizer was Adam. the parameter selection method was the grid search method. 3.2. Selection of Evaluation Indicators The root mean square error (RMSE) (15) and mean absolute error (MAE) (16) were chosen as metrics to evaluate the prediction performance of the TPA-CNN-LSTM model at each site and to compare it with other models.  2 1 1 ˆ N i i i RMSE y y N    (15) 1 1 ˆ N i i i MAE y y N    (16) where, iy , ˆiy and N denote the observed, predicted and evaluated sample sizes for each station respectively. y is the average of the data from the N observation samples. RMSE and MAE are common evaluation metrics used in time series forecasting. The lower their values are, the better the performance of the model. 3.3. TPA-CNN-LSTM prediction model construction In order to overcome the limitation of CNN in capturing only local information, this paper proposes a model that combines TPA with CNN. The model first performs feature extraction on the input time series sample and then uses it as the input to the CNN convolutional layer. This combination can reduce the computational complexity of CNN and speed up the feature extraction process. After the CNN performs the second feature extraction, the output time series is processed for dimensionality reduction. Considering the complex model structure of LSTM, when inputting long time series, the model training time is long. However, using the time series extracted and dimensionality-reduced by TPA-CNN as the input to LSTM can improve the accuracy of LSTM prediction and reduce its operating time. The prediction process of the entire TPA-CNN-LSTM model is shown in Figure 5. Meteorological  data Air pollutant data Adjacent  monitoring station  data t1 t2 ... end t1 t2 ... end t1 t2 ... end Normalization TPA CNN Train Prediciton Feature  extraction LSTM Denormalization Output Figure 5. Flow chart of TPA-CNN-LSTM model The main steps of the PM2.5 concentration prediction model based on TPA-CNN-LSTM are as follows: Step 1: Correlation analysis. Analyze the factors affecting the PM2.5 concentration changes and determine the input sequence. Step 2: Data preprocessing. Handle missing values and outliers, and normalize the input time series. Step 3: Feature extraction. Use TPA-CNN to extract important features from the normalized time series. Step 4: PM2.5 concentration prediction. Input the TPA-CNN processed time series into the LSTM network for model training. Step 5: Output. Reverse normalize the predicted data and output the results. Through the above steps, the goal of improving prediction accuracy and computational efficiency is achieved. 177 4. Experimental Results and Analysis 4.1. Correlation Analysis This paper uses the Pearson correlation coefficient analysis method to analyze the correlation between various variables and PM2.5 concentration. The Pearson correlation coefficient is used to describe the correlation between two variables and is often expressed as an R2 score. 12 ( )( ) 1 1 i i X Y n X X Y Y R Score n i         (17) Where ( )i X X X   , X , and X are respectively the standardized scores, sample mean, and sample standard deviation of sample iX . The value of R ranges from [-1,1], and the closer its value is to 1 or -1, the stronger the correlation. The closer its value is to 0, the weaker the correlation. The PM2.5 concentration value of a region can be analyzed from two aspects: historical concentration values and current meteorological conditions. Historical concentration values will have a certain impact on the concentration value of the next moment, while other pollutants and meteorological factors will have a greater impact on the concentration value of the next moment. 4.1.1. Correlation Analysis of PM2.5 Concentrations with Other Pollutants Figure 6 analyzes the correlation between PM2.5 and other pollutants. It can be seen that PM10, NO2, CO, and SO2 are positively correlated with PM2.5, while O3 is negatively correlated with PM2.5. This indicates that the concentration of PM2.5 increases with the increase of PM10, NO2, CO, and SO2 concentrations, and decreases with the increase of O3 concentration. Figure 6. Correlation coefficient of PM2.5 with other pollutants 4.1.2. Correlation analysis between PM2.5 concentration and meteorological factors Figure 7 analyzes the correlation between PM2.5 and meteorological factors. It can be seen from the graph that there is only a certain correlation between PM2.5 and meteorological factors. However, the correlation between weather, humidity, wind speed, and PM2.5 is relatively strong, while the correlation between other meteorological factors and PM2.5 is weak. Among them, weather, humidity, and PM2.5 are positively correlated, while wind speed and PM2.5 are negatively correlated. Although the correlation between other factors and PM2.5 is small, there is still a certain correlation. Therefore, this paper will select these data for training. Figure 7. Correlation coefficient of PM2.5 with other pollutants 178 4.2. Model Comparison Experiment Results To validate the stronger learning capability and more accurate prediction results of the TPA-CNN-LSTM model proposed in this paper, the LSTM model and CNN-LSTM model were used to predict PM2.5 concentration on the same dataset, and the results of the three predictions were compared and analyzed, as shown in Figure 8. 0 100 200 300 400 500 0 200 P M 2 . 5 ( μ g / m 3 ) time/h True LSTM CNN-LSTM TPA-CNN-LSTM Figure 8. Overall Comparison of Results of Different Prediction Models According to Figure 8, the three prediction models have consistent trends with the actual values, with the TPA-CNN- LSTM model having predicted values that are closer to the real values, while the LSTM model has a larger deviation in its predicted values. CNN can perform network learning from the original input sequence, avoiding the accumulation of errors caused by manually extracting features. By adding the TPA network, important information can be prioritized and the input data of the LSTM network can be optimized. Therefore, the TPA-CNN-LSTM network can more effectively extract features of the time series of PM2.5 concentration and related influencing factors and improve the prediction accuracy. Table 2 and Table 3 summarize the evaluation indicators for the three prediction models. Although the prediction accuracy of these three models decreases over time, it is noteworthy that the TPA-CNN-LSTM model performs better in the prediction results for each hour, and its prediction accuracy is higher than that of the LSTM and CNN-LSTM models. This result is also confirmed at the average prediction level of 6 hours. Table 2. The performances of the different models in terms of root-mean-square error (RMSE) time/h RMSE/(μg/m3) LSTM CNN-LSTM TPA-CNN-LSTM t+1 23.83 21.77 20.65 t+2 28.34 27.40 23.51 t+3 36.51 32.26 27.34 t+4 38.82 36.18 30.97 t+5 41.62 38.53 33.62 t+6 44.18 40.72 37.72 average 35.55 32.81 28.97 Table 6. The performances of the different models in terms of mean absolute error(MAE) time/h MAE/(μg/m3) LSTM CNN-LSTM TPA-CNN-LSTM t+1 14.07 12.78 11.09 t+2 18.65 16.59 14.87 t+3 21.32 19.42 18.54 t+4 24.18 22.42 20.94 t+5 27.52 24.58 22.43 t+6 29.84 26.41 24.67 average 22.60 20.37 18.76 179 From the table, it can be seen that in the multi-hour prediction task, the TPA-CNN-LSTM model has the lowest MAE and RMSE. The predicted values of the TPA-CNN- LSTM model in the multi-scale prediction task are closer to the true values. In addition, in the one-hour PM2.5 prediction task in Figure 8, the R2 of the TPA-TCN-LSTM model is the highest. After adding the time pattern attention mechanism, the performance of TPA-CNN-LSTM in the multi-scale prediction task is better than that of LSTM and CNN-LSTM. The results indicate that the proposed TPA-CNN-LSTM model can effectively learn the spatiotemporal correlations of air pollutants and is suitable for related tasks of predicting PM2.5 concentration. 5. Conclusions and Future Work This paper proposes a CNN-LSTM model based on temporal pattern attention for predicting the concentration of PM2.5 over multiple hours. The model uses air quality data, meteorological data, and PM2.5 concentrations from neighboring monitoring stations as inputs to capture the spatiotemporal correlation and long-term dependency of PM2.5. The temporal pattern attention mechanism helps to capture the importance of different feature states and improve the prediction accuracy of the model. Experimental results show that the TPA-CNN-LSTM model performs well in multi-scale prediction tasks. The main conclusions of this paper are as follows: Analysis of air pollution data shows that PM2.5 concentrations have a strong spatiotemporal correlation. Due to the propagation of air flow, PM2.5 concentrations in the prediction range are easily affected by PM2.5 concentrations from neighboring monitoring stations. In addition, because PM2.5 stays in the air for a long time, past feature states also affect future PM2.5 concentrations. Furthermore, compared with a single model, an ensemble model combines the feature extraction capabilities of multiple models, resulting in more stable training results and stronger generalization ability. The TPA-CNN-LSTM prediction model can better capture the nonlinear relationship between each input variable and PM2.5 concentration than other models, and has higher prediction accuracy. Although the model achieved good prediction performance, there are still some limitations. The data collected in this study did not consider emissions from factories and vehicle exhaust, which are also important factors in the formation of air pollutants. Additionally, in the future, we will explore the use of this model for large-scale prediction of other air pollutants and incorporate satellite meteorological data into the input of the prediction model. 6. Funding Supported by The Innovation Fund of Postgraduate, Sichuan University of Science & Engineering under Grant Y2022184. References [1] Liang B, Li X, Ma K, et al. Pollution characteristics of metal pollutants in PM2. 5 and comparison of risk on human health in heating and non-heating seasons in Baoding, China[J]. Ecotoxicology and Environmental Safety, 2019, 170: 166-171. [2] KIOUMOUＲTZOGLOU M A，SCHWAＲTZ J，JAMES P， et al．PM2. 5 and mortality in 207 US cities modification by temperature and city characteristics[J]. Epidemiology, 2016, 27( 2) : 221-227． [3] ZHANG Y, GUO A, WU H, et al. Seasonal prediction of PM2. 5 based on the PCA-BP neural network[J]. JOURNAL OF NANJING FORESTRY UNIVERSITY, 2020, 44(5): 231-241. [4] Samal K, Babu K, Das S. Multi-directional temporal convolutional artificial neural network for PM2.5 forecasting with missing values: A deep learning approach[J]. Urban Climate, 2021, 36, 100800:1-10. [5] LIU Linbo, LIU Lilong, Li Junyu, et al. Predition of PM2. 5 mass concentration based on GA-BP neural network with water vapor[J]. Journal of Guilin University of Technology, 2019, 39(02): 420-426. [6] Li S , Xie G , Ren J , et al. Urban PM2.5 Concentration Prediction via Attention-Based CNN–LSTM[J]. Applied Sciences, 2020, 10(6),1953: 1-17. [7] Liu X, Liu Q, Zou Y, et al. A self-organizing LSTM-based approach to PM2. 5 forecast[C]//Cloud Computing and Security: 4th International Conference, ICCCS 2018, Haikou, China, June 8–10, 2018, Revised Selected Papers, Part IV 4. Springer International Publishing, 2018: 683-693. [8] Chaloulakou, A. Kassomenos, P. Spyrellis, N. Demokritou, P. Koutrakis, P. Measurements of PM10 and PM2.5 particle concentrations in Athens, Greece. Atmos. Environ. 2003, 37, 649–660. [9] Hussein, T. Karppinen, A. Kukkonen, J. Härkönen, J. Aalto, P.P. Hämeri, K.; Kerminen, V.M.Kulmala, M. Meteorological dependence of size-fractionated number concentrations of urban aerosol particles. Atmos. Environ. 2006, 40, 1427–1440. [10] Chen, J, Lu, J. Avise, J.C. DaMassa, J.A. Kleeman, M.J. Kaduwela, A.P. Seasonal modeling of PM2.5 in California’s San Joaquin Valley. Atmos. Environ. 2014, 92, 182–190.