Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 7, No. 2, 2024 34 A Sigma‐Pi‐Sigma Neural Network Model with Graph Regularity Term Qianru Huang *, Qingmei Dong, Yunlong Liu, Deqing Ji, Qinwei Fan Xi’an Polytechnic University, Xi’an, Shaanxi, 710600, China * Corresponding author: Qianru Huang Abstract: In recent years, Sigma-Pi-Sigma neural network (SPSNN) as a special kind of higher-order neural network has attracted wide attention for its fast convergence speed and good approximation ability. However, an inappropriate number of hidden layer neurons may also lead to model underfitting or overfitting, which affects the performance and generalization ability of the model. Therefore, we propose a Sigma-Pi-Sigma neural network with graph regularity by adding a graph regularity term to the network. The results show that the proposed algorithm performs well in terms of training accuracy, testing accuracy and efficiency. Keywords: Sigma-Pi-Sigma Neural Network; Graph Regularization; Squared Error. 1. Introduction In recent years, neural networks have become one of the focuses of today's science and technology field and have received widespread attention due to their powerful learning ability and wide application prospects. Sigma-Pi-Sigma neural network (SPSNN) is a kind of feed-forward neural network [1], which consists of different orders of Sigma-Pi neural networks (PSNs), and is a high-order neural network with a multiplicative unit and stronger nonlinear mapping ability, with its fast convergence speed and good approximation ability. It is a higher order neural network with product unit and stronger nonlinear mapping ability, which has its fast convergence speed and good approximation ability [2]. Due to its strong nonlinear mapping ability, this network has been widely used in many fields such as medical and industrial [3-4]. Generally, we use gradient descent method to train the network [5]. However, when we use the squared error sum function to train the SPSNN, the inappropriate number of hidden layer neurons and the magnitude of the weights may lead to underfitting or overfitting of the model, which affects the performance and generalization ability of the model. Therefore, the structural design of BP neural networks and the acquisition of optimal parameters remain a great challenge. For the time being, adding a regularization term to the error function is the most common way to solve this problem. The main common penalties we use to train networks are weight decay, weight elimination and approximate smoothing [6-8]. While the traditional conventional regularization term is very effective in sparsifying and preventing overfitting, it does not take into account the geometric structure of the original data [9]. Recently popular graph regularization terms perform well in this regard [10-11]. Therefore, in this paper, we consider introducing the graph regularization term in the SPSNN model to obtain the optimal parameters and the best network structure by applying the graph regularization term to the error function. The rest of the paper is organized as follows: In Section 2, the sigma-pi-sigma neural network and the batch gradient algorithm based on graph regularization are described in detail. Section 3 analyses the experimental results. Finally, Section 4 gives some conclusions of this paper. 2. The Proposed Approach (1) Sigma-Pi-Sigma neural network (SPSNN) In this section, we introduce the SPSNN model. P , N , Q , and 1 are the number of neurons in the input layer, 1 layer,  layer, and 2 layer, respectively (see Fig. 1). Let   1 , J P j j j X O R R    be the set of training samples, where jO is the desired output of input sample jX and J is the number of training samples. 1 1 1 1 Output Input  layer nW 0W Figure 1. Topology of an SPSNN. Let 1 2( , , , ) P i i i iPW w w w   be the weight vector between the input layer and the 1 layer at 1,2, ,i N  Let 0 01 02 0( , , , ) Q QW w w w   be the weight vector connecting the 2 layer to the  layer. 0 1( , , , )T T T T Q NP NW W W W    We write all the ownership values in a tightened form. Let j 0( )jy g W   as the real output, i.e., 35 0 1( , , , )T T T T Q NP NW W W W    . define :g R R to be the activation function. ( )E w  to be the conventional squared error function:   1 0 1 1 ( ) 2 ( ) J j j j J j j j E W y O g W           Where 1 ( ) ( ( ) ), , 1 j J 2 j jg r g r O r R     . At the same time, we take the following forms. 1 2 1 2 1 2( , , ) ( , , ) ( ( ), ( ), ( )) Q Q j j j j Q i i i i i i j j j i i i i i i g W X g W X g W X                             So, ~ ( )E w can be represented as follows: ~ 0 1 1 ( ) ( ( )) QJ j j q i j q i Q E W g w g w X         and we can get the partial derivative as follows: 0 ~ ' 0 1 ( ) ( ) J j j w j j E W g w      In addition, according to ( ) q j j q i i g W X    ,we get ( ) ( ) , ? 1 ? 0, ? 1 ? q j j j i i q q i n n q g W X X if q and n W if q or n                Then, in light of the above definitions and (II.4), we have ~ ' 0 1 ' 0 1 \ ( ) ( ) ( ) ( ) ( ) n jJ qj w j oq j q n n j J j j j oq i j q n i q n j i i Q E W g w w w g w w X g w X                              A Sigma-Pi-Sigma neural network with graph regularity terms (SPSNNGR) In this section, we describe our SPSNN model with graph regular terms, firstly we construct the weighted graph from the input data, secondly, we compute its Laplace matrix L based on the information given by the weighted graph, and finally we compute the graph regular terms based on the above information. In this paper, we use a Gaussian kernel to measure the similarity between any two instances xi and xj to generate a weighted graph G . Let 1{ , }j j J p jX O     be a given set of training samples, we have 2 2|| || / 0, ? i jx x ij e if i j W if otherwise      Where 2 2 1 1 J j j x J      . Therefore, the normalized Laplace matrix L of the weighted graph G is defined as L D W  where ii ij j i D W   . In order to make the outputs of hidden layers from the same class as close as possible and thus improve the network accuracy, we define the penalty term function R as 2 , 1 2 1 , 1 1 ( ( ) ( )) 2 ( ) ( ) ( ) ( ) ( ) ( ) i j ij i j i ii i j ij i i J T T J j T J R f x f x W f x D f x f x W Tr VDV Tr VWV Tr VLV              ( )Tr X in the above equation is the trace of the matrix X , ( )if x is a function of the mapping of sample points ix in high-dimensional space to low-dimensional, and V is the low-dimensional representation matrix. Therefore, the error function can be defined as ~ 0 ~ ( ) ( ) ( , ) 2 ( ) ( ( ) ( ) ) 2 n T n n E W E W R W W E W tr f W X Lf W X       For the sake of simpler calculations, we consider ~ ( ) ( ) (( ) ( ) ) 2 T n nE W E W tr W X L W X    So, we can have 0 0 ~ ' 0 1 ( ) ( ) ( ) J j j w w j j E W E W g w       ( ) ( ) ( ( ) ( '( ) ) ) 2 n n T w w n nE W E W tr f W X L f W X X     The initial weight 0W is given, and the iterative updates of the weights of the SPSNNGR algorithm are as follows: 1 1, 2 ,m m mW W W m     and ( )m m WW E W  . where  is the learning rate. 3. Simulation Results (1) Function Approximation Problems To illustrate the effectiveness of the SPSNNGR algorithm, the following is an example to compare the SPSNN algorithm and the SPSSNNGR algorithm with a nonlinear function that 36 is formulated as 0.2 (sin( ) 1.2)y x   For the function approximation problem, we used the network utilizing this structure to simulate all the algorithms(input 2P  , 1 layer 4N  ,  layer 16Q  and 2 layer). In this experiment, 151 input variables were selected in the interval [ 3,3] . The initial weight W is randomly selected in the interval [ 0.5,0.5] , the learning rate 0.01  and the penalty term coefficient 0.001  . The experimental termination condition is that the error is less than 51 10 or the number of iterations reaches 5000 times. Figure 2 shows the network approximation graph of SPSNNGR, and Figure 3 shows the error function graphs of SPSNN algorithm and SPSNNGR algorithm, it can be seen that the SPSNNGR algorithm has good approximation ability and the error of SPSNNGR algorithm decreases faster, which indicates that our proposed algorithm performs better. Figure 2. Approximation result of SPSNNGR Figure 3. Errors of SPSNN and SPSNNGR. (2) Real-World Classification Problems To further validate the algorithmic performance of SPSNNGR, we conducted numerical experiments on 4 real datasets, all selected from the UCI database. In this experiment, we compare the performance of the SPSNN algorithm and the SPSNNGER algorithm. We set the initial network architecture of SPSNN as(input 2P  , 1 layer 4N  ,  layer 16Q  and 2 layer), where the learning rate 0.001  and the penalty term coefficient 0.001  . In addition, our initial weights W are randomly generated from the interval, and we set the training to stop when the error is less than 51 10 or the number of iterations reach 5000 . We used a cross-validation approach that, 70% of each dataset is selected as the training set, and the rest is used as the test set, to obtain the average results of each algorithm under 10 simulation experiments, and to provide a detailed analysis and comparison of their performance. Table 1. Performance comparison of classification problems Data Set Accuracy Rate SPSNN SPSNNGR Tic Training 0.9584 0.9642 Testing 0.9061 0.9229 Heart Training 0.9670 0.9952 Testing 0.9187 0.9890 Wireless Training 0.8350 0.8486 Testing 0.8171 0.8404 Wholesale Training 0.8539 0.8571 Testing 0.8045 0.8271 The performance comparison results are shown in the Table 1, we can clearly see that for most of the datasets, the SPSNNGR algorithm has higher training and testing accuracy compared to the SPSNN algorithm, which indicates that the SPSNNGR algorithm is more generalized and stable. 4. Conclusion In his paper, a new Sigma-Pi-Sigma neural network model with graph regularity is established by introducing the graph regularity term, and the algorithm not only improves the generalization ability of the network, but also improves the convergence speed. In addition, we conducted numerical experiments to verify the effectiveness of the new algorithm, and it can be seen that, compared with the SPSNN algorithm, the new algorithm not only has good performance in the function approximation experiments, but also has higher accuracy and better generalization performance in the classification experiments References [1] C K Li. A Sigma-Pi-Sigma Neural Network (SPSNN)[J]. Neural Processing Letters, 2003(17): 1-19. [2] Q W Fan, Q Kang, J M Zurada. Convergence analysis for sigma-pi-sigma neural network based on some relaxed conditions[J]. Information Sciences, 2022(585): 70-88. [3] S N Arsla. A hybrid sigma-pi neural network for combined intuitionistic fuzzy time series prediction model[J]. Neural Comput and Applic,2022(34): 12895-12917. [4] R S Neville. Reuse of information in multi-layer sigma-pi neural networks[J]. Connection Science, 2006(18): 43-59. [5] G Welper. Universality of gradient descent neural network training[J]. Neural Networks, 2022(150): 259-273. [6] Y Liu, D K Yang, F Li. Smoothed regularizer learning for split- complex valued neuro-fuzzy algorithm for TSK system and its convergence results[J]. Journal of the Franklin Institute, 2018 (355): 6132-6151. [7] P May, E Zhou, C W Lee. A Comprehensive Evaluation of Weight Growth and Weight Elimination Methods Using the 37 Tangent Plane Algorithm[J]. International Journal of Advanced Computer Science and Applications, 2013(6):149-156. [8] E Geoffrey. Connectionist learning procedures[J]. Artificial Intelligence, 1989(40):185-234. [9] P L Zhai, S H Zhang. Learnable Graph-Regularization for Matrix Decomposition[J]. Association for Computing Machinery, 2023(17):1-20. [10] R Nassif, S Vlaski. A Regularization Framework for Learning Over Multitask Graphs[J]. in IEEE Signal Processing Letters, 2019(26):297-301. [11] Y T Fan, W Y Yang. A backpropagation learning algorithm with graph regularization for feedforward neural networks[J]. Information Sciences, 2022(607): 263-277.