Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 9, No. 1, 2024 150 Comprehensive Review of Backpropagation Neural Networks Mingfeng Li College of Mechanical Engineering, Xi'an Shiyou University, Xi’an, Shaanxi 710065, China Abstract: The Backpropagation Neural Network (BPNN) is a deep learning model inspired by the biological neural network. Introduced in the 1980s, the BPNN quickly became a focal point in neural network research due to its outstanding learning capability and adaptability. The network structure consists of input, hidden, and output layers, and it optimizes weights through the backpropagation algorithm, widely applied in image recognition, speech processing, natural language processing, and more. The mathematical model of neurons describes the relationship between input and output, and the training process involves adjusting weights and biases using optimization algorithms like gradient descent. In applications, BPNN excels in image recognition, speech processing, natural language processing, and financial forecasting. Researchers continuously experiment with optimization algorithms, including the Grey Wolf Algorithm, Genetic Algorithm, Particle Swarm Algorithm, Simulated Annealing Algorithm, as well as comprehensive strategies and improved gradient descent algorithms. In the future, with the ongoing development of deep learning, BPNN is poised to play a crucial role in tasks such as image recognition and speech processing. Keywords: Backpropagation Neural Network (BPNN); Deep learning; Network structure; Optimization algorithms. 1. Introduction As technology continues to evolve, the Backpropagation Neural Network (BPNN) is emerging as a key driver in the field of deep learning, playing a pivotal role in the development of artificial intelligence. Since its introduction in the early 1980s, BPNN has gained prominence in both research and practical applications, evolving from a single- layer network to a deep neural network with the rise of computational capabilities and the advent of deep learning. BPNN has demonstrated outstanding performance in various domains such as image processing, speech recognition, and natural language processing. Its flexibility and powerful fitting capabilities position it as a robust tool for addressing complex problems. Neurons, serving as the fundamental units of the network, weight the outputs of the previous layer and produce an output through activation functions. The network structure comprises input, hidden, and output layers, with collaborative efforts and the backpropagation algorithm adjusting weights to optimize the network for specific tasks. This article delves into various aspects of BPNN, including its mathematical model, network structure, feedforward and backpropagation algorithms, weight and bias updates, as well as training and optimization. The goal is to provide readers with a clear and in-depth understanding of BPNN's core elements and offer insights into its future development. In this era of information explosion, BPNN, with its learning capabilities and extensive application areas, is at the forefront of driving innovations in artificial intelligence. 2. Basic Concepts of Backpropagation (BP) Neural Networks 2.1. Neurons and Network Structure The BP neural network, initially proposed in the 1980s[1], draws inspiration from biological neural networks. Its superior learning ability and adaptability quickly made it a focal point in the field of neural network research. With the improvement of computing power and the rise of the deep learning trend, the BP neural network has evolved continuously. From the initial single-layer network to the emergence of deep neural networks, BP networks have made significant progress in various domains. Its powerful nonlinear modeling capability positions the BP neural network as a crucial element in machine learning. In numerous fields such as image processing, speech recognition, and natural language processing, BP neural networks demonstrate outstanding performance. The neurons in the network are the fundamental units that constitute the entire system. Each neuron receives the output from the neurons in the previous layer, performs a weighted sum through the assigned weights, and generates an output through an activation function. The entire BP neural network consists of multiple layers of neurons, including the input layer, hidden layers, and output layer. These layers work collaboratively, adjusting weights continuously through the backpropagation algorithm to optimize the network for specific tasks. With its profound research foundation and successful applications, the BP neural network has become a crucial pillar in the field of deep learning. In the future, as technology advances and application scenarios expand, the BP neural network will continue to play a key role in the field of machine learning. 2.1.1. Mathematical Model of Neurons The mathematical model of neurons describes the relationship between their inputs and outputs: 1 ( ) i i i n y f w x b      (1) As shown in Equation (1): ix , represents the input, iw corresponds to the weights, b is the bias, f is the activation function, and y is the output of the neuron. 151 2.1.2. Network Structure BP neural networks generally consist of three main layers[2]: 1.Input Layer: The input layer receives external input data and passes it to the next layer of the network. The neurons in this layer are responsible for receiving and transmitting the raw input information. 2.Hidden Layer: The hidden layer is a core component of the neural network, responsible for processing input data and extracting features. Each neuron in the hidden layer gradually adjusts its weights through the learning process, capturing patterns and correlations in the input data. The number of hidden layers and the activation function type for each neuron can be adjusted based on the specific task and network design. 3.Output Layer: The output layer generates the final output result of the neural network. Neurons in this layer integrate the features passed from the hidden layer, forming the network's overall understanding of the input data. The choice of the activation function in the output layer often depends on the nature of the problem; for example, the Sigmoid function may be used in binary classification problems, while the Softmax function might be employed in multi-class classification problems. The structural diagram is shown in Figure 1. X1 X2 h1 h2 . . . . . . . hq Hidden layer Output layerInput layer Y2 Xw . . . . . . . Y1 . . . . . . . Ye Figure 1. Schematic diagram of BP neural network structure 2.2. Feedforward and Backpropagation Algorithms Feedforward propagation is the process of information transmission in a neural network from the input layer to the output layer. For a neuron j in the l-th layer，the calculation formula for its output ( )l ja is given by: 1 ( ) ( ) ( 1) ( ) i l l l l j ij i j n a f w a b          (2) As shown in Equation (2): ( )l ijw is the weight connecting neuron I and j， ( 1)l ia  is the output of neuron i in the previous layer, ( )l jb is the bias of neuron j。 Backpropagation optimizes the network weights and biases by computing the gradient of the loss function with respect to the network parameters. The gradient formulas for weights and biases are given by: ( ) ( ) Δ l ij l ij E w w     (3) ( ) ( ) Δ l j l j E b b     (4) As shown in Equations (3) and (4): s the learning rate, and E represents the loss function. 2.3. Weight and Bias Updates The learning process involves adjusting weights and biases using optimization algorithms such as gradient descent, where the learning rate (\(\eta\)) determines the step size of parameter updates. ( ) ( ) ( )Δl l l ij ij ijw w w  (5) ( ) ( ) ( )Δl l l j j jb b b  (6) 2.4. Training and Optimization of BP Neural Networks In the training process of BP neural networks, the loss function is used to evaluate the difference between the model's output and the actual labels. Commonly used loss functions include Mean Squared Error (MSE) and cross- entropy loss. MSE is suitable for regression problems, while cross-entropy is typically used for classification problems. The formulas for Mean Squared Error (MSE) and cross- 152 entropy loss are as follows: 1 21 ( ) 2 ˆ i i i n E y y n     (7)   11 log( ) (1 ) log( ˆ )ˆ 1 i i i i i n E y y y y n        (8) As shown in Equations (7) and (8): n is the number of samples, iy is the actual label, and îy is the model's predicted output. Gradient descent is a common method for adjusting network parameters[3], but there are some variants that can improve performance. Among them, Stochastic Gradient Descent (SGD), Batch Gradient Descent, and Mini-Batch Gradient Descent are the three most common. 1. Stochastic Gradient Descent (SGD): In each iteration, SGD updates parameters using a single sample. This method has a lower computational cost, but the noise from individual samples may lead to unstable parameter updates. Despite this, SGD is often widely applied, especially on large datasets. 2. Batch Gradient Descent: Batch Gradient Descent updates parameters using the entire training set, calculating the average gradient. The advantage of this method is that the gradient calculation is relatively stable, but it comes with a higher computational cost, especially for large datasets. Batch Gradient Descent is typically used for small datasets or situations where computational resources are abundant. 3. Mini-Batch Gradient Descent: Mini-Batch Gradient Descent is a compromise between the above two methods, updating parameters using a small subset of samples in each iteration. This approach balances computational efficiency and relatively stable parameter updates, making it the preferred method for most deep learning tasks. Mini-Batch Gradient Descent often exhibits good convergence performance, especially in the case of large datasets and deep networks. The choice among these three variants of gradient descent depends on task requirements and dataset sizes in practical applications. In the field of deep learning, Mini-Batch Gradient Descent is a common and effective optimization method, often leading to good training results by combining the advantages of the previous two approaches. 2.5. Applications of BP Neural Networks BP neural networks find wide applications in various fields, including image recognition, speech processing, natural language processing, and more. Their flexibility and powerful fitting capabilities make them essential tools for solving complex problems. 3. Applications and Performance Optimization of BP Neural Networks 3.1. Image Recognition and Classification In the field of image recognition, BP neural networks have achieved significant success. The introduction of Convolutional Neural Networks (CNNs) enables the network to effectively capture spatial features in images, leading to excellent performance in tasks such as image classification and object detection. Typical applications in image recognition include face recognition, object recognition, and the hierarchical structure of deep neural networks automatically learns abstract features from images, thereby improving classification accuracy. 3.2. Speech Processing and Speech Recognition BP neural networks play a crucial role in the field of speech processing. The temporal nature and complexity of speech signals make traditional methods challenging, but BP neural networks, especially those with Long Short-Term Memory (LSTM) structures, can better capture the temporal information in speech. They are widely used in tasks such as speech recognition and speaker identification. Their successful applications have greatly advanced speech processing technology in areas such as intelligent assistants and voice search. In speech recognition tasks, BP neural networks can accurately identify words and speech features by learning the temporal patterns in speech signals. The LSTM structure enables the network to handle long-term dependencies in speech signals, effectively capturing contextual information in speech signals. For speaker identification, BP neural networks model the speech features of speakers and distinguish between different speakers through weight adjustments during the learning process. This is valuable in applications such as voice assistants, voice search, and secure authentication. These successful applications not only drive the development of speech processing technology but also provide robust support for the practical application of intelligent assistants. Through BP neural networks, advancements have been made in speech interaction technology, including speech command recognition and speech synthesis, enabling users to engage more naturally and conveniently with smart devices. 3.3. Natural Language Processing In the field of Natural Language Processing (NLP), BP neural networks have made significant progress. The introduction of structures like Recurrent Neural Networks (RNNs) enables networks to process text data more efficiently, achieving success in tasks such as sentiment analysis, text generation, and machine translation. The successful application of deep learning models in the NLP domain has significantly improved the efficiency and accuracy of automatically processing textual information. In sentiment analysis tasks, BP neural networks, by learning the semantic and emotional information in textual data, can accurately determine the sentiment tendency of the text. This provides a reliable solution for applications such as social media sentiment analysis and sentiment evaluation in product reviews. In text generation, BP neural networks, through learning the language patterns of large amounts of textual data, can generate text content with a certain semantic structure. This finds widespread applications in automatic text summarization and dialogue systems. In machine translation tasks, BP neural networks, by learning the correspondence between different languages, can achieve high-quality text translation. Their role in multilingual communication and international collaboration is crucial. These successful applications not only elevate the level of NLP technology but also provide robust support for the practical application of text information processing. Through 153 BP neural networks, the NLP field has made significant progress in sentiment analysis, text generation, and machine translation, laying the foundation for more intelligent and efficient text processing. 3.4. Sequence Data Analysis BP neural networks demonstrate distinct advantages in handling sequential data. In the financial sector, these networks find extensive applications in tasks like predicting stock prices and optimizing trading strategies. By learning from historical market data, BP neural networks can capture the changing trends in stock prices, providing robust support for investment decisions. In meteorology, BP neural networks enhance the accuracy of predicting future climate changes by assimilating temporal patterns from meteorological data. This application makes weather forecasts more reliable, aiding in addressing the challenges posed by climate change and offering crucial information for decision-makers. 3.5. Performance Evaluation and Optimization 3.5.1. Introduction of Optimization Algorithms To enhance the performance of BP neural networks, researchers have explored various optimization methods. In addition to improving network structures and adjusting hyperparameters, introducing different optimization algorithms has become a key means of improving performance. Some common optimization algorithms include: 3.5.1.1 Grey Wolf Algorithm Optimization The Grey Wolf Algorithm simulates the hunting behavior of grey wolves and is introduced into the optimization process of BP neural networks. This algorithm includes stages such as searching for prey, chasing and surrounding prey until the prey stops fleeing, and besieging prey. By applying the Grey Wolf Algorithm for optimization, the network can converge more effectively during training, improving learning efficiency. 3.5.1.2 Genetic Algorithm Optimization Genetic Algorithm[4] is an optimization algorithm that simulates the biological evolution process and is applied to optimize the parameters of BP neural networks. This algorithm optimizes the weights and biases of the network through operations such as selection, crossover, and mutation. Due to the advantage of genetic algorithms in global search, the neural network is more likely to find global optimal solutions. This approach helps improve the performance of neural networks, enabling faster convergence and better results during training. 3.5.1.3 Particle Swarm Algorithm Optimization The Particle Swarm Algorithm[5] simulates the collective behavior of birds or fish and is applied to parameter adjustment in BP neural networks. By simulating the flight and collective cooperation of particles, this algorithm can search for the optimal solution in parameter space. The advantage of the Particle Swarm Algorithm lies in its balance between local search and global search, making it easier for BP neural networks to converge to good solutions. This way, the network can more effectively adjust parameters during training, enhancing performance. 3.5.1.4 Simulated Annealing Algorithm Optimization The Simulated Annealing Algorithm simulates the physical principles of metal annealing, gradually searching for the global optimal solution in the solution space as the temperature decreases. Introducing the Simulated Annealing Algorithm into the training process of BP neural networks helps the network escape from local optima and better converge to global optimal solutions. This method, by simulating the gradual cooling process of annealing, makes the network more flexible in searching parameter space, increasing the probability of finding global optimal solutions and enhancing the training effectiveness of the neural network. 3.5.2. Comprehensive Optimization Strategies Researchers have also attempted to combine multiple optimization algorithms to form comprehensive optimization strategies. By integrating Genetic Algorithm, Particle Swarm Algorithm, and Simulated Annealing Algorithm, for example, researchers can fully leverage their respective strengths, improving the efficiency of parameter search. This comprehensive strategy allows for a more comprehensive exploration of parameter space, effectively enhancing the performance of BP neural networks. Through the combination of multiple algorithms, researchers can more flexibly address different network structures and types of problems, further optimizing the training process of neural networks for better performance. 3.5.2.1 Improved Gradient Descent Algorithms Gradient Descent is a commonly used optimization algorithm in deep learning. However, it has some issues, such as slow convergence and susceptibility to local optima. To address these problems, researchers have introduced methods with adaptive learning rates, such as Adagrad and Adam. These algorithms dynamically adjust the learning rate to adapt more flexibly to different parameter update situations during training, improving the convergence speed and stability of the algorithm. This adaptive learning rate strategy makes Gradient Descent more suitable for deep learning tasks, accelerating the model training process and reducing the risk of getting stuck in local optima. 3.5.2.2 Reinforcement Learning Algorithms Reinforcement learning algorithms are introduced to optimize the parameters of BP neural networks. By establishing an interactive model between the network and the environment, reinforcement learning algorithms can continuously optimize the network's parameters through trial and error, improving its performance. This approach is often used to handle complex tasks and enhance the network's generalization ability. Through reinforcement learning, the network can adjust its parameters based on environmental feedback, gradually learning and optimizing its behavior strategy. This learning method helps the network make more flexible and intelligent decisions when facing unknown environments or complex tasks, enhancing the adaptability and performance of BP neural networks. 4. Advances and Future Prospects in Deep Learning In recent years, significant progress has been made in the field of deep learning, including innovative activation functions, regularization methods, multimodal fusion, and cross-domain applications. These advancements not only enhance the performance of deep learning models but also expand their application domains. Firstly, in terms of activation functions, traditional ReLU has been widely adopted, but the introduction of new- generation activation functions such as Leaky ReLU and ELU has enhanced the nonlinear expressive power of neural networks, effectively mitigating the vanishing gradient 154 problem. These innovations provide more stable and efficient solutions for the training of deep learning models. Secondly, in regularization methods, the evolution of L1 and L2 regularization, Dropout technique, and Batch Normalization improves the model's generalization ability, accelerates convergence speed, and enhances robustness to initial weights. Furthermore, multimodal fusion and cross-domain applications are essential directions for the development of deep learning technology. The fusion of deep learning and Convolutional Neural Networks allows for more flexible processing of data in different domains, such as images, text, and speech, achieving significant results and providing possibilities for the widespread application of deep learning in practical scenarios. However, challenges remain in the future, such as model interpretability, data privacy issues, and computational resource requirements. Future research needs to focus on the interpretability of deep learning models, develop more robust data privacy protection strategies, and strive to develop more efficient computing models. Overall, as a core technology in artificial intelligence, the future development prospects of deep learning are full of infinite possibilities. Through continuous innovation and efforts, we are confident in welcoming broader development opportunities in the field of deep learning and making greater contributions. References [1] Yang S, Luo L, Tan B. Research on Sports Performance Prediction Based on BP Neural Network[J]. Mobile Information Systems, 2021,2021:1-8. [2] Bai Y, Luo M, Pang F. An Algorithm for Solving Robot Inverse Kinematics Based on FOA Optimized BP Neural Network[J]. Applied Sciences, 2021,11(15):7129. [3] Liu Y, Dai J, Zhao S, et al. A bidirectional reflectance distribution function model of space targets in visible spectrum based on GA-BP network[J]. Applied Physics B, 2020,126(6). [4] Yang J, Hu Y, Zhang K, et al. An improved evolution algorithm using population competition genetic algorithm and self- correction BP neural network based on fitness landscape[J]. Soft Computing, 2021,25(3):1751-1776. [5] Quan G, Zhang Y, Lei S, et al. Characterization of Flow Behaviors by a PSO-BP Integrated Model for a Medium Carbon Alloy Steel[J]. Materials, 2023,16(8):2982.