Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 2, No. 3, 2022 109 Object Detection in Junk Image Based on SSD Model Zhihao Guan Yunnan University of Finance and Economics, School of Information, Kunming 650221, China guanzhihao1997@163.com. Abstract: This paper first introduces the background and significance of the direction in target detection, and introduces other target detection methods besides SSD. Secondly, this paper also introduces the principle and implementation process of SSD. It describes in detail the important steps to implement SSD and attaches relevant codes. Finally, the achieved results are compared with different methods, different data sets VOC 07 and networks using different methods, and conclusions are drawn. Keywords: Target detection; SSD; Single-shot multi-frame detection; Taco dataset; Garbage detection. 1. Introduction 1.1. Background and Significance Vision plays an extremely important role in human perception and acquisition of external world information, and images are the most reliable and direct carrier to convey visual information. Computer vision is a discipline in which human beings’ study how to use computers to process and analyze image information like the human brain. As one of the most basic problems in the field of computer vision, object detection (Object Detection) lays the foundation for solving more complex or higher-level visual tasks such as image segmentation, image understanding, image captioning, object tracking, and activity recognition. At the same time, it has a wide range of applications in the fields of artificial intelligence and information technology, including robot vision, security, consumer electronics, autonomous driving, human-computer interaction, intelligent video surveillance, and reality augmentation. As shown, the definition of object detection is to determine whether any object instance from a certain category (such as human, bicycle, TV, dog and cat, etc.) exists in a given input image, and if so, return the Spatial Location and Extent, which aims to locate and identify a wide range of natural object categories. The ideal goal for object detection is to develop algorithms that achieve two goals: high accuracy and high efficiency. However, in the field of target detection, it is often difficult to have both the detection speed and accuracy of the system. Usually, high accuracy means that the speed is relatively slow, while high efficiency means that the accuracy is relatively low. High-accuracy detection must accurately locate and identify objects, distinguishing a wide variety of object classes. High efficiency requires detection tasks to run at a sufficiently high frame rate, that is, to achieve real-time detection. Despite decades of research, the combined goal of high accuracy and speed has yet to be achieved. Among these many target detection algorithms, the Single Shot multibox detector algorithm (Single Shot multibox Detector, SSD) uses multi-scale features to detect targets of different sizes, which is a relatively fast and robust method, so as a The basic algorithm framework, which is widely used in face recognition, pedestrian detection and tracking, road vehicle detection, license plate recognition and security video surveillance and other fields, and has achieved good results. Generally speaking, when the size of the target is relatively small relative to the size of the entire original image, it can be called a small target. In practice, objects with a scale smaller than 32×32 pixel values are roughly judged as small objects. Since the SSD algorithm does not consider the connection between different output layers, it ignores the context information that is important for small target detection, which makes the use of feature information insufficient, which will lead to the problem that small target detection is not robust enough. At the same time, there are sample categories and multi-task imbalance problems in the training process. The accumulated loss value of a large number of easy-to-segment samples and particularly difficult-to-segment samples may overwhelm the loss of relatively few ordinary difficult-to- segment samples, directly increasing the positioning loss. The weight of will make the model more sensitive to outliers. This is not conducive to the improvement of the judgment of the model, and will limit the further optimization of the model to a certain extent. Therefore, aiming at the shortcomings and deficiencies of the traditional SSD algorithm, this paper proposes and adopts three different improved methods based on deep learning to optimize the traditional SSD target detection algorithm, mainly including feature fusion, attention mechanism and loss function. direction. At the same time, an in-depth and detailed analysis of the proposed method is carried out, and quantitative and qualitative comparative analysis is carried out with popular algorithms under a fair experimental setting. For the field of target detection, the results of this paper have expanded new ideas in a sense, and can enrich the technical methods of target detection. 1.2. Research status at home and abroad Early research on object detection was based on template matching techniques and models of simple handcrafted features, focusing on specific objects with roughly rigid spatial layouts, such as eyes, mouths, and human faces. Traditional object detection algorithms are generally based on manually designed geometric features as input. Common artificially designed feature operators include: SIFT (Scale Invariant Feature Transform, scale invariant feature transformation), HOG (Histogram of Oriented Gradient, histogram of directional gradient) and SURF (Speeded Up Robust Features, accelerated robust features), etc. Then, use common statistical classifiers, such as neural network, support vector machine (Support Vector Machine, SVM) and Adaboos , etc. to classify the target. By the late 1990s and 110 early 2000s, research on object detection had made remarkable progress. Frameworks based on proposal regions and multi-class appearance features are continuously proposed, which makes these multi-level detection frameworks consisting of hand-drawn local description features and discriminative classifiers have been dominating various fields of computer vision, including object detection. Until the important turning point in 2012, as DCNN achieved record-breaking results in image classification, the field of object detection also began to enter the era based on DCNN. However, the sliding window technology and multi-scale pyramid strategy proposed in the traditional field are still widely used in DCNN-based methods. Since deep learning entered the field of target detection, various target detection methods based on deep convolutional neural networks have made remarkable progress and quickly become the mainstream. At present, the main target detection algorithms can be roughly divided into two categories: (1). The object detection algorithm proposed by the candidate region, that is, the two-stage detection framework algorithm. (2). Regression-based target detection algorithm, that is, a one-stage detection framework algorithm. The Mask RCNN proposed by He Kaiming et al. uses the network structure of ResNet and FPN to mine multi-scale information and feature extraction, and adds a fully convolutional network (Fully Convolutional Network, FCN) at the end of the network to provide each target candidate The region outputs a binary mask, while avoiding the original RoI The offset caused by the pooling layer , the ROI Align layer was proposed to maintain the pixel-level spatial correspondence, and Mask RCNN achieved the best results at that time, but due to its complex network structure, its detection speed was too slow. The Mask RCNN proposed by He Kaiming et al. uses the network structure of ResNet and FPN to mine multi-scale information and feature extraction, and adds a fully convolutional network (Fully Convolutional Network, FCN) at the end of the network to provide each target candidate The region outputs a binary mask, while avoiding the original RoI The offset caused by the pooling layer , the ROI Align layer was proposed to maintain the pixel-level spatial correspondence, and Mask RCNN achieved the best results at that time, but due to its complex network structure, its detection speed was too slow. In addition, the researchers found that deep features in the basic backbone network have more semantic information, while shallow features have more content descriptions, and utilizing multi-scale features to generate discriminative feature pyramid representations is crucial for detection performance. Most of the above detector structures only use single-scale features, such as R-CNN, Fast R-CNN, Faster R- CNN and YOLO, which only use the feature map of the last layer of the network to extract features and complete the detection task. In order to improve the detection accuracy of multi-scale objects, especially small-scale objects, some object detection model architectures have also made significant progress in multi-scale. Among them, MS-CNN proposed two sub-networks, and introduced multi-scale features into Fast R-CNN for the first time to improve the recognition ability of small targets. FPN exploits lateral connections and top-down paths to create feature pyramids and achieve more powerful representations. SSD uses multi- scale features, and sets the corresponding default frame according to the size of the receptive field of the feature layer. Therefore, SSD can detect objects of different scales and aspect ratios from multiple layers. On the other hand, since the basic backbone network of the target detection task is derived from the classification network, with the development of deep neural networks, the network structure is getting deeper and deeper, and basic backbone networks such as ResNet and DenseNet that can obtain better feature information have emerged. In order to improve the accuracy of the SSD algorithm, DSSD replaced the basic backbone network of the original SSD algorithm with ResNet-101, and proposed to aggregate context information through deconvolution layers to enhance the high-level semantics of shallow features. DSOD uses DenseNet with higher parameter efficiency than ResNet as the basic backbone network to study how to train object detectors from scratch. 1.3. Main research content and arrangement of each chapter This chapter mainly introduces the background and significance of target recognition, and also introduces the research status at home and abroad. Next, this article will focus on the principle of SSD and its implementation process. 2. The principle and analysis of single- shot multi-frame detection 2.1. Introduction to single-shot multi-frame detector algorithm In recent years, with the development of deep learning and target detection technology, there have been many methods to improve the detection accuracy of the SSD algorithm. For example, the DSSD algorithm replaces VGG-16 with a more complex ResNet to obtain a feature map with better representation capabilities, but the complex The network structure means a huge amount of network parameters, this method greatly increases the memory and computational complexity, and seriously reduces the detection speed. In addition, the researchers found that the deep features in the basic backbone network have more semantic information, while the shallow features have more content descriptions, that is to say, the deeper advanced features have a stronger ability to distinguish classification tasks, while the shallow features Low-level features are more discriminative for the object location regression task. For example, in recent years, RSSD has designed a rainbow connection method that uses pooling and deconvolution operations at the same time to perform feature fusion on different feature layers. FSSD generates a large-scale feature by fusing multiple shallow feature layers of different scales, and then constructs a new feature pyramid for detection by downsampling on this large- scale feature layer. RefineDet effectively improves the accuracy of the SSD algorithm by refining and refining the position and size of the anchor frame. Although the above methods have made good progress, their architecture is very complex, which greatly reduces the detection speed, but at the same time these methods reveal that the feature information of the shallow layer and the feature information of the deep layer are complementary in target detection. How to utilize shallow and deep features to effectively integrate feature pyramid representations determines the detection performance. Among many object detection algorithms, Single Shot multiBox Detector (SSD) is a fast and robust method. As shown in the figure, it uses the feature maps of different output layers of the network to directly predict the target 111 category and bounding box, and adopts a multi-scale strategy to integrate the detection results of different layers, and finally uses the non-maximum value suppression method (Non- Maximum Suppression, NMS) suppresses redundant detection frames. Compared with many object detection algorithms, Single Shot multiBox Detector (SSD) is a fast and robust method. As shown in Figure 2.1, it uses the feature maps of different output layers of the network to directly predict the target category and bounding box, and adopts a multi-scale strategy to integrate the detection results of different layers, and finally uses the non-maximum value suppression method ( Non-Maximum Suppression, NMS) suppresses redundant detection frames. Unlike other two- stage detection algorithms, SSD does not need to generate region proposals, it directly detects objects on the feature layer of the output layer, so SSD can achieve real-time detection. 2.2. Principle of SSD Algorithm As shown in the figure, the SSD algorithm uses VGG-16 as the basic backbone network. VGG-16 is a convolutional neural network model proposed by K. Simonyan and A. Zisserman in 2014. SSD uses the two full The connection layers Fc6 and Fc7 are converted into 3×3 convolutional layer Conv6 and 1×1 convolutional layer Conv7, then the dropout layer and the fully connected layer Fc8 are removed, and then 4 convolutional layers are added to construct the network structure of the algorithm, Finally, conv4_3, conv7, conv8_2, conv9_2, conv10_2 and conv11_2 are used to construct a multi-scale feature pyramid for detection tasks to complete detection and recognition tasks. Based on the above network structure, the image is passed through a deep neural network to generate a series of feature maps. The traditional SSD abandons the region proposal step in the two-stage detection framework and instead uses multi- scale feature maps for detection. First, a series of default boxes (Default Box) are generated for the six feature layers (feature maps) of different sizes in the feature pyramid according to the size and aspect ratio, and then the prior box (Prior Box) is selected for training, and finally the first The inspection box is the initial bounding box and returns it to the correct target ground truth . The whole process can be completed through one forward propagation of the network. The mechanism of Default Box is similar to Anchor Boxes in Faster RCNN. For feature layers of different scales, the default boxes have different sizes and aspect ratios in order to detect targets of different scales. As shown in the figure, picture (a) is a picture marked with a real bounding box, picture (b) and picture (c) are feature layers of two different scales, respectively, 8 × 8 and 4 × 4 the default box is set for each unit of the feature layer A series of fixed-sized borders on a grid. Assuming that the size of a feature layer is 𝑚 × 𝑛, and each cell has 𝑘a default box, for each default box, it is necessary to predict the confidence loss (confidence loss, conf) of all categories and the localization loss (localization loss, loc) of the border regression, then this feature layer has a total of 𝑘𝑚𝑛(𝑐 + 4)output. In practice, two 3 × 3 convolution kernels are used to convolve the feature map, and the confidence score conf for classification and the location information loc for regression are respectively output. For the Pascal VOC dataset, each default box generates confidence scores for 21 categories (including background) ((𝑐1, 𝑐2, . . . , 𝑐𝑝)) , and each default box generates 4 position prediction coordinate values ((𝑐𝑥, 𝑐𝑦, 𝑤, ℎ), which represent the center coordinates, width and height of the bounding box, respectively. In the actual training process, the algorithm needs to match the real bounding box with the default box. The purpose is to find a best matching default box for each real bounding box, which is called a priori box, which is the default box selected in the actual process. frame. The specific operation is as follows: firstly, for each ground-truth bounding box, find the IoU) value of the prior box and match it to ensure that each real bounding box must match a certain prior box and be recorded as a positive sample. As shown in Figure 2.4, in target detection, IoU represents the ratio of the intersection and union of the predicted border and the real border. The calculation formula of IoU is as follows: 𝐼𝑜𝑈(𝐺, 𝑝) = | 𝐺 ∩ 𝑃 𝐺 ∪ 𝑃 | On the contrary, if an prior box does not find any ground- truth bounding box to match it, then the prior box can only match the background and be recorded as a negative sample. Secondly, for the remaining unmatched a priori boxes, if the IoU value between them and a real bounding box is greater than the threshold 0.5, then the a priori box is also matched with the real bounding box . Although one ground-truth bounding box can be matched with multiple prior boxes, the number of ground-truth bounding boxes is too small relative to the prior box, which leads to a lot more negative samples than positive samples. In order to ensure the balance of the number of positive and negative samples in the training process as much as possible, SSD adopts the strategy of mining difficult examples to arrange the negative samples in descending order according to the confidence error and sample them to ensure that the ratio of positive and negative samples in the training process is close to 1:3. Among them, the smaller the confidence of the predicted background is, the larger the error is, and the Top-k with the larger error is finally selected. Feature layers of different scales have different receptive fields. Therefore, the SSD algorithm matches the size of the receptive field by setting default boxes of different sizes on six feature layers of different scales. The setting of the default frame size of each layer mainly includes two aspects: scale and aspect ratio. The default boxes have different scales in different feature maps, and have different aspect ratios in the same feature map, so it can detect most objects of different scales. The scale formula of the default box for each feature layer is: 𝑆𝑘 = 𝑆𝑚𝑖𝑛 + 𝑆𝑚𝑎𝑥 − 𝑆𝑚𝑖𝑛 𝑚 − 1 (𝑘 − 1), 𝑘 ∈ [1, 𝑚] Among them, 𝑚 represents the number of feature maps, 𝑆𝑘represents 𝑘the ratio of the size of the prior frame of the 𝑆𝑚𝑖𝑛 = 0.2feature map of the first layer to the original image, and 𝑆𝑚𝑎𝑥 = 0.9 respectively represents the ratio of the minimum and maximum scales of the prior frame to the original image is 0.2 and 0.9, and the length and width of the algorithm The value of the ratio is: , so the formulas for calculating the sum of the width 𝑎𝑟 = {1 , 2 , 3 , 1 2 , 1 3 } and height ℎ𝑘 𝑎of each default box 𝑤𝑘 𝑎are: 𝑤𝑘 𝑎 = 𝑆𝑘√𝑎𝑟 ℎ𝑘 𝑎 = 𝑆𝑘 √𝑎𝑟 In addition, when the aspect ratio is 1 , a scale is added as 𝑠𝑘 ′ = √𝑆𝑘𝑆𝑘+1the default box. The center of each default box 112 is distributed in the center of the position cell, namely: (𝑖 + 0.5 |𝑓𝑘| , 𝑗 + 0.5 |𝑓𝑘| ) , 𝑖, 𝑗 ∈ [0, |𝑓𝑘|), where |𝑓𝑘|represents 𝑘the feature map size of the th layer. In the detection stage of SSD, because multiple feature maps will generate a large number of target positioning frames according to the settings, but these target positioning frames contain many wrong, overlapping and inaccurate results, so the SSD algorithm needs to use non- the maximum value suppression algorithm processes the prediction results to eliminate the positioning boxes and repeated boxes that do not contain the target, and obtain the final detection results. The loss function of the traditional SSD algorithm consists of the weighted sum of the bounding box localization loss function 𝐿𝑙𝑜𝑐 and the classification confidence loss function 𝐿𝑐𝑜𝑛𝑓: 𝐿(𝑥, 𝑐, 𝑙, 𝑔) = | 1 𝑁 | (𝐿𝑐𝑜𝑛𝑓(𝑥, 𝑐) + 𝛼𝐿𝑙𝑜𝑐(𝑥, 𝑙, 𝑔)), (𝑥 = 𝑥𝑖,𝑗 𝑝 𝜖{0,1}) Among them, 𝑁the number of positive samples of the prior box 𝑥𝑖,𝑗 𝑝 is an indicator parameter. When 𝑥𝑖,𝑗 𝑝 = 1the th 𝑖prior box matches the th 𝑗real bounding box, and the category of the target object contained in the real bounding box is 𝑝, the predicted category confidence is 𝑐. 𝑙is the position prediction value of the bounding box corresponding to the prior box, 𝑔and is the position parameter of the real bounding box, and the weight coefficient 𝛼 is set to 1 and used to adjust the proportional weight between the bounding box regression loss and the classification loss through cross-validation. For the classification confidence loss function 𝐿𝑐𝑜𝑛𝑓 , Softmax Loss is used, and its formula is: 𝐿𝑐𝑜𝑛𝑓(𝑥, 𝑐) = − ∑ 𝑥𝑖,𝑗 𝑝 𝑁 𝑖∈𝑃𝑜𝑠 log(𝑐�̂� 𝑝 ) − ∑ log(�̂�𝑖 0) 𝑖∈𝑁𝑒𝑔 �̂�𝑖 𝑝 = exp (𝑐𝑖 𝑝 ) ∑ exp (𝑐�̂� 𝑝 )𝑝 Among them, 𝑃𝑜𝑠denotes a positive sample, 𝑁𝑒𝑔denotes a negative sample, denotes the probability that the target in the th prediction frame of the target 𝑖 is the th 𝑝 category, denotes that there is no object in the prediction frame, that is, refers to the category number, and 𝑝𝑝 = 0 denotes the background. For the frame positioning loss function 𝐿𝑙𝑜𝑐 , Smooth L1 is used and its formula is: 𝐿𝑙𝑜𝑐(𝑥, 𝑙, 𝑔) = ∑ ∑ 𝑥𝑖,𝑗 𝑝 𝑆𝑚𝑜𝑜𝑡ℎ𝐿𝐼(𝑙𝑖 𝑚 − �̂�𝑗 𝑚) 𝑁 𝑖∈𝑃𝑜𝑠∈{𝑐𝑥,𝑐𝑦,𝑤,ℎ} 𝑆𝑚𝑜𝑜𝑡ℎ𝐿𝐼 = { 0.5𝑥2 |𝑥| < 1 |𝑥| − 0.5 |𝑥| ≥ 1 Among them, 𝑙𝑖 𝑚 is the 𝑖 th prediction frame, and 𝑖 the location information of the th prior frame is 𝑑𝑖denoted by , where 𝑑𝑖 = (𝑑𝑖 𝑐𝑥 , 𝑑𝑖 𝑐𝑦 , 𝑑𝑖 𝑤 , 𝑑𝑖 ℎ) is a four-dimensional vector, respectively representing the center coordinates, width and height of the prior frame. �̂�𝑗 𝑚In order to 𝑗convert the encoded value of the first ground truth bounding box relative to the prior box, the coordinate encoding formulas of the ground truth bounding box are :𝑔𝑗 = (𝑔𝑗 𝑐𝑥 , 𝑔𝑗 𝑐𝑦 , 𝑔𝑗 𝑤 , 𝑔𝑗 ℎ) �̂�𝑗 𝑐𝑥 = (𝑔𝑗 𝑐𝑥 − 𝑑𝑖 𝑐𝑥)/𝑑𝑖 𝑤 �̂�𝑗 𝑐𝑦 = (𝑔𝑗 𝑐𝑦 − 𝑑𝑖 𝑐𝑦 )/𝑑𝑖 ℎ �̂�𝑗 𝑤 = log ( 𝑔𝑗 𝑤 𝑑𝑖 𝑤) �̂�𝑗 ℎ = log ( 𝑔𝑗 ℎ 𝑑𝑖 ℎ ) 2.3. Chapter summary This chapter introduces the main principles of the SSD network and summarizes the formulas. Next, we will implement the SSD network and make it train the Taco dataset and test its effect. 3. Implementation of single-shot multi- frame detection 3.1. Configuration environment and preparations First of all, we need to install the deep learning framework, we choose Pytorch here. Open the Pytorch official website and choose the version that suits us according to our conditions. What I choose here is the LTS version in Linux. We follow the prompts on the official website and enter in the terminal pip3 install torch==1.10.0+cu113 torchvision ==0.11.1+cu113 torchaudio ==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html After the run is over, we have installed Pytorch suitable for our c uda version. This string of code will also automatically install the latest cuda for us . We can query our cuda version using nvidia-smi . In addition to the need to install Pytorch , we also need to install other packages. The list of packages is as follows: scipy , numpy , matplotlib , opencv -python , torch , torchvision , tqdm , Pillow , h5py . We can manually install these packages using pip , or we can open my TacoSSD folder and run the following code: pip3 install -r requirements.txt In this way, the system will automatically install the required packages for us. I will put the download address of the TacoSSD file at the end of the article 3.2. Introduction to Taco Dataset and Dataset Production The dataset we choose here is Taco. T aco is a growing image dataset of garbage images. It contains images of garbage taken in various environments: woods, roads, and beaches. These images are manually labeled and segmented according to hierarchical classifications to train and evaluate object detection algorithms. Taco dataset: https://github.com/pedropro/TACO 15 different groups of pictures in the data set , and each group has about 100 pictures. In order to reduce the training time, we only select half of them, and first manually select some garbage and clear images, and delete the unclear images. In the end we are left with 502 images that meet the criteria. Once the images are selected, we can start the production of the dataset. Here we use label abelimg , label img is a software that is convenient for data scientists to make data sets, and it can manually mark labels in images . Let's open the terminal first, and enter in it: pip3 install labelimg After the installation is complete, we enter in the terminal: lableImg In the picture, we can choose the folder to open and the location of the output xml file. Here we choose JPEGImages 113 as the folder where the picture is located, and Annotations as the folder where the output xml is located. manually marking 502 pictures here, we can start the training of the network. 3.3. SSD network training Due to the complexity of the SSD network, if we do not use pre-trained weights, it may be difficult to make the network converge with the discrete graphics card of my laptop. So we found other people's pre-trained weights on the Internet using general-purpose datasets. put the s sd_weights.pth downloaded from the Internet under the model _data folder. Then activate the environment we configured before, and enter in the terminal: conda activate Pytorch After activating the environment, we first run the file t ac o _annotation.py. t ac o _annotation.py is the code used to set which is the training set and which is the verification set. When we finish running, two files t aco_train.txt and t aco_val.txt will appear under the folder, which are the target information of the randomly selected images. Usually, we choose a ratio of 9:1 for the training set and the validation set. When the above work is completed, we can start the training of our network. 3.4. Spam detection the file with the smallest l oss in the logs file . Here l oss is the smallest ep 98 , so we choose it as the weight we use for prediction. Before we can predict, we also have to modify the model_path in the ssd.py file . Change the previously selected s sd_weights.pth to our trained weight ep 98 . At this point we can run the predict.py file for garbage detection. We enter in the terminal: python3 predict.py Then enter the image path: img /photo512/00000193.jpg 、 The result is as follows: Other predictions: 4. Summarize Figure 1. Average Precision m AP of this garbage detection is 0.58811, which is still slightly insufficient compared with the 0.78 of VOC 07+12 trained by this network. I have summarized several reasons. First, the number of training sets is too small. The training set I chose this time has only 500 pictures, which is too small compared to the 9000 pictures of VOC 07. Too few training sets may It lead to incomplete network training. If the amount of training set is increased, mAP may be improved. And I manually mark the pictures. If I want to mark 1500 pictures in the Taco dataset, the workload is a bit heavy, so I can improve the labeling method and use some methods of image processing to reduce the workload. Compared with the R - CNN series and the yolo series, the mA P of the SSD still has a gap. It can be seen from the figure that the SSD series still has room for improvement. Also, due to the limitation of the machine, I can't use pictures with more pixels for training. When the batch size is 16, my computer will report an error because of insufficient video memory. Therefore, the next step to improve this network is to increase the quantity and quality of the data set, provide more data for network training, and use cloud GPU to train this network, and use pictures with more pixels for training. References [1] Hochreiter S,Schmidhuber J. Long short-term memory[J]. Neural Computation,1997,9(8):1735-1780. [2] Chung J,Gulcehre C,Cho K H,et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXiv preprint arXiv:1412. 3555,2014. [3] Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning [J]. Nature, 2015, 518 (7540):529-533. [4] Weston J,Chopra S,Bordes A. Memory networks[J]. arXiv preprint arXiv:1410. 3916,2014. [5] [Sukhbaatar S,Weston J,Fergus R. End-to-end memory networks [C] / / Advances in Neural Information Processing Systems (NIPS),2015:2440-2448 [6] Pritzel A,Uria B,Srinivasan S,et al. Neural episodic control[C] / / International Conference on Machine Learning(ICML),2015. [7] Graves A,Wayne G,Danihelka I. Neural turing machines[J]. arXiv preprint arXiv:1410. 5401,2014. [8] Graves A,Wayne G,Reynolds M,et al. Hybrid computing using a neural network with dynamic external memory[ J]. Nature,2016, 538(7626):471-476. [9] Sukhbaatar S, Szlam A, Weston J,Fergus R. End-to-end memory networks//Proceedings of the Annual Conferenceon Neural Information Processing Systems. Montreal, Canada, 2015:2440-2448 [10] Vaswani A,Shazeer N,Parmar N,etal .Attention is all you need//Proceedings of the Annual Conferenceon Neural Information Processing Systems . LongBeach, USA, 2017: 5998-600 [11] Lin L J. Self-improving reactive agents based on reinforcement learning,planning and teaching[ J]. Machine Learning,1992,8(3- 4):293-321. [12] Schaul T,Quan J,Antonoglou I,et al. Prioritized experience replay [C] / / International Conference on Learning Representations (ICLR),2016.