Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 2, No. 2, 2022 66 Research on Lovelorn Emotion Recognition Based on Ernie Tiny Yuxin Huang School of Computing and Data Science, Xiamen University Malaysia, Selangor Darul Ehsan, Sepang 43900, Malaysia SWE1909470@xmu.edu.my Abstract: Topics related to sentiment classification and emotion recognition are an important part of the Natural Language Processing research field and can be used to analyze users' sentiment tendencies towards brands, understand the public's attitudes and opinions on public opinion events, and detect users' mental health, among others. Past research has usually been based on positive and negative emotions or multi-categorized emotions such as happiness, anger and sadness, while there has been little research on the recognition of the specific emotion of lovelorn. This study aims to identify the lovelorn emotion in text, using deep learning pretrained model ERNIR Tiny to train a dataset consisting of 5008 pieces of Chinese lovelorn emotion text crawled from social media platform Weibo and 4998 pieces of ordinary text extracted from existing available dataset. And finally, it was proved that ERNIE Tiny performs well in classifying whether a text contains lovelorn emotion or not, with F1 score of 0.941929, precision score of 0.942300 and recall score of 0.941928 obtained on the test set. Keywords: ERNIE Tiny; Lovelorn Emotion; Emotion Recognition. 1. Introduction Sentiment analysis is an attractive and valuable task in the discipline of Natural Language Processing, which is the process of analyzing, processing, generalizing and reasoning about text with sentiment, focusing on people's emotional attitudes toward various entities, including products, organizations, events, etc. According to the granularity of the analyzed object, it can be divided into document, sentence and aspect level. Sentiment analysis plays an important role in many fields, for example, movie reviews can be used to analyze whether users rate movies positively or negatively; sentiment tendency of users towards attributes such as price and usefulness of products can be mined in product review texts. In addition, sentiment analysis is also important in the field of mental health, which can understand the psychological needs of users and provide emotional support and treatment. There are a large number of emotionally charged texts on social media platforms, including the lovelorn emotion, but there has been little research on this specific emotion. This study focuses on detecting the emotion of love loss in texts, which can identify whether lovelorn emotion is contained from any text, helping heterosexual dating platforms to identify potential target users of love loss and advertise their dating software accordingly. While social media software can also cooperate with platforms that provide psychological therapy, to implement emotional care for these users and push them information that helps to get out of negative emotions, so that the target group can face lovelorn emotions in a healthier state of mind, showing the humanistic care of the organization. 2. Literature review Detecting emotions with natural language processing techniques is one of the applications of artificial intelligence in mental health. To address the problem that traditional machine learning models fail to effectively focus on some features in sentiment analysis tasks, the deep learning model LSTM was used by Chen et al. to extract and classify users' emotional features and applied to the scenario of postpartum depression screening. The role of emoticons was also considered by extracting features and modeling, and good results were achieved in general agreement with the Edinburgh Postpartum Depression Scale. The study significantly reduces the length of sentiment screening and served as a good reference for document-level emotion classification tasks in specific domains. Liu, Shi and Jiang used the (BSC + RFS)-fs, WEC-fs model proposed by the optimal classification model fusing single and multidimensional features to identify posts with suicidal ideation (including the stress of love loss) in social media, and the results demonstrated that an appropriate combination strategy and the selection of a targeted classification model contributed to the improvement of suicidal ideation detection rate. Traditional sentiment analysis techniques include supervised methods, such as Support Vector Machines, Maximum Entropy, Plain Bayes and other supervised machine learning methods, and unsupervised methods, including methods based on sentiment lexicon, syntactic analysis and syntactic patterns. The lexicon-based methods mainly use a series of sentiment lexicons and rules developed for paragraph disassembly and syntactic analysis of texts to calculate sentiment values, and finally use the sentiment values as a basis for judging the sentiment tendency of texts. Most of the machine learning-based methods treat this problem as a classification problem, manually labeling the training text, structuring the text content, and inputting it to the algorithm for training and learning, and classifying the target sentiment into positive and negative categories to judge the sentiment polarity. These early methods are relatively simple and have good results for some related tasks. However, sentiment lexicon-based approaches lack semantic associations between contexts, and the training of machine learning models relies on the quality of annotated datasets, 67 which require high quality feature construction and selection, and therefore have limitations. With the popularity of deep learning, the application of deep learning architectures in sentiment analysis tasks has become a very popular research topic in recent years, using multilayer neural networks to learn features and extract them automatically, overcoming many of the problems of traditional classification techniques. A large number of sentiment analysis studies based on deep learning (e.g., convolutional neural networks CNN, recurrent neural networks RNN, long-short term memory networks LSTM, etc.) have emerged and obtained state-of-the-art results in many tasks of sentiment classification. Kim discussed that fine-tuning task-specific word vectors in a CNN model can improve performance. With the performance of pre-trained word vectors on sentence classification tasks, it is demonstrated that simple single-layer CNN models with less hyperparameter tuning and static word vectors can achieve better performance on multiple benchmarks, optimizing tasks including sentiment analysis and problem classification. Tang, Qin, and Liu used the idea of memory network for aspect-level sentiment analysis, constructed the memory by context information, introduced attention to capture the important information in the sentiment tendency of different aspects, and achieve great results on the experimental dataset, which was simpler and faster to compute than the neural network models such as RNN and LSTM. In addition, the research demonstrated that combining content information and location information to learn context weight was a more suitable method for aspect-level sentiment analysis, and the multi-layer computational unit can learn more abstractive information, which can improve the model performance. An attention-based bi-directional CNN-RNN depth model (ABCDM) was proposed by Basiri et al. for sentiment analysis. By using two independent bidirectional LSTM and GRU layers, temporal information in both directions was considered to extract context. Also, the attention mechanism was applied in the output of the model's bidirectional layers to emphasize contributions of different words, achieving SOTA results on the Twitter dataset. The BERT+Attention model proposed by Wang and Tong improved the BERT pre-training model by introducing an attention mechanism and weighting key features to analyze the emotions expressed on microblogs during the epidemic, and obtained higher accuracy than the textCNN, BILSTM and the BILSTM+Attention model. 3. Research methodology 3.1. ERNIE Tiny model ERNIE Tiny, proposed by Baidu, is obtained by compressing ERNIE 2.0 Base model through the method of model structure compression and model distillation, using 3- layer transformer structure, which significantly improves the prediction speed by 4 times and can quickly land the project. And ERNIE 2.0 consists of two parts, Transformer Encoder and Task Embedding, which improves the masking strategy and corpus on the basis of BERT, making it more suitable for Chinese sentiment classification tasks. (1) Transformer Encoder Consistent with several pre-trained models such as BERT, the ERNIE 2.0 model also uses a 12-layer Transformer as encoder. Unlike BERT, the Transformer encoding layer of BERT is used in the first 6 layers, but in the 7th custom knowledge fusion layer BertLayerMix, the aligned entity vector and the denominator vector are summed for the first time and transmitted to the knowledge encoding module and the text encoding module, respectively, and in the remaining 5 custom knowledge encoding layers BertLayer, the entity sequence and the text sequence, which have been fused with both information, are encoded using the self-attention mechanism, respectively. Transformer captures the contextual information of each token in a text sequence by self-attention and generates contextual representation embeddings. For a given sequence, the starting position is a predefined separator [CLS]; for tasks where the input is multi-segment, the different segments are separated by a predefined [SEP]. (2) Task Embedding The task embedding in the model is used to apply different tasks, and each task id has its own specific task embedding, which is input to the model along with token embedding, position embedding and sentence embedding. In the fine- tuning phase, task ids can be selected to initialize the model. And the sentence loss and token loss are used as the loss function. The structure of ERNIE 2.0 model is shown in Figure 1. Figure 1. ERNIE 2.0 model structure (3) Model Distillation ERNIE Tiny compresses the pre-trained model through a progressive distillation framework using four stages of distillation, including General-Enhanced Distillation, where the student model receives augmented knowledge from finetuned teachers and generalization capability is improved. Task-Adaptive Distillation smooths the transition through designed learning objectives. Figure 2 shows the model distillation work flow of ERNIE Tiny. Figure 2. Workflow of ERNIE Tiny Based on previous studies, it can be seen that deep learning models, especially pre-trained models like BERT, have superior results for sentiment classification tasks. At the same time, there are fewer studies on the psychological emotion analysis for the emotion of lost love, so this paper focuses on 68 the study of the recognition of the lovelorn emotion using deep learning model. 3.2. Experiment 3.2.1. Data preparation and processing The first step was to obtain the data. Since there was no readily available textual dataset for love loss, a web crawler program was written to crawl the comments of articles about "love loss", "regret of youth" and other related words on Chinese social media platforms Weibo. Then data cleaning was performed, using Python to remove the empty and duplicate lines in the comments, and then manually filter the comments that are not related to the emotion of lost love, and finally got 5008 pieces of text with the emotion of lovelorn, and set the label to 1, which means it contains the emotion of lost love. The next step was merging the data. The comments from the existing general Weibo sentiment seven-category comment dataset OCEMOTION [10], which is not specific to any topic, was extracted and merged with the lost love sentiment text into a csv file. Since ordinary comments may also contain comments related to love loss, 4998 comments under the "happiness" sentiment tag were selected as non- love-loss sentiment data, and the tag was set to 0, resulting in a dichotomous (lovelorn and normal) Weibo sentiment dataset. The sample comments were translated into English, as shown in Table 1. Finally, the dataset was divided. The training set, validation set and test set were divided according to 8:1:1 and the data were randomly shuffled, making the data of different categories as uniformly distributed as possible. Table 1. Sample review text Text Label (1: lovelorn; 0: normal) No more, he said he doesn't love me anymore. 1 I said the breakup, but why do I miss him so much, and also sad, I seem to have lost him. 1 Eight years, how can I give up. You are so good, so good that I want to spend my life with you. But I was disqualified. I'm lost, I don't want to go the rest of the way. 1 Today go out to play so cool ah, both wind and sun and sun UV weak. 0 Happiness is not about having more, but about thinking less. 0 Wake up and find that can still stay in bed! 0 3.2.2. Build and Train Model After processing the data into a format acceptable to the model, using PaddleHub, a pre-training model management and migration learning tool, the pre-trained model ERNIE Tiny was loaded and the text was processed using the model's built-in tokenizer. Next, the AdamW optimizer was selected as the optimization strategy for pretrained model ERNIE Tiny, setting the learning rate and using the model default parameters. Then run the configuration to control the fine- tuning of the training task, specify the training and validation sets of the model, and configure the training parameters such as the number of training epochs, batch size, etc. 4. Result discussion In this experiment, AdamW was chosen as the optimizer of the pre-trained model to solve the problem of L2 regularization failure in the Adam optimizer [11]. Finally, the test set was evaluated with macro F1-score, Precision (P), and Recall (R) metrics which are the common indexes used in classification tasks. F1 value is the harmonized average of P and R values, and macro F1 score directly calculate the average of F1 values for each category. The precision is the percentage of samples with positive true values among all samples with positive predicted values, and the recall is the percentage of samples with positive predicted values among all samples with positive true values. The final test score results were obtained as shown in Table 2, with F1 score, precision score and recall score of 0.941929, 0.942300 and 0.941928 respectively. It can be seen that ERNIE Tiny model can accurately identify and classify the emotions of lovelorn, and get satisfying results. Table 2. Evaluation index result F1-score Precision Recall 0.941929 0.942300 0.941928 5. Conclusion In this study, the basic architecture and principles of the deep learning model ERNIE Tiny were introduced. Three approaches to sentiment classification (sentiment lexicon- based approach, machine learning based approach and deep learning approach) were explored. In order to address the problem that few existing sentiment analysis studies detect love loss emotion texts, the author crawled love loss emotion texts on the social media platform Weibo, combined with normal sentiment reviews as dataset, and the ERNIE Tiny model was trained to recognize lovelorn emotion texts, which can effectively classify texts containing love-loss emotion from ordinary non-love-loss texts, and finally achieve satisfactory results. On the one hand, this study contributes to the work on emotion perception in digital mental health treatment solutions, reflecting the important value of sentiment analysis in this area and having valuable social significance. On the other hand, since punctuation marks may also contain some enhanced emotions, and the number of training texts is not very large, future work is to consider the role of punctuation marks into the problem of sentiment analysis with more training data, further improve the accuracy of sentiment classification. References [1] Chen, Y., Zhou, B., Zhang, W., Gong, W., & Sun, G. (2018). Sentiment Analysis Based on Deep Learning and Its Application in Screening for Perinatal Depression. 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC). https://doi.org/10.1109/dsc.2018.00073 [2] Liu, J., Shi, M., & Jiang, H. (2022). Detecting suicidal ideation in social media: An ensemble method based on feature fusion. International Journal of Environmental Research and Public Health, 19(13), 8197. doi:10.3390/ijerph19138197 [3] Zhang, L., Wang, S., & Liu, B. (2018). Deep Learning for Sentiment Analysis: A Survey. Cornell University - arXiv. https://doi.org/10.48550/arxiv.1801.07883 [4] Kim, Y. (2014). Convolutional neural networks for sentence classification. Proceedings of the 2014 Conference on 69 Empirical Methods in Natural Language Processing (EMNLP). doi:10.3115/v1/d14-1181 [5] Tang, D., Qin, B., & Liu, T. (2016). Aspect Level Sentiment Classification with Deep Memory Network. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/d16-1021 [6] Basiri, M. E., Nemati, S., Abdar, M., Cambria, E., & Acharya, U. R. (2021). ABCDM: An attention-based bidirectional CNN- RNN deep model for sentiment analysis. Future Generation Computer Systems, 115, 279-294. doi:10.1016 / j.future.2020.08.005 [7] Wang, X., & Tong, Y. (2021). Application of bert+attention model in emotion recognition of Metizens during epidemic period. Journal of Physics: Conference Series, 1982(1), 012102. doi:10.1088/1742-6596/1982/1/012102 [8] Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., & Wang, H. (2020). Ernie 2.0: A continual pre-training framework for language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 8968-8975. doi:10.1609/aaai.v34i05.6428 [9] Su, W., Chen, X., Feng, S., Liu, J., Liu, W., Sun, Y., . . . Wang, H. (2021). ERNIE-Tiny: A Progressive Distillation Framework for Pretrained Transformer Compression. arXiv: Computation and Language. [10] Li, M., Long, Y., Lu, Q., & Li, W. (2016). Emotion corpus construction based on selection from hashtags. Language Resources and Evaluation, 1845–1849. [11] Loshchilov, I., & Hutter, F. (2018). Decoupled Weight Decay Regularization. International Conference on Learning Representations.