Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 9, No. 3, 2024 46 A Review of Facial Micro Expression Recognition Yali Yu, Zonghou Huang and Jinchan Liang Guangdong University of Science and Technology, Dongguan Guangdong, 523668, China Abstract: With facial expression recognition gradually becoming a hot topic in the fields of image processing and artificial intelligence research, more and more scholars are paying attention to the fact that the micro expressions instantly revealed on the face can better reflect human inner emotions and thoughts. In this paper, firstly, the research status of micro expression and the commonly used micro expression datasets are described, and the advantages and disadvantages of each dataset are analyzed. Then the feature extraction of micro expression is analyzed from the two algorithms. Finally, the application fields of micro expression research and the challenges facing the future development are discussed. Keywords: Micro Expressions; Feature Extraction; Image Processing. 1. Introduction Facial expressions are one of the common ways for humans to express emotions in daily communication. During communication, people can express their inner thoughts and feelings to the other person through some facial expressions and actions. In different facial expressions, facial muscles have different states and can reveal different information. With the rapid development of artificial intelligence and the application of deep learning algorithms in recent years, research on facial expression recognition has achieved remarkable results, and subtle changes in muscles and instant expressions have aroused scholars' research interest. There are many differences between micro expressions and macro facial expressions. Micro expressions last for only 1/25 of a second, with weak changes and small muscle movement areas [1]. However, these fleeting expressions can better reveal people's true feelings and emotions deep inside. Micro expressions are usually spontaneous and subconscious, and it is difficult for humans to disguise and control the expression of these emotions [2]. These micro expressions that last for a short period of time often flash by suddenly, and sometimes express emotions opposite to those in the heart. Therefore, it is difficult for people to recognize micro expressions with the naked eye and the accuracy is very low. Deep learning algorithms and deep neural network models are needed to improve the accuracy of recognizing micro expressions. The development history of micro expressions can be traced back to 1969, when Ekman [3] and team members discovered in a video of a psychologist talking to a patient with depression that the patient attempted to conceal their suicidal thoughts by using a positive emotional state. Through repeated research on the video, Ekman's team found that there were several frames in the video where the patient's face showed very faint expressions of pain. This brief and imperceptible emotion is called a micro expression, which is the true expression of the patient's inner emotions. In 2002, Ekman et al. developed a micro expression recognition tool called METT (Micro Expression Training Tool) [4,5], which aims to help people recognize and understand micro expressions. In addition, Ekman and his research team also designed a Facial Action Coding System (FACS) through in- depth research. The system divides the face into several independent and interconnected motion units (AU) based on its anatomical characteristics, and analyzes the motion characteristics of these motion units, the main regions they control, and the expressions related to them. Additionally, each individual AU is associated with one or more local muscles, and the corresponding muscle movements can be represented using AU encoding [6,7]. Zhang Yanliang et al. proposed a local region partitioning and analysis method, which determines the most important local regions of the face based on facial action units, and then extracts spatiotemporal pattern features of each region for expression analysis, discovering the correlation between various micro expressions and facial regions [8]. Yu Yang et al. proposed a micro expression detection method based on a multi-scale spatiotemporal attention network to prevent feature distortion when extracting temporal features and maintain the original temporal relationship during spatial processing. The method uses a three-dimensional convolutional neural network (3DCNN) to extract multi-scale spatiotemporal features, and then uses a global spatiotemporal attention module (GSAM) to enhance the global spatiotemporal correlation of spatiotemporal features. Finally, weights are assigned to features at different time points to highlight key temporal information, effectively completing the micro expression detection work [9]. She Wenxiang et al. used parallel neural networks to extract features from local and global regions, and then aggregated the extracted motion and spatial features to reduce the loss of subtle features. Finally, they combined the local and global aggregated features into a new hybrid feature for micro expression recognition, improving the accuracy of micro expression classification [10]. This article mainly elaborates on the commonly used micro expression datasets in micro expression research, and analyzes the advantages and disadvantages of each dataset. Then analyze and extract micro expression features from two algorithms; Finally, analyze the application areas of micro expression research and the challenges faced in future development. 2. Commonly Used Micro Expression Datasets The research and recognition of micro expressions require a large amount of data as support when training neural network models, but publicly available micro expression datasets are limited and each dataset has very few micro expression image samples. When collecting micro expression 47 samples, the emotions of the subjects need to remain neutral, and then micro expressions are spontaneously generated by environmental factors in specific scenes. The annotation of the images after collection is also quite time-consuming and laborious. Therefore, more efforts need to be made in data collection and analysis for micro expression research to further promote the development and maturity of this field. Effective micro expression recognition rate relies on high- quality micro expression datasets and constructed neural network models. In the field of micro expression research, there are mainly three micro expression datasets: MMEW, SAMM, and CASMEII. (1)MMEW dataset [11]: This dataset is a micro macro expression dataset jointly released by a research team from Shandong University in 2021. It contains 300 samples of micro expressions and 900 samples of macro expressions from Asians, with an average age of 22.35 participants. This dataset was collected in a laboratory environment, where each sample was labeled by experts with a start frame, vertex frame, and termination frame, and provided with AU annotation. The resolution is 1920 × 1080 pixels, which provides more detailed sentiment classification compared to other datasets. This dataset contains the following emoticon labels: Happiness (36), Anger (8), Surprise (89), Disgust (72), Fear (16), Sadness (13), and Others (66). (2)SAMM dataset [12]: This dataset was captured in an experimental environment and controlled for lighting to prevent interference with the collected data. It includes 32 subjects from different places and 13 different races, with a total of 159 micro expression samples collected. In addition, the age range of the subjects is from 19 to 57 years old, with an average age of 33.24 years old. Therefore, it can be concluded that the sample sources of this dataset are more diverse and abundant. In addition, the dataset annotated detailed information of the starting frame, vertex frame, and offset frame, using a high-speed camera with 200 frames per second, and the image resolution reached 2040 ×1088 pixels. This dataset contains the following emoticon labels: contempt, disgust, fear, anger, sadness, happiness and surprise. (3)CASMEII dataset [13]: This dataset was created by the Institute of Psychology, Chinese Academy of Sciences, Chinese Academy of Sciences. They shot and collected videos in the experimental environment, and controlled light and sound to prevent interference with the collected data. It is mainly used to study the identification and analysis of micro expressions. During data collection, participants are required to watch emotional stimulation videos that trigger their natural emotions, and use a high-speed camera with a frame rate of 200 fps and a resolution of 640 × 480 pixels to capture the footage. The CASME II dataset categorizes micro expressions into 5 types for annotation, namely happiness, disgust, surprise, depression, and others; In addition, each video clip is also labeled with the start time, end time, category, etc. of the micro expressions for analysis of micro expression classification. 3. Methods for Extracting Micro Expression Features 3.1. LBP-TOP Algorithm (Local Binary Pattern from Three Orthogonal Plans) With the continuous development of facial expression recognition research, more and more scholars have proposed many methods based on traditional feature extraction algorithms and deep learning algorithms to solve the problem of facial expression recognition. Local Binary Pattern (LBP) can effectively handle the effects of lighting changes in images, but this method can only process static two- dimensional images one by one. Therefore, in order to solve the problem of micro expression recognition in videos, a LBP-TOP algorithm [14] for extracting high-dimensional features is proposed, which is an effective dynamic texture feature extraction algorithm that can simultaneously extract spatial and temporal information. In a video or image sequence, in addition to the X and Y directions, there is also a T direction along the timeline, where X-Y is the normal planar image, X-T is the texture scanned along the timeline for each row, and Y-T is the image scanned along the timeline for each column. After extracting LBP features from all three planes, the LBP features of the three planes are connected together to form LBP-TOP. Lu Guanming et al. used the LBP- TOP algorithm to extract features for micro expression recognition, and then used ReliefF and Locally Linear Embedding (LLE) manifold learning algorithms to reduce the dimensionality of the extracted LBP-TOP feature vectors. Finally, a Support Vector Machine (SVM) classifier was used for classification, and good results were achieved through experiments in the CASME II micro expression database [15]. 3.2. Micro Expression Peak Frame Detection Algorithm based on RHOOF Features RHOOF is an algorithm used for micro expression feature extraction. It divides facial regions based on detected keypoint coordinates (such as eyes, eyebrows, mouth, etc.), and then performs HOOF feature extraction on the face within the divided regions to describe the facial muscle movement in each region. Finally, detecting the peak of micro expressions based on changes in facial muscle movement direction is a crucial step in micro expression analysis, which can effectively improve the accuracy of micro expression recognition. Ma Haoyuan et al. proposed a new method RHOOF for extracting micro expression features, which detects facial landmarks and selects 5 regions of interest (ROIs) from facial regions based on the frequency of action units. Finally, the HOOF of these ROIs was calculated and experiments were conducted on the micro expression databases CASME and CASME II [16], achieving good results compared to other methods. 4. Application Scenarios The study of micro expressions is to interpret people's true emotional states by analyzing subtle changes in facial muscles. As micro expressions are often inadvertently revealed when people try to hide their true emotions, micro expression analysis is widely used in the field of psychology. For example, computer vision technology and psychological knowledge can be used to analyze a person's micro expressions to test whether they are lying [17]. In terms of customer service, the emotional state of customers can be analyzed by recognizing their micro expressions. If there is dissatisfaction or confusion, service personnel can adjust service policies in a timely manner. In terms of medical diagnosis, micro expression recognition technology can help medical staff identify subtle emotional changes in patients, thereby quickly and accurately detecting and diagnosing the condition [18]. With the continuous deepening of research on 48 micro expressions, on the one hand, it can enhance the recognition and understanding of human emotional expression, and on the other hand, it can play an important role in human-computer interaction, security monitoring, diplomatic negotiations and other fields. 5. Challenges Facing Future Development At present, there are many technologies for micro expression recognition, and the accuracy of micro expression recognition has been greatly improved. However, the existing publicly available micro expression datasets have extremely limited image sample sizes and types, and the feature extraction of facial muscle movements is relatively difficult, resulting in weak generalization ability of micro expression recognition. In future in-depth research, data augmentation operations can be performed on the data during the preprocessing stage to increase the sample size. In addition, to avoid a single feature extraction method, deep neural network models or a combination of multiple models can be used to extract micro expression features, which can enhance the generalization ability and robustness of the network model in recognizing micro expressions. Acknowledgments Guangdong University of Science and Technology, School level Research Project (Natural Sciences) Youth Project+ GKY- 2023KYQNK-2. References [1] W. J. Yan, Q. Wu, J. Liang, et al. How fast are the leaked facial expressions: The duration of Micro-expression[J]. Journal of Nonverbal Behavior, 2013, 37(4): 217-230. [2] PAUL EKMAN. Darwin, Deception, and Facial Expression[J]. Annals of the New York Academy of Sciences,2003, 1000 (1):205-221. [3] Ekman P, Friesen V W .Nonverbal Leakage and Clues to Deception[J].Psychiatry,2016,32(1):88-106. [4] E. L. Rosenberg, P. Ekman. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS)[M]. USA: Oxford University Press, 2020: 49-62. [5] Ekman P, Friesen W V. Facial Action Coding System. Palo Alto: Consulting Psychologists Press, 1977. [6] EKMAN P. Lie catching and micro- expressions[J].The Philosophy of Deception, 2009, 1(2): 5. [7] Li Y, Wei J, Liu Y, et al. Deep Learning for Micro-Expression Recognition: A Survey[J]. IEEE Transactions on Affective Computing, 2022, 13(4): 2028-2046. [8] Zhang Yanliang, Lu Bing, Jiang Hanxiao, etc Research on the Analysis Method of Correlation between Micro expression Categories and Regions [J] Computer Engineering and Applications, 2020,56 (19): 146-151. [9] Yu Yang, Sun Fangfang, Lv Hua, etc Micro expression detection method based on multi-scale spatiotemporal attention network [J] Computer Engineering, 2024, 50 (6): 228-235. [10] She Wenxiang, Liu Bin, Tao Jianhua, etc Micro expression recognition method based on multi-channel motion feature fusion [J] Journal of Computer Aided Design and Graphics, 2021, 33 (9): 1457-1465. [11] XIANYE BEN, YI REN, JUNPING ZHANG, et al. Video- Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,44(9):5826-5846. [12] Davison A K, Lansley C, Costen N, et al. Samm: A spontaneous micro-facial movement dataset[J]. IEEE transactions on affective computing, 2016, 9(1): 116-129. [13] W. J. Yan, X. Li, S. J. Wang, et al. CASMEII: an improved spontaneous micro expression database and the baseline evaluation[J]. PloS One, 2014, 9(1): e86041. [14] GUOYING ZHAO, MATTI PIETIKAINEN. Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2007,29(6):915-928. [15] Lu Guanming, Yang Cheng, Yang Wenjuan, et al. Micro expression recognition based on LBP-TOP features [J]. Journal of Nanjing University of Posts and Telecommunications (Natural Science Edition), 2017, 37 (06): 1-7. [16] HAOYUAN MA, GAOYUN AN, SHENGJIE WU, et al. A Region Histogram of Oriented Optical Flow (RHOOF) feature for apex frame spotting in micro-expression[C]. //2017 International Symposium on Intelligent Signal Processing and Communication Systems: ISPACS 2017, Xiamen, China, 6-9 November 2017, [v.1]. Institute of Electrical and Electronics Engineers, 2017:281-286. [17] Shen Xunbang, Sui Huajie, Fu Xiaolan The application of micro expressions in deception detection [J] Progress in Psychological Science, 2017, 25 (2): 211-220. [18] Ma Lin, Chen Wenfeng, Fu Xiaolan, etc Facial expression and micro expression recognition in patients with depression [J] Scientific Bulletin, 2018, 63 (20): 2048-2056.