Research On The Neural Representation And Recognition Of Micro-expressions

With the widespread popularity of the American TV series "Lie to Me," microexpressions have quickly gained global attention.As a swift and subtle expression that occurs within 1/25-1/2 of a second,micro-expressions often arise in high-risk situations involving deception,making them a crucial indicator for determining someone’s true intentions.Due to the difficulty of directly discerning micro-expressions with the naked eye,micro-expression recognition methods based on image recognition,expert evaluation,or a combination of both have been widely applied in fields such as mental health diagnosis,judicial assistance,intelligence gathering,and personnel security assessment.However,these methods have significant limitations,such as relying on expert identification of micro-expressions requiring specialized training and being limited by professional knowledge and time costs,making it difficult to be widely promoted for large-scale application;vision-based micro-expression recognition models have limitations,including single evaluation criteria,difficult annotation,and susceptibility to complex environmental influences.Faced with these limitations,it is urgently necessary and important to obtain a more efficient and objective way to recognize microexpressions.To develop a more accurate and convenient recognition method for microexpressions,this paper proposes using electroencephalogram(EEG)brain activity signals for the first time.This approach overcomes the limitations of computer vision technologybased or expert identification methods and makes micro-expression recognition and annotation more efficient and objective.Compared to other neural signals such as magnetic resonance imaging and near-infrared,EEG technology has significant advantages.Firstly,EEG’s millisecond-level temporal resolution can quickly capture brain activity when micro-expressions occur;secondly,EEG directly collects electric signals from the central nervous system of the brain,which is less susceptible to artificial suppression or masking,enabling micro-expression recognition based on objective EEG signals instead of subjective expert evaluations.However,until now,there has been no effective and available dataset for building a micro-expression recognition model based on EEG signals,and the neural representation of the micro-expression process is also unclear.Therefore,this paper first constructs the Southwest University Micro-expressions Database(SWUME)based on EEG signals,examines the neural representation of microexpressions from different perspectives,and compares it with the neural representations of macro-expressions and no expressions.Based on this,machine learning and deep learning methods are used to develop an EEG-based micro-expression recognition method,which verifies the feasibility of recognizing micro-expressions based on EEG signals and establishes a high-performance micro-expression recognition model to evaluate its recognition effect.The main research content and innovative points of this paper are as follows:1.Constructed micro-expression EEG-image dataset(SWUME).Establishing a micro-expression dataset based on EEG signals is an important and difficult issue due to the low natural induction rate of micro-expressions and the high quality requirements for data analysis.To address these issues,a scientific and precise data collection paradigm needs to be designed.This study proposed a new paradigm,SEES(real-time supervision and emotional expression suppression experimental paradigm),which improved upon the suppression-induction paradigm and included three components: first,selecting video stimuli with high arousal;second,recording positive and negative micro-expressions induced using EEG and high-speed camera synchronization technology;third,confirming the EEG-image data pairs when micro-expressions occurred.Using this paradigm,a sufficient amount of micro-expression data based on EEG signals was successfully induced and collected,ultimately constructing the SWUME micro-expression EEGimage dataset needed for research.2.Studied the neural representation of micro-expressions.Due to the unclear neurophysiological activity during micro-expression occurrence,there is a lack of physiological evidence and support for micro-expression recognition based on EEG signals.Therefore,based on the SWUME dataset,this study investigated the neural representation of micro-expression occurrence from various perspectives and compared it with the neural representation of macro-expressions and neutral expressions,identifying distinctive EEG features among the three.Taking into account the unique characteristics of micro-expression EEG data,this study systematically analyzed the neurophysiological activity during micro-expression occurrence from three dimensions:frequency domain,source domain,and brain network,using multiple methods such as power spectral density,source localization,functional connectivity,graph theory,and tensor decomposition.Four different neurophysiological activity characteristics of microexpressions were summarized,laying the foundation for establishing an EEG-based micro-expression recognition model.3.The feasibility of using machine learning methods to recognize micro-expressions has been verified.To address the issue of whether micro-expressions can be recognized solely based on EEG signals,this paper extracts important and highly discriminative neural representation information under micro-expression states-functional connectivity features and graph node efficiency indicators based on the results of micro-expression neural representation research.Machine learning models are used for feature selection and to evaluate the classification performance of micro-expressions compared to macroexpressions and neutral expressions.The study found that:(1)the highest binary classification accuracy for micro-expressions versus macro-expressions was 85.29%,and the highest binary classification accuracy for micro-expressions versus neutral expressions was 81.10%,demonstrating that brain network features can recognize microexpressions;(2)brain network connections and node efficiency in the frontal and temporal areas are more important for micro-expression recognition;and(3)brain network features in the beta frequency band are more important for micro-expression recognition.4.Built a deep learning model for micro-expression recognition based on EEG signals.To further improve the recognition accuracy of micro-expressions and establish a high-performance recognition model,inspired by Transformer networks and functional connectivity features,this article proposes a multi-windows functional connectivity attention network(MFCAN).The model appropriately combines the interaction characteristics of brain regions with dynamic complexity of micro-expressions over time by incorporating Transformer networks,successfully improving the recognition rate of micro-expressions.Firstly,the original EEG signals are transformed into brain functional connections using an embedding block;then,multiple heads of attention mechanism respectively represent time windows of multiple functional connections,calculating the internal attention of each time window and the attention difference between multiple functional connection time windows;finally,after capturing and fusing internal and different relationships,more distinguishable neural representations of micro-expressions are extracted.Experimental results show that the MFCAN model has better performance in micro-expression recognition,achieving minimum,maximum and average classification accuracy of 94.45%,95.52% and 96.10% for the three-classification of micro-expressions,macro-expressions,and neutral expressions,respectively.This successfully builds a deep learning model for micro-expression recognition based on EEG signals and solves the important problem of low accuracy in micro-expression recognition based on EEG signals.In summary,the paper successfully induced micro-expressions and constructed the SWUME dataset based on the new paradigm of micro-expression induction in SEES.Then,by using EEG as a starting point to investigate the neurophysiological activity associated with micro-expression generation,effective neurophysiological indicators for detecting and recognizing micro-expressions were found,and machine learning methods were used to identify micro-expressions based on EEG signals.Finally,a multi-window functional connectivity attention network model(MFCAN)based on Transformer network and functional connectivity features was proposed,which improved the recognition rate of micro-expressions.The research results of the paper will help people understand the neural representation of the process of micro-expression generation and provide evidence for identifying micro-expression states based on EEG signals.The paper developed an objective and effective method for recognizing micro-expressions based on EEG,overcoming the shortcomings of existing micro-expression recognition models based on computer vision technology and expert evaluation,and expanding the application scenarios of micro-expressions.It is believed that in the near future,EEGbased micro-expression recognition systems will be better applied in relevant fields.