Occluded Facial Expression Recognition And Micro-expression Recognition Enhanced By Privileged Information

Facial expression analysis is one of the most important research topics in the field of affective computing and computer vision.The existing researches on facial expres-sion analysis focus on extracting more discriminative features and training more robust classifiers,while ignore the influence of privileged information on facial expression analysis.Privileged information is auxiliary information,which is only available dur-ing training phase,but not be available during testing phase.Making use of privileged information can improve the performance of classifier.The thesis focuses on two is-sues related to facial expression analysis:occluded expression recognition and micro-expression recognition.This thesis adopts the learning framework based on privileged information.This thesis proposes to use non-occluded expression as privileged infor-mation to assist the training process of occluded expression recognition,and proposes to use macro-expression as privileged information to assist the training process of micro-expression recognition.Therefore,this thesis proposes an occluded facial expression recognition and micro-expression recognition method enhanced by privileged informa-tion.The details are as follows:1.This thesis proposes an occluded expression recognition method based on cur-riculum learning.The proposed method uses unpaired non-occluded expression images as privileged information to assist the learning process of occluded ex-pression classifier in the feature and label space.This method uses distribution density to measure and rank the complexity of non-occluded images,and designs a curriculum learning strategy to split non-occluded images into three subsets.This allows the occluded expression classifier to first learn basic but clear visual features from easy samples,serving as the fundamental features.Then the oc-cluded expression classifier learns more meaningful and discriminative features from harder samples.This method uses adversarial learning in global-level and local-level feature spaces,forcing the distribution of the occluded features to be close to the distribution of non-occluded features.This method also uses loss inequality regularization in the label space.The experimental results on both synthesized occluded databases and realistic occluded databases demonstrate the effective of the method.2.This thesis proposes a micro-expression recognition method based on feature dis-entangle network.The proposed method uses macro-expression samples as priv-ileged information to assist the learning process of micro-expression recognition.This method first introduces expression-identity disentangle network as the fea-ture extractor to disentangle expression-related features from micro-expression and macro-expression samples.This method uses adversarial learning and triplet loss in the feature space,and loss inequality regularization in the label space to assist the training of micro-expression classifier.The experimental results on the SMIC,CASME ? and SAMM databases demonstrate the effective of the method.3.This thesis proposes a micro-expression recognition method based on spatial and temporal pattern.The proposed method transfers spatial and temporal pattern ex-isted in macro-expression to micro-expression recognition,which is an improve-ment of the second method.In spatial domain,this method uses adversarial learn-ing to align the static texture features of micro-expression and macro-expression.In temporal domain,this method introduces relation classification task,forcing micro-expression classifier to learn the dynamic pattern of muscles from macro-expression.In spatial-temporal domain,this method introduces contrastive loss to pull same class samples together in the feature space,while simultaneously pushing apart clusters of samples from different classes.The experimental re-sults on the SMIC,CASME ? and SAMM databases demonstrate the effective of the method.In summary,this thesis adopts the learning framework based on privileged infor-mation,and further explores the inner connection between occluded expression and non-occluded expression,micro-expression and macro-expression.This thesis provides a new solution direction for the research of facial expression analysis,and improves oc-cluded expression recognition and micro-expression recognition.