Facial Action Unit Recognition Based On Weakly Supervised And Semi-supervised Learning

Facial behavior analysis is one of the most important research topics in the field of affective computing and computer vision.Compared to expression categories,facial action unit(AU)can describe facial behavior more meticulously.Most of the current researchs on AU recognition are based on supervised learning,which needs the facial images with AU annotations.The labeling of AU needs the trained experts.Therefore,manually annotating accurate AU labels for a large number of facial images is time-consuming and laborious,but expression labeling is relatively easier.How to make use of a large number of facial images without AU annotation but with expression label when learning AU recognizer,is a challenging and urgent problem to be solved in the study of facial behavior analysis under big data environment.To this end,this thesis pro-poses AU recognition study based on weakly supervised learning and semi-supervised learning that combines domain knowledge.The details are as follows:1.This thesis proposes weakly supervised AU recognition framework based on domain knowledge.In weakly supervised scenario,there are only expression labels but no AU labels.Therefore,this thesis detailedly summarizes the probabilistic dependen-cies between expression and AU,i.e.,facial domain knowledge,and then digitizes the domain knowledge,i.e.,generates pseudo AU labels from the probabilistic de-pendencies between expression and AU by sampling.Then,this thesis makes the distribution of the predicted AU labels be similar to that of pseudo AU labels under each expression.Three weakly supervised learning methods are proposed as follows:· The first is the proposed weakly supervised AU recognition method based on AU prior model.We capture the joint distribution of all AUs under each expression category through restricted boltzmann machine as the prior distribution of AUs.Then,we learn AU recognizer by maximizing the log likelihood of the output of AU recognizer.· RBM assumes the distribution form of all AUs,which may bring some errors to the estimation of AU distribution.To this end,this thesis also proposes a weakly supervised AU recognition method based on adversarial learning.The pseudo AU samples are regard as real AUs,and the output of AU recognizer is regard as fake AUs.We introduce an AU discriminator,and make the distribution of the predicted AU labels by AU recognizer converge to the distribution of pseudo AU labels through the competition between AU discriminator and recognizer.· The above methods both ignore the mutual assistance between AU recognition task and its dual task(face generation).Further,this thesis proposes a weakly su-pervised AU recognition method based on dual learning.We introduce the dual task of AU recognition based on the first proposed method,and enhance the per-formance of AU recognition through the mutual assistance between the primal task and the dual task.The above three methods have been verified by experiments on the CK+,MMI,and UNBC databases.The performances of three methods are higher than those of the compared two methods,i.e.,HTL and LP-SM,like the method based on dual learning achieves 57.66%improvement over HTL on the CK+database,demonstrating the effectiveness of the proposed methods.2.This thesis proposes dual semi-supervised AU recognition method DSGAN.When partial samples have AU labels,both duality of reconstruction considered by the above weakly supervised dual learning method and probabilistic duality for AU-labeled samples should be considered.This method leverages adversarial learning to make the joint distribution of the input and output of both AU recognizer and face generator similar to the distribution of real paired data,thus utilizing the constraint of probabilistic duality to enhance the performance of AU recognition,and the re-lations in label level and feature level to further assist AU recognition.In addition,this thesis extends the above three weakly supervised AU recognition methods to semi-supervised AU recognition.The experimental results on the CK+,MMI,and UNBC databases demonstrate the effectiveness of the proposed methods,like when the missing rate is 0.1,DSGAN achieves 38.19%improvement over SHTL on the CK+ database.