Research On Facial Image Analysis In The Wild

Human face conveys rich information, including gender, race, identity, emo-tion and cognitive state, etc. In particular, facial expression is an important non-verbal communication channel. The goal of Facial Image Analysis (FIA) is to decode the contained information by analyzing facial image. As a crucial part of Human-centered Computing, FIA has wide real-world application potentials.Most existing facial image analysis researches limit their scopes to the images taken from controlled lab environment. Lab facial images emphasize the subject or certain aspects of the subject, which is helpful in defining and developing the research topics. However, there are definitely perceivable differences between Lab facial images and those from real world (in-the-wild). When facial image analysis techniques are applied to real-world situations, these differences become obvious, and hinder the performance to a large extent. With this in mind, in this dissertation, we first identify and discuss three major differences between facial image in the lab and in the wild. Then in the following chapters, we respectively propose facial image analysis algorithms, and discuss theoretical problems behind the algorithms.In this dissertation, three major differences are identified and discussed.1) 2D projection and 3D object. Most existing FIA researches treat face in Lab images as 2D projections, while in-the-wild facial images show more characteristics of 3D objects. 2) static image and dynamic process. Most Lab facial images are static images, while in-the-wild facial images are accompanied by constant and informative facial move-ments.3) known domain and unknown domain. Due to the constrains in partici-pants, imaging condition and environment, lab facial images are limited to certain data domain. Therefore, current research works assume training and test data come from a same distribution. Meanwhile, in the real-world scenario, facial images have vast variety, which demands research works that not confined by same distribution assumption.The first part of our work is propose a 3D face reconstruction method works on low quality video.3D face modeling can help subsequent analysis tasks in over-coming interferences caused by illumination and head poses, which is a popular research direction among face recognition and expression analysis. We combine the strengths of image matching and morphable model approaches, and efficiently reconstruct 3D face model from low quality video. The main contributions are three-fold:1) we fuse information from 3D face dataset and input video, and reconstruct realistic 3D face model.2) the algorithm automatically detects face, fits morphable model and generate new 3D model without manual interference.3) the reconstruc-tion process is efficient in both execution time and computational cost.The second part of our work is to propose a facial action unit event detection algorithm. Existing facial expression analysis researches usually work in static images. Nonetheless, a facial expression event is a temporal process, which has its start, ending and duration. The various aspects of the process conveys different mean-ings. We proposed a method based on cascade of tasks (CoT) to model the whole temporal process of facial expression events. The main contributions are twofold. 1) We identify three detection tasks:frame detection, segment detection and tran-sition detection, which cover various aspects of expression events. By sequentially integrating the three detection tasks, we are able to detection complete expression events in video.2) We propose a new event-level metric for detection results evalua-tion. The event-level metric preserves temporal adjacency between frames, and thus is more suitable to real-world applications and natural to human perception.The third part of our work is to investigate the domain adaptation problem in facial expression analysis. Most facial expression researches take the assump-tion that training and test data come from a same distribution. When testing on in-the-wild facial images, the difference with training images (lab image) becomes prominent. This data difference vastly hinders analysis performance, and cannot be neglected. We discuss cross-domain facial expression analysis under domain adap-tation learning framework. Moreover, we discard the same distribution assumption and propose a novel facial expression analysis algorithm with domain adaptation ability.As the last part of our work, we give a real-world application example. We apply facial image analysis techniques in intelligent health care area, and design an assistant diagnosis and evaluation system for depression, based on automated facial expression analysis. The system detects complete expression events in interview video and automatically adapts to participants’ personal data domain, by using algo-rithms discuss in previous chapters. Based on analyzing participants’ facial move-ments, the system highlight the locations and segment that are most worth noting in input video, and then give predictions on depression severity. Through above two forms of output, the system can help clinicians’ diagnosis and evaluate depres-sion. Moreover, the system fuses data from all the participants into a dataset, in which features are extracted and underlying patterns are mined. As opposed to tra-ditional research methods that investigate small scale or isolated participants, the dataset-level analysis approach provides a new way to better understand depres-sion. Based on the depression diagnosis interview video data collected up to now, we perform preliminary data analysis. Our experimental results largely agree with reported studies taken in related behavioral researches.