Researches On Recovering Faces From Low-Quality Images

Face recognition(FR)is one of the major research directions in computer vision.Although great progress has been achieved in recent years along with the rapid development of deep learning based methods and the availability of large-scale face datasets,face recognition is still a challenging problem especially in unconstrained environments.This is due to the fact that most of face images collected in unconstrained environments are of low-quality.There are many interference factors which can significantly decrease the performance of face recognition methods,such as illuminations,occlusions,pose variations and so on.To handle this problem,various methods have been proposed.These methods can be mainly categorised into two groups:(a)Latent Space Learning(LSL)and(b)Analysis-by-Synthesis(AbS).The former one attempts to achieve a feature representation that is robust and invariant to interference factors through latent space learning,while the other aims to synthesize(or reconstruct)an ideal face image by removing the interference factors.In this thesis,we focus on the latter one,i.e.,AbS based methods.Specifically,we propose 3 AbS based methods,aiming to reconstruct ideal faces by removing the undesirable factors(e.g.,illuminations,occlusions,pose variaitions,etc)from the low-quality images.The main contributions of this thesis are summarized as follows:(1)Recovering variations in facial albedo from low resolution imagesA novel framework is proposed to jointly remove undesirable interference factors and perform super resolution.To deal with the interference factors,we use a linear model by utilizing the global structure information.To perform super resolution,a local patch based sparse coding method is applied.We then propose a novel framework by combining the global and local models into a single objective function,which is optimized to obtain high resolution face images without interference factors.In addition,we propose an efficient alternating optimization strategy to address the optimization problem.To the best of our knowledge,we are the first to propose a face image enhancement framework which can jointly remove interference factors and perform face super resolution.Experiments demonstrate that the proposed method can not only effectively remove the interference factors,but also significantly improve the performance of face recognition and clustering.(2)Feature transformation based face frontaliationExisting CNN based face frontalization methods commonly adopt an encoder-decoder architecture network and perform the two main tasks of face frontalization(i.e.,the global structure transformation from non-frontal to frontal and the local texture reconstruction)simultaneously.However,they usually generate blurry outputs.This is because the input(i.e.,the non-frontal face image)is encoded into a pooled representation,then the reconstructed frontal face is obtained by decoding the pooled representation(bottleneck),which leads to detail-losing problem.To solve the blurry reconstruction problem,we propose FTFF-CNN(Feature Transformation based Face Frontalization Convolutional Neural Network),aim at performing global structure transformation and local texture reconstruction in a jointly learning way.Specifically,an identical network(i.e.,the input and output of the network is the identical)is designed to focus on encoding and decoding texture information,while a feature transformation network is proposed to transform the feature of non-frontal face to the frontal one in latent space.In this way,unlike other CNN based methods that map the faces with arbitrary poses into the same latent space by encoder,we build a non-linear mapping between the latent spaces of non-frontal and frontal faces,which eases the training difficulty of the encoder.Experimental results demonstrate the effectiveness of FTFF-CNN in preserving and reconstructing texture details.(3)Appearance flow based face frontaliationTo solve the blurry reconstruction problem in image synthesis,recently,the novel flow-based image synthesis approaches have attracted considerable attentions.The key idea underpinning these methods is to synthesize the desired image by 'moving' pixels from single or multiple input images instead of 'producing' them.Inspired by these flow-based methods,we propose A3F-CNN(Appearance Flow based Face Frontalization Convolutional Neural Network).Unlike other CNN based face frontalization methods that directly 'producing' pixels,A3F-CNN learns to establish the dense correspondence between the non-frontal and frontal faces.Once the correspondence is built,the frontal face can be synthesized by explicitly 'moving'pixels from the non-frontal one.To effectively train A3F-CNN,we propose an appearance flow guided learning strategy.Specifically,we first apply the SIFT-FLOW algorithm offline to establish coarse correspondences between non-frontal and frontal face images.Then the prebuilt correspondences are used to guide the training.Experimental results show that A3F-CNN can synthesize more photorealistic faces than existing methods in both controlled and uncontrolled environments.In summary,in this thesis we propose 3 methods which can recover high-quality faces from low-quality images,including a method that can recover variations in facial albedo from low resolution images and two face frontalization methods.Qualitative and quantitative experimental results demonstrate the effectiveness of these methods in recovering high-quality faces from low-quality images.Moreover,the identifiabilities of recovered face images are also validated in the experiments.