Deep Learning Methods For Face Detection And 3D Reconstruction

Image based face analysis technique is a very important task in computer vision area,which can be used in many applications,such as face swipping,movie animation and medical beauty.Face detection is the key preprocessing process in face analysis,because we first need detect faces from images,and then we do face analysis.We need handle images with occlusion,tiny size,blurry conditions,which is challenging for face detection task.So,in this paper,we need handle the face detection first.The 3D faces contain information that invariant to poses and illuminations.While reconstructing 3D faces from hardware is not applicable in daily life.So we propose methods to recon-struct 3D faces from images.For face detection,recent studies witnessed that deep convolutional neural network(i.e CNN)significantly improve the performance of face detection in the wild.However,detecting faces with small scales,large pose variations,and occlusions is still challeng-ing.In this paper,to detect challenging faces,we improve two key components in face detector:proposal generator and classifier.To enhance the proposal generator,we pro-pose enhanced region proposal network(RPN).To improve the classifier,we propose Online Hard Proposal Mining and Offline Hard Image Mining.Experimental results on the FDDB,WIDER FACE,Pascal Faces,and AFW datasets show that our method sig-nificantly improves the Faster-RCNN face detector and achieves performance superior or comparable to the state-of-the-art face detectors.After detect faces from images,we can face analysis of 3D face reconstruction.For 3D face reconstruction,we give studies in three subtasks:(1)Detailed 3D face re-construction,(2)3D facial texture reconstruction and(3)Joint 3D shape and texture reconstruction.For(1)detailed 3D face reconstruction is challenging due to its ill-posed property.The statistic model based methods cannot capture detail structures.What's more,there is not exist public available RGB-D dataset for deep model train-ing.This paper proposes a deep Dense-Fine-Finer Network(DF~2Net)to address this challenging problem.DF~2Net decomposes the reconstruction process into three stages,each of which is processed by an elaborately-designed network.In addition,we intro-duce three types of data to train these networks,including 3D model synthetic data,2D image reconstructed data,and fine facial images.Qualitative evaluation indicates that our DF~2Net can effectively reconstruct subtle facial details such as small crow's feet and wrinkles.Our DF~2Net achieves performance superior or comparable to state-of-the-art algorithms in qualitative and quantitative analyses on real-world images and the BU3DFE dataset.Texture reconstruction receive less attention compared with face shape reconstruc-tion due to the unavailability of large-scale training datasets and the low representation ability of previous statistical texture models(e.g.3DMM).In this paper,we introduce a novel deep architecture trained by self-supervision with multi-view setup,to reconstruct3D facial texture.To train the architecture with self-supervised fashion,we propose a novel multi-view consistency loss that ensures consistent photometric,face identity,3DMM identity,and UV texture among multi-view facial images.Extensive experi-ments show that our method achieves state-of-the-art performance in both qualitative and quantitative comparisons.Now we reconstruct 3D facial shape and texture jointly from single image.For(3)Joint 3D shape and texture reconstruction,recent years witnessed significant progress on imagebased 3D face reconstruction using deep convolutional neural networks.How-ever,current reconstruction methods often perform improperly at inaccurate correspon-dence between 2D input image and 3D face template,putting a curb on its real ap-plications.This is because the statistics based model reconstruct over ten thousands of vertices with vectors with few hundreds dimensions.Which lead to the significant2D-3D errors.Thought dense based 3D face reconstruction methods have been pro-posed,they are also low-dimensional methods due to their training dataset.To address these problems,we propose a deep Shape Reconstruction and Texture Completion Net-work(SRTC-Net),which jointly reconstructs 3D facial geometry and completes texture with correspondence from a single input face image.Precisely,the SRTC-Net pipeline consists of three elaborately-designed stages.For establish accurate 2D-3D correspon-dence,in the first stage,we decompose this hard problem into two more trackable prob-lems:image segmentation and PNCC prediction.With accurate 2D-3D correspon-dence,we can obtain high quality face texture.We also mining structure cues in 3D face texture to capture detailed 3D shapes.We examine our methods on 3D reconstruc-tion task as well as face frontilization,pose invariant face recognition tasks,using both in-the-lab datasets(MICC,Multi PIE)and in-the-wild datasets(CFP).The qualitative and quantitative results demonstrate the effectiveness of our methods on inferring 3D facial geometry and complete texture which outperform or comparable to recent SOTA methods.