Research And Implementation Of Multi-modal Learning For 3D Mapping Facial Landmark Detection

With the increasing demand of face analysis and recognition in the fields of social security,the human-computer interaction,the intelligent services and the research significance and value of face analysis and recognition are gradually reflected.Facial features and contours are the most important parts of the face.How to accurately and efficiently locate these parts that contain most semantic information of the face is an important advance in face analysis and recognition.Now,most facial landmark detection methods only focus on the distribution of the facial landmarks in 2D face mode.These methods pay more attention to the visible parts of the face,which leads to the neglect of the real spatial structure of the face in the case of self-occlusion caused by the large poses.This paper aims to propose a new facial landmark method which can be applied to multiple modes simultaneously.The detection of 2D facial landmarks and self-occlusion 3D mapped facial landmarks is still a challenging problem,especially in the case of large head pose.In this paper,a new deep learning model is proposed to solve the joint detection of 2D face key points and 3D mapped facial landmarks under large poses.The main work of the paper is as follows:1.To solve the problem of large poses in unconstrained environment,an MDP(Markov Decision Process)adjustment strategy is proposed to optimize the initialization sensitivity problem in cascade regression methods.Compared with the traditional cascade regression methods,MDP initialization strategy will learn a set of actions from the reward function.This series of actions will adjust the cropping of images,so that the initial input of the network covers more semantic areas of the face.2.Deep learning requires a large amount of data for the training of neural networks,and the training of multiple different models will greatly increase the time cost.In the third chapter,a parallel multi-task model is established for facial landmarks detection in two modes.For this purpose,a multi-agent co-learning model is designed.In this model,2D facial landmarks detection agent and 3D facial landmarks agent share the public features,so as to enhance feature representation.3.In order to solve the problem of extreme face deformation caused by large poses and expression in multiple modes,the constraint relationship between face regions is proposed in chapter 4.This refined constraint relationship between face regions can reduce the negative influence between deformation regions.In this paper,a relational-structural network is designed to learn both global and local face constraints,and the whole iterative process is optimized by reinforcement learning method.4.An experimental platform system is designed,and the proposed algorithm is integrated,which can realize 2D facial landmarks detection,3D facial landmarks detection and multi-view facial landmarks detection.