Multimodal Face Generation And Recognition Based On Generative Adversarial Network

Recently,face recognition technology has been widely used in the security field.Recognizing a face based on its facial sketch is an important,yet challenging problem in the face recognition community.It has widely practical applications in security and law enforcement,since the photo image of a suspect is unavailable in many cases.It would be helpful for the police to narrow down potential suspects quickly and determine the identity of the suspect if we can automatically recognize a face using the sketch.The mechanism of face sketch recognition is to mine the invariant features of sketches and face images,and to achieve identity association through feature matching.A face sketch is generated by the forensic artist or facial composite software based on eye-witness description.The suspect is then matched in the face database to determine the identity of the suspect.But the sketch can only capture facial visual information,however,the natural language descriptions include semantic meaning(gender,race,age,hair color,etc).Few algorithms combine cross-modal information of sketches and attributes to identify faces.Low-level visual sketches and high-level semantic attribute information can complement each other to reduce the uncertainty of generation.To this end,this paper is based on the recognition method of ”Recognition by Generation”,and we propose a Multi-Modal Conditional GAN(MMC-GAN)that includes the multi-modal generator and multi-modal discriminator to generate face images,which can generate a photorealistic face from a sketch along with the descriptive attributes.Among them,the multimodal generator can not only restore the overall structure of a face but also recover key details.In order to better utilize multi-modal information in the generation process,we also propose a fusion method based on residual structure.For the discriminator,the attribute description is also used as a supplementary input to make true and false judgments.At the same time,we also enhanced the identity constraints between the sketch and the face image during the generation process,so that the generated image retains the original identity information.Because the face structure and local details are preserved during the generation process,we can use the generated face image for further recognition.Firstly,we use the theory of metric learning to design a network model of face recognition,and then design a multi-modal task verification and retrieval protocol for the current dataset,and carry out a large number of face verification,retrieval work and sufficient contrast experiments.The experimental results fully validate the effectiveness of our algorithm.