Heterogeneous Face Recognition Methods Based On Deep Learning

Heterogeneous face recognition(HFR)aims to identify a person from different facial modalities such as visible and near-infrared images.It has attracted increasing attention due to its wide range of applications in surveillance and law enforcement agencies.The main challenges of HFR lie in the large modality discrepancy caused by different distribution of heterogeneous data,and the over-fitting problem resulted from insufficient training samples in certain modalities.To alleviate the modal discrepancy,many previous methods first extract hand-crafted features,e.g.SIFT(Scale-Invariant Feature Transform)and LBP(Local Binary Patterns),and then project face features of different modalities into latent common subspace.However,those methods obtain unsatisfactory performance and reach the bottleneck due to the limited representation power of hand-crafted features.To describe the highly non-linear relationship of different modalities,some works utilize deep CNN(Convolutional Neural Networks)to extracts more discriminative features and show superior performance compared with these methods based on hand-crafted features.Nevertheless,deep CNN can easily over-fit small HFR datasets due to its massive parameters.In addition,a generic face CNN model itself is not designed for effective modal-invariant feature extraction,which also limits its performance on HFR datasets.In this paper,two effective methods are proposed to deal with the challenges of HFR based on deep CNN.These proposed methods are suitable to extract modal-invariant features for a single facial image in the test phase,which is valuable in practice since off-line feature extraction is possible.Specifically,the proposed methods are as follows:· The Mutual Component Convolutional Neural Networks(MC-CNN),a modal-invariant deep learning framework,is proposed to tackle these two issues simultaneously.The proposed MC-CNN incorporates a generative module,i.e.the Mutual Component Analysis(MCA),into modern deep convolutional neural networks by viewing MCA as a special fully-connected(FC)layer.Based on deep features,this special FC layer is designed to extract modal-independent hidden factors,and is updated according to maximum likelihood analytic formulation instead of back propagation which prevents over-fitting from limited data naturally.In addition,an MCA loss is developed to update the network for modal-invariant feature learning.Extensive experiments show that the MC-CNN outperforms several fine-tuned baseline models significantly.The proposed method achieves the state-of-the-art performance on CASIA NIR-VIS 2.0,CUHK NIR-VIS and?T-D Sketch dataset..A new two-branch network architecture,termed as Residual Compensation Networks(RCN),is proposed to learn separated features for different modalities in HFR.The RCN incorporates a residual compensation(RC)module and a modality discrepancy loss(MD loss)into traditional convolutional neural networks.The RC module reduces modal discrepancy by adding compensation to one of the modalities so that its representation can be close to the other modality.By fixing the pre-trained backbone CNN and only tuning the light RC module,the number of learnable parameters is greatly reduced.Therefore,the RC module of RCN is useful for tackling the challenge of over-fitting.In addition,the MD loss alleviates modal discrepancy by minimizing the cosine distance between different modalities.Furthermore,we explore different architectures and positions for the RC module,and evaluate different transfer learning strategies for HFR.Extensive experiments on IIIT-D Viewed Sketch,Forensic Sketch,CASIA NIR-VIS 2.0 and CUHK NIR-VIS show that the RCN outperforms other state-of-the-art methods significantly.