| Multimodal identity recognition is a technique to recognize the target identity by fusing multiple biometric features,which can improve the stability and security of identity recognition.In the multimodal identity recognition task,how to extract the unimodal features and choose the appropriate fusion method according to different situations are the difficulties of this task.This dissertation mainly addresses the feature extraction of gait,voice and face identity features under the data set collected by the hidden cart,the fusion methods between different features and the problem of missing modalities in multimodal fusion.In this dissertation,a multimodal fusion model is designed using deep learning and traditional machine methods with the application context of person identity recognition under a hidden cart.Firstly,we study and design the identity recognition model for gait and voice pattern fusion,then design a fusion method under missing modality for the case of missing modality,and finally study the identity recognition model based on DS evidence theory for face,gait and voice pattern fusion.The innovation of the work in the thesis is mainly reflected in the following three aspects:(1)A vocal pattern feature extraction model and fusion method are proposed for gait and vocal pattern features,forming an identity recognition model with gait and vocal pattern fusion.The vocal pattern feature extraction model combines PNCC features with NAM attention and residual networks to extract significant feature information and enhance the robustness of vocal patterns in noisy environments.Also,at the feature level,the vocal pattern is fused with gait features by an improved attention mechanism,which enables the model to fully learn the weight relationship between gait and vocal pattern features.The experimental results show that the proposed vocal pattern feature extraction algorithm and the identity recognition model based on the fusion of feature levels perform better than the unimodal methods in both the unnoised and noised cases,improving the performance and enhancing the robustness of the model.(2)To address the problem of degraded performance of multimodal identity models with missing modes,a missing mode fusion method is proposed and an identity model is formed.The model uses a dense linkage self-encoder to complement the missing multimodal features,and fuses the complemented missing modal features with the missing modalities through Embrace network with improved attention mechanism to improve the robustness of the model.The proposed model is experimented in the three cases of complete modality,20% missing rate and 40% missing rate,and the experimental results show that the model designed in this thesis performs better than other methods in all three cases,and effectively improves the robustness of the model in the missing modality environment.(3)In order to more fully utilize the information collected by the hidden cart,a hybrid level fusion model of face,gait and voice pattern is proposed.The face feature extraction model uses a multi-headed attention mechanism to fuse multiple face features,combines with a capsule network to improve the robustness of the model under transformed perspectives,and performs decision-level fusion of face and gait voice features based on the improved DS evidence theory to finally form a three-modal hybrid-level fusion model.The experimental results show that the face feature extraction model and the trimodal fusion model designed in this thesis have better performance compared with other methods and improve the performance of the model. |
