Research And Design Of System For Multimodal Verification Of Face And Ear Based On Deep Learning

With the rapid development of artificial intelligence technology,biometric technology has been widely used in fields like education,medical,military,finance,etc.However,in practical applications,biometric systems based on a single biometric is often restricted by many factors,such as noise,coverage and poor anti-spoofing.On the other hand,with the benefits of multiple biometric traits,the multimodal biometric systems are able to alleviate these problems to a certain extent,and improve the recognition performance.Face biometric systems recognize a person in a distance via the face like human beings.Compared with fingerprint and iris recognition,face recognition can be applied in a wider range of applications.However,the face appearance is easily affected by light,posture,expression,make-up,aging,etc.On the contrary,the ear has a stable shape and is hardly affected by the expression,make-up and aging,meanwhile it is near the face and can be captured with the same sensor.Multimodal recognition utilizing face and ear can improve the accuracy and robustness of the system,and allows a wider range of sensor angles.This paper focuses on the deep learning technologies for face and ear detection and their multimodal fusion recognition,and designs an examinee check-in system with the proposed methods.For the face and ear detection,we divide the head image into face,ear and head regions,and annotate an amount of training samples for each class.Then,we train a Mask Scoring R-CNN detector to detect them simultaneously.Due to the small size of human ear and limited training samples,we often get many false ear detection results in an image.To solve this problem,we propose an ear filtering algorithm by using the adjacency and membership among the human ear,face and head,so as to eliminate the false detection and reduce the jitter phenomenon of the ear mask.Our experimental results show that the proposed method achieves evidently improved detection performance for both the face and ear.In the multimodal verification phase,we propose to combine the deep learning based features of face and ear in sparse representation classification framework.We first learn the face and ear features via individual deep neural networks,and then combine them at feature level or score level using sparse representation technique.In our experiments,we study the pros and cons of the fusions at feature level and score level,and the loss function in training deep neural networks.Our results demonstrate that the proposed method significantly improve the verification accuracy,the score level fusion scheme is much more robust than the fusion at feature level,the loss function designed with the idea of improving inter-class separability and intra-class compactness can bring evident performance improvement.Finally,we design and implement an examinee check-in system based on the proposed face and ear detection and multimodal verification methods by using Pytorch and PyQt on an Ubuntu system.The system can automatically detect the face and ear of an examinee andverify his/her identity.The system has a friendly user interface,and is easily to used.In our experience,the recognition is accurate.We hope it can be deployed in real applications in the future.