Expression Recognition Based On The Restoration Of Occluded Face Images In VR Scenarios

In recent years,the application of Virtual Reality(VR)technology in the field of intelligent medicine has been widely concerned,especially in the diagnosis and treatment of depression.VR devices obtain the corresponding relationship between scene and psychology by analyzing the facial expression of the user after viewing the stimulus materials,to diagnoses depression,and then conducts immersion-guided therapy in the virtual environment.Traditional visual expression recognition schemes can't recognize facial expressions through the face images occluded by VR devices,so most of the existing VR devices use electromyographic(EMG)sensors to collect electrical signals generated by facial muscle movements to analyze expression categories.However,the additional sensors will bring discomfort to the wearer,and the limited sampling points of sensors result in low accuracy of facial expression recognition.Therefore,in view of the high demand for user's facial expression information such as VR Intelligent Medical System and the shortcomings of existing VR devices in analyzing facial expression categories through EMG signals,this paper proposes a facial recognition algorithm based on face restoration,which decomposes the problem into two sub-problems: the restoration of VR-occluded facial image and the expression recognition of restored facial image,in order to accurately recognize facial expressions that are occluded by VR devices.The research work of this paper is summarized as follows:Firstly,a method of generating VR scene occluded facial image data set is proposed.Multitask Cascaded Convolutional Networks(MTCNN)is used to detect,align and crop VGGFace2 face data set.Uses Dlib machine learning library to detect 68 face feature points and uses affine transformation to simulate wearing VR devices.11000 face image data are constructed which are occluded by VR device.Among them,10000 groups were used as training set and 1000 groups were used as test set.Secondly,drawing on the idea of image translation,a neural network model of face restoration is designed.The occluded face image and the reference face image are connected together in channel dimension as input.ResNet-50 is used to extract face feature vectors and introduce identity(ID)loss.The restored face image is realistic,with an average Peak Signal to Noise Ratio(PSNR)of 23.20 and an average Structural Similarity(SSIM)of 0.79.Meanwhile,the face identity features are largely preserved.FaceNet is used to calculate the similarity distance.The average similarity distance between restored face and groundtruth is 0.6873,and the average similarity distance between restored face and reference face is 0.8307.Thirdly,based on the existing models,an expression recognition neural network model for small data sets based on frame is designed.The recognition rate of standard CK+ data set is 98.8% by 10 fold cross validation and 94.8% by restoring CK+ data set.By using the fine-tuning method of restoring the face data set with the pre-training of standard data set,the expression recognition rate of the restored CK+ data set is increased to 97.8%.Through the confusion matrix of facial expression seven-classification,the causes of misrecognition are analyzed,and the validity and application prospects of the combination of facial expression recognition model and face restoration model are verified.