Neonatal Pain Expression Recognition Based On Deep Learning From Dual-stream Features

Neonatal care is often accompanied by painful operations,and a large number of pain stimuli will have a short-term or long-term impact on the physical and mental health of neonates and lead to irreversible damage.Thus,real-time evaluation of the neonatal pain has practical significance.However,due to the limitation of medical resources,there are some shortcomings in artificial evaluation,such as intermittent attention and subjective judgement.In order to avoid the evaluation bias caused by artificial factors and reduce manual work,this thesis studies neonatal pain expression recognition based on dual-stream deep learning.The main research contents are as follows:Neonatal pain expression database is built.Firstly,high quality videos of neonatal pain expression are shot.Then the videos are labeled by medical staff,which can be divided into four categories: calm,crying,mild pain and severe pain.The video and image databases suitable for model training are established after preprocessing.The method of neonatal pain expression image classification based on transfer learning and deep learning is studied.On neonatal pain expression image database,three transfer learning methods are used to compare the performance of AlexNet,VGG16,Inception-V3,ResNet-50,Xception models transferred from ImageNet features and VGG-Face model transferred from the VGG-Face features.The best model for test set is the VGG-Face model trained top layers only,and its recognition rate can reach 79.2%.The results show that the deep model has stronger discriminant ability than the shallow one,which is helpful to solve complex tasks.The appropriate transfer learning method according to the data size can alleviate the problem of data scarcity and restrain over-fitting.Transferring from similar domain can accelerate convergence and improve the recognition rate.A neonatal pain expression recognition method based on temporal and spatial features was studied.For the task of this thesis,a suitable 3D convolution neural network is designed,and the video dataset is preprocessed for training.The performance of 3D convolution neural network and VGG-Face model based on spatial features are compared.Experiments show that the 3D convolution neural network with concision structure and compact features can extract spatial and temporal features well.The test recognition rate of 3D convolution neural network is 59.27%.The two models have their own advantages and disadvantages.The spatial model VGG-Face based on transfer learning has better recognition effect on calm,crying and mild pain,while the 3D convolution neural network based on spatio temporal model has better recognition effect on severe pain.A novel neonatal pain expression recognition method based on dual-stream features with deep learning is proposed.The model based on spatial information can achieve high image recognition rate through transfer learning,and the model based on temporal series can acquire inter-frame relations.In order to combine the advantages of the two models,a method of fusion of VGG-Face model and 3D convolution neural network is proposed.When the fusion coefficient is 0.5,the best fusion effect is obtained.The test accuracy of the test set is 60.96%,which is 1.69% higher than that of the 3D convolution neural network.Finally,A real-time neonatal pain expression recognition system is developed.The system consists of three parts: detection module,recognition module and system interface.The detection module uses Dlib tool library for face detection and landmarks detection.Based on this,the face part is preprocessing by interception and alignment,then the processed images are sent to the recognition module for prediction.The recognition module uses the MobileNet model to balance the recognition rate and real-time performance.The recognition results are displayed visually.