Research On Egocentric Activity Recognition Algorithm Based On Hierarchical Deep Fusion Framework

Recently,egocentric activity recognition has attracted considerable attention in the pattern recognition and artificial intelligence communities because of its wide applicability in medical care,smart homes,security monitoring and so on.In the traditional egocentric activity recognition algorithm,the traditional image processing and computer vision technology are mainly used to classify different behaviors,so there are some defects,such as low recognition accuracy,large amount of computation,complex algorithm and so on.Recently,with the continuous development of deep learning technology,it has made a comprehensive breakthrough in various applications of computer vision,so it will bring greater advantages to introduce deep learning technology into egocentric activity recognition.In this paper,we first classify the egocentric activity based on the image flow data using the image deep features,and then choose a better algorithm model by comparing the results of experiments on different data sets.Then we propose and implement the multi stream direct fusion framework and multi stream hierarchical fusion framework for ebutton data set and multi-modal data set.In the two fusion frameworks,different network models are used to model the motion sensor data and image stream data,respectively.Finally,the first person behavior recognition is realized through different fusion strategies.In the multi-stream direct fusion framework,Long Short-Term Memory network(LSTM)and a convolutional neural network + Long Short-Term Memory network(CNNLSTM)are used to model the sensor data and image stream data,respectively,and then pooled according to the average value and Maximum pooling uses two different strategies to directly fuse the classification results of sensor data and image streams.In the multi-stream hierarchical fusion framework,the sensor data is first modeled using Long Short-Term Memory network(LSTM),then convolutional neural network(CNN)and convolutional neural network + Long Short-Term Memory network(CNN-LSTM)are used to model the low frame rate and high frame rate image stream data respectively.In the multi flow hierarchical fusion framework,sensor data is only used to classify the behavior according to the motion state,while photo flow data is used for further specific behavior recognition in the motion state grouping.Therefore,both motion sensor data and image stream data are classified in the most appropriate way to significantly reduce the negative impact on fusion results caused by sensor differences.The experimental results show that the proposed multi flow direct fusion framework has no outstanding advantages over the existing direct fusion framework in recognition accuracy,and the accuracy of multi flow hierarchical fusion framework is 6% higher than the existing direct fusion framework.Moreover,the two fusion frameworks proposed in this paper avoid the time-consuming of the existing methods,so they are more suitable for practical applications.