Deep Learning On Video Based Human Action Recognition

Human Action Recognition has wide application scenarios,including video surveillance,human-computer interaction,intelligent robots,video retrieval,medical diagnosis,elderly care and sports analysis and so on.Therefore,the video based human action recognition problem is widely concerned and studied.In recent years,with the success of deep learning in various computer vision fields,more and more methods based on deep learning have been proposed for human action recognition problem.One of the state-of-the-art methods is the two-stream convolutional neural networks(ConvNets).Imitating the human vision system,the two-stream ConvNets method decomposes videos into spatial component as image frames and temporal component as optical flows and extracts features from both of them by ConvNets.Based on the two-stream ConvNets method,some fusion methods are also proposed to study the spatial-temporal features.However,we found that these methods tend to distinguish actions by short video clips instead of digging the long-term dependences inside actions.We proposed a convolutional recurrent neural network fusion method to model long-term dependency inside actions.This approach uses convolutional recurrent neural network to fuse the two-stream ConvNets so as to learn the evolution of appearance feature and motion feature in the long term.In addition,we introduced the attention mechanism into our fusion method.We proposed RNN attention mechanism and cross-stream mechanism so as to enable the network to learn to focus on areas related to the action at different times.We show that our approaches improve the recognition accuracy of two-stream ConvNets by a large margin through experiments.