Human Action Recognition With Pose Estimation And Body Segmentation

Human action recognition,one of the most fundamental challenges in computer vision,has wide applications in video surveillance,virtual reality and automatic driving.Our research mainly focuses on RGB video streams due to massive data storage and outdoor detection,compared to depth images.State-of-the-art action recognition methods based on deep learning divide in two steams: CNN for spatial feature extraction and RNN for temporal information learning,joint feature learning methods like 3D-CNN and CNN-Pooling.While most research lay stress on model structure by optimizing network layers and fine-tuning dataset,multiple visual cues like objects,motion,gestures i.e.the has been ignored.Hence,how to choose proper visual features would be the new challenge.To address the problems above,we propose a multi-stream model that integrates raw image,optical flows and human body segmentations.The spatial and temporal features are learned respectively by CNN and LSTM,and achieve overall result by late model fusion.The contributions of the paper can be concluded as:1.We analyze the difficulty in pose estimation problem,and apply open-pose and kalman-filtering methods to multi-operator detection and tracing in production line.2.We use deep encoder-decoder structure to solve body segmentation problem and apply CRF optimization method to CNN network.Experiments on MPII and AI Challenger dataset show great performance of this method.3.We propose a multi-stream architecture for human action recognition.Optical flow learned from deep neural network and body segmentation are thrown to CNN and RNN as inputs along with raw images.Experimental results on the public datasets HMDB51 and UCF101 show that human gestures and optical flow information help improve the overall accuracy.