Research On Video-based Human Action Recognition And Prediction

The improvement of computer vision and the rapid raising of artificial intelligence has aroused people to develop the computer to understand the contents in videos as human beings,especially the action and the intention of humans,which achieves the human-robot interaction and provides convenience for people.The video includes many modalities such as RGB video,depth video,skeleton sequence,etc.In the comprehension of videos,there are two critical tasks for humans:1)action recognition,which aims to recognize the action categories from the complete videos,and 2)action prediction,which aims to predict the actions based on the partially observed videos.However,the disadvantage of the existing methods for action recognition based on the shallow model,which usually utilizes the hand-crafted features,has limited the representation of the actions.Although the current methods for action prediction make full use of the partially observed sequences of RGB videos,there are still two disadvantages,which are also two significant points for the task of action prediction.1)The structure information of the human body is crucial for action prediction that is ignored by the existing methods.2)The subsequent sequence of the observed sequence is an essential cue for representing the ongoing actions that are not exploited during the model design.Depending on these problems,the dissertation makes four significant contributions,which are:1.A method of learning-based principal orientations and residual descriptors(PORD)in an unsupervised manner for action recognition is proposed.The proposed model aims to learn a projection weight matrix consisting of several principal orientation vectors by minimizing the loss of quantization.By learning the prin-cipal orientations,the projected features can be clustered to the nearest principal orientations,making the locally learned features more informative.By maximiz-ing the variance of the residual vectors,the extracted local features are discrimina-tive.The experimental results have demonstrated the effectiveness of the proposed method.2.A method of part-activated deep reinforcement learning method(PA-DRL)for action prediction is proposed.Based on the structural information of the human body,the action-related parts of the human body are activated by skeleton propos-als,and the noise parts are deactivated by deep reinforcement learning.Depending on the skeleton of the human body,the features decided by the joints of the skele-ton in region proposals are extracted.At the same time,a part-activated policy is learned for activating and deactivating the parts of features with deep reinforce-ment learning by the proposed method.The experiments on popular datasets have demonstrated the effectiveness of the proposed method.3.A method of recurrent semantic preserving generation(RSPG)method for action prediction is put forward.The aim of RSPG to predict the ongoing action by complementing the subsequent action without utilizing the observation ratio.In the RSPG,the method designs two crucial criteria to preserve the semantics of generation being consistent with the observed sequence and decrease the inter-ference from the inappropriate generation length.The dissertation presents the experiments and demonstrates the effectiveness of the proposed method.4.A method of ambiguousness-aware state evolution(AASE)method is put forward to exploit the information of the observed partial sequence and evolve the subse-quent skeletons to the most reasonable full-length sequence for action prediction.The method aims to develop an evolution method to predict the most probable ac-tion by generating the reasonable candidate subsequent actions so that the ambi-guity of partial actions can be effectively alleviated.The dissertation provides the experimental results for demonstrating the effectiveness of the proposed method.