Key-Frame Based Multi-Feature Fusion Human Action Recognition System

Human action recognition is one of the core supporting technologies with many applications,for example,intelligent video monitoring and human-computer interaction.It has been widely studied in recent years.Most of currently existing methods based on video are using original video or equal sampling to extract RGB image features and optical flow features of video for action recognition,which is easily affected by the redundancy of video information,and with the Increasing diversity of the research object,it is difficult for RGB image features and optical flow features to express all the information in the video.In order to solve the above problems,in this paper,we focus on video key-frame extraction methodology and video feature expression,and proposed a key-frame based multi-feature fusion human action recognition method,and finally construct a prototype system based on the method.The main contents are as follows:1.To dress the problem of video information redundancy in human action recognition,we proposes a key-frame extraction method with the contribution of joint point weights.Firstly,our method constructs video feature descriptors based on human joint point information,and combines with the activity level of joint point during human body moving.Therefore,a joint similarity algorithm based on joint weight is proposed.Then according to the similarity between adjacent frames,the video is divided to obtain the initial cluster.Finally,using K-means clustering algorithm to extract the video key-frame.Compared with the traditional key-frame extraction method,our method refines the difference between actions and ensures the temporality of key frames,both the fidelity rate and the recall rate has improved through our method.2.A single feature is insufficient for expression video information.Aiming at this problem,we propose a key-frame based multi-feature fusion human action recognition.The method combines with RGB image,optical flow information,and skeleton features for human action recognition.Firstly,we use Resnet152 to construct the deep two-stream CNN model to extract the RGB features and optical flow characteristics of the video.Then use the spatial-temporal graph convolutional network model to extract the human skeleton features.Finally,the softmax layer outputs of the two models are merged to construct a key-frame based multi-feature fusion human action recognition model.The verification results on the UCF101 dataset show that compared with the single feature human action recognition our method improves the performance by 7.42% and more.3.According to the above,in order to verify the feasibility of our method,a key-frame based multi-feature fusion human action recognition system is constructed by using PyQT development framework on Ubuntu16.04 platform.The main functions of the system include file selection,video preprocessing,pose estimation,key-frame extraction,and action recognition.