Research And Application Of Action Recognition Based On Two-Stream Convolutional Network

With the continuous development of computer vision technology,action recognition has become an important research direction in the field of computer vision.Action recognition refers to the analysis of a person or an object in a video or image sequence to identify the behavior or activity being performed.Due to the development and application of depth cameras,the acquisition of human3 D skeleton data has become easy.Human skeleton data can better adapt to complex environments and backgrounds without the influence of texture,background,lighting,occlusion,and other factors that may be unrelated to behavior.In thesis,I conduct research and application of action recognition based on dual-stream network and human skeleton,and design and carry out a series of studies to address the two problems that the whole human skeleton is more difficult to extract association information between body parts and the viewpoint diversity has a greater impact on action recognition performance respectively.Specifically,the main research content and contributions of thesis are summarized as follows:1.Skeleton Partition Graph Convolutional Network study.The human body has parts division,and different parts have different relationships and weights in action,so it is important to extract the association information between different parts.Thesis designs a Skeleton Partition Graph Convolutional Network,and the whole network is based on a two-stream network structure,divided into two parts.One is Part Level Graph Convolutional Network,which performs a learnable partitioning of the input human skeleton into different parts in a data-driven manner,and then extracts spatial information such as relationships and features between body parts,while highlighting the important body parts.The other is a Multiscale Mixed Time Convolutional Networks,which uses dynamic temporal perceptual fields to extract temporal information at different scales.The spatio-temporal information is fused and classified to obtain the action recognition results.The effectiveness of the whole network is proved by the improvement of the studied network in the objective evaluation index of the action recognition algorithm.2.Multi-view Transformation Fusion network research.To address the problem that the diversity of viewpoints of the input data has a large impact on the performance of action recognition,a Multi-viewpoint Transformation Fusion Network is designed,which automatically learns a new set of observation viewpoints,converts the input human skeleton sequence to this new set of viewpoint coordinate system representation,and then fuses the converted set of human skeletons.This allows the classification module to see a more consistent representation of the skeleton under the new viewpoints,and the difficulty in extracting features,fitting them during classification,and training the model is greatly reduced.The Multi-viewpoint Transform Fusion Network can effectively improve the robustness of the action recognition algorithm to multi-view inputs.The effectiveness of the designed network is verified by comparing with other preprocessing strategies,effect visualization and other experiments.3.Design and implementation of a skeleton-based human abnormal action monitoring system.The thesis designs and implements a skeleton-based human abnormal action monitoring system,which can automatically identify human abnormal action,such as fighting,falling,climbing,etc.,so as to provide timely warning and treatment.In order to expand the training dataset and alleviate the problem of lack of dataset,the skeleton dataset is enhanced and expanded using the Eulerian angle rotation method.