Group Activity Recognition Based On Hypergraph Neural Network

Group activity recognition aims to identify the collective activity of the entire crowd(group)from a video clip.It can be used in various complex scenes such as live broadcast of sports events and educational classrooms.Therefore,it has attracted wide attention from all walks of life.Group activity recognition methods generally first identify the action of each individual in the video,and then further infer group activity on this basis.The difficulty in the research of group activity recognition lies in how to effectively model the interaction relationship between individuals after identifying individual actions.Most of the previous methods focused on one-to-one modeling between individuals,ignoring that the action of individuals in a social environment may be interacted and jointly influenced by multiple individuals,that is,there is not just a simple one-to-one relationship between individuals,also contains complex many-to-one higher-order relationships.To solve this problem,this paper introduces a hypergraph model to capture the high-order relationships.The main work is as follows:First,the hypergraph structure is introduced into the field of group activity recognition for the first time and a spectral domain hypergraph algorithm(SDH)for group activity recognition is constructed.SDH uses the cosine similarity and KNN algorithm to construct a hypergraph and derives spectral domain hypergraph convolution algorithm based on spectral domain graph convolution algorithm.Each hyperedge in SDH can accommodate the characteristics of multiple nodes to capture the high-order relationships between nodes.Specifically,the hypergraph distance relationship matrix is calculated first.Then the hypergraph correlation matrix is generated.Finally,the spectral domain hypergraph convolution is performed.The experimental results show that the hypergraph model can effectively extract high-order interaction relationships between multiple people.And the comparison with the graph convolution algorithm proves the superiority of the hypergraph model.Second,a group activity recognition network based on the multi-hyperedge hypergraph(MHH)in the spatial domain is proposed.Aiming at the problems that there are multiple high-order influencing factors among individuals,the calculation of hypergraph in spectral domain is complex,and it is difficult to construct an incidence matrix for various hyperedges,a hypergraph convolution algorithm based on spatial domain is proposed.Specifically,MHH defines three kinds of hyperedges that capture different interactive relationships such as visual relationship and spatial position.Then,MHH constructs node convolution and hyperedge convolution operations for node features to propagate and update features.At the same time,the three types of characteristics are effectively fused by late-fusion methods,thus further enhancing the group activity representation.A large number of ablation experiments show the effectiveness of the three types of hyperedge and the proposed spatial hypergraph convolution.Compared with the SDH algorithm,MHH can better capture the high-order relationships between individuals,thereby improving the recognition accuracy.