Hierarchical LSTM Based On "Individual-group" Association Description For Activity Recognition

Group activity recognition is an important research content in the field of video analysis.Groups often contain a large number of individuals,and there is a relationship among these individuals,therefore modeling structured relationships between people in a scene is an important step toward visual understanding.And in a scene that include group activities,typically only a small subset contributing to group activity.On the basis of summarizing the group activity recognition methods,this thesis considers the important role of individuals in group activity recognition,and mainly solves the following problems in group activity recognition: First,for sparse groups,it is usually necessary to analyze by individual.And the location for the individual target in the group is based on the premise that the individual analyzes the group,so the target location for the individual in the group is a problem that the group activity recognition process needs to solve;Second,there is relationship between individuals in the group,analysis of the relationship between individuals in the group is also a key issue in group activity recognition;Third,identifying key individuals in the group,and explore the impact of key individuals in the group on group activity recognition.In response to these questions,this thesis does the following:(1)For the location of individual targets in the group,this thesis uses the method of combining the inter-frame significance with faster-RCNN network to suppress the individuals outside the group,highlighting the individuals with the sports characteristics in the scene,and thus obtaining the location of all individuals targets in the group;then use the two-stream network to extract the appearance and motion information of the video sequence,and use LSTM network to learn the global description of the video time evolution to achieve accurate individual activity recognition.(2)In view of the influence of individuals in group activity recognition,this thesis proposes a hierarchical LSTM network with “individual-group” association description to analyze group activity.In this thesis,by clustering the location information of individuals in the group for individual grouping,and the hierarchical association network is constructed,that for obtain the scene representation including the relationship between individuals.At the same time,each individual attention score is calculated by using the multi-layer perceptron combined with individual features and scene features.That is to obtain the individual most relevant to the group activity at each moment,and achieve the role of key individuals in group behavior recognition,and the hierarchical LSTM network is constructed to capture the timing information of the video scene,thereby realizing more accurate group activity recognition,experimental results demonstrate that our proposed model is effective on the public dataset.