Graph Neural Network Based Interaction Learning

Modeling the interaction between objects in the same scene is essential for visual understanding and motion behavior modeling.In the static scene,the specific and clear interactions between the objects together form the abstract and complex events in the scene.In the dynamic scene,the objects cooperate to avoid collisions or carry out group activities.Here,graph-neural-networks-based message passing neural network framework is introduced to consider the interaction between objects in the same scene.We design algorithms for visual relationship detection and motion prediction.Besides,further experiments give the interpretative analysis of interaction learning.Visual relationship detection task aims to detect the visual relationships between objects in the scene.Based on the graph-neural-networks-based message-passing-neural-network-framework,we design the neural message passing core based visual relationship detection algorithm.Considering the directivity of the visual relation triplet <subject-predicate-object> and the correlation between triplets in the same scene,a directed interaction graph is constructed for each scene,where the nodes are objects within the scene(subject,object),the edges are the interactions between the objects(predicate),and a neural message passing algorithm is proposed to explicitly model the nodes and edge features.We further integrate language priors and spatial cues to rule out unrealistic interactions and capture spatial interactions.Experiments on public benchmark dataset VRD show our algorithm improves by more than 12.0 % and 6.9 % respectively compared with previous algorithms.Motion prediction task aims to forecast the future motions for the actors in the scene based on their previous trajectories.Based on the graph-neuralnetworks-based message-passing-neural-network-framework,we design the neural motion message passing core based visual relationship detection algorithm.Considering that the motion of an object is jointly determined by its own moving pattern and interaction with others in the scene,we construct an individual branch and an interactive branch,which two work together to obtain the final trajectory prediction.Considering the asymmetry of the interaction between the objects,a directed interaction graph is constructed for each scene,where the nodes are the objects in the scene,and the edges are the interactions between the objects.A neural motion message passing algorithm is proposed to explicitly model the nodes and edge features.Further experiments give the interpretative analysis of interaction learning.In addition,considering the uniqueness of different scenarios,we design systems for pedestrian motion prediction in open scenarios and joint pedestrian and vehicle motion prediction in urban traffic scenarios.The experimental results show that both systems outperform the previous state-of-the-art methods on several existing benchmarks.Our algorithm improves ADE/FDE to0.41/0.82 m on ETH-UCY datasets;improves Standford-Drone by 9.8% and9.1%.On the joint dataset Nuscenes,our algorithm improves the ADE by10.5% and 7.2% for pedestrian and vehicle respectively.