Visual Relationship Detection Based On Neural Networks

Visual relationship detection is a task aiming at mining the information of interactions between the paired objects in the image by describing the image in the form of <subjectpredicate-object>.Also,the task is required to give the bounding boxes of the objects.The task of visual relationship detection is recognized as an intermedia bridge between object detection and higher-level tasks,for instance,image caption.According to the characteristic of the task,several deep models are proposed based on convolutional neural networks and recurrent neural networks.A deep model based on convolutional neural networks is introduced.Most of the previous works regard it as a pure classification problem by taking the integrated triplets as labels of the image;however,the numerous combinations of objects and the diversity of predicates are the tough challenges for these studies.The traditional algorithmic usually gets modest generalization performance on the task of visual relationship detection.Therefore,multi-branch convolutional neural networks are used to classify object and predict predicate simultaneously.And the results prove the effectiveness of the model.Additionally,a model based on a modified bidirectional recurrent neural network(BRNN)is introduced.Although the CNN model do increase the recall of the visual relationship detection,however,the networks overlook the inner connection between the elements of the triplet,leading to a modest result of detection.Therefore,the BRNN is employed to improve the accuracy of the description of the images.Moreover,the existing works are inproved by introducing a paired non-maximum suppression method.The results of the evaluation show the success of the proposed new network.Lastly,a mutiple feature infusion methods is proposed.The network uses a infused feature concatenated by special feature,appearance feature and classification feature.By using the infused feature,the hidden information of the relationship in the image is extracted.The experiments show that the approach is competitive with the state-of-the-art works.