Visual Relationship Detection Based On Object Pair Sifting And Joint Predicate Recognition

With the rapid development of deep learning,especially convolutional neural networks,problems related to computer vision have become very active research fields in recent years.Nowadays,the object detection and recognition algorithm has been relatively mature.Further understanding and exploration of image content has become the trend of the times,and visual relationship detection has emerged.The task of visual relationship detection is to find the visual relationship represented by the<subject-predicate-object>triplet from the picture and mark its corresponding position.Since the number of possible relationships is very large,how to learn a large number of possible relationships from a limited sample becomes a difficult task.In recent years,scholars have carried out a lot of research on this issue,including methods based on language prior,methods based on knowledge representation learning and methods based on statistical dependence.This thesis considers that the same predicate may vary greatly in different scenarios,and the phenomenon that the number of obj ect pairs in a graph may be much larger than the number of object pairs in the actual existence relationship.For the different subtasks of visual relationship detection,this thesis proposes a predicate recognition algorithm based on multi-feature joint statistical prediction and a visual relationship detection algorithm based on obj ect pair filtering and joint predicate recognition,respectively.The main work of this thesis includes:1)A predicate recognition algoritlhm based on multi-feature joint statistical predictionAiming at the predicate recognition task in the case of known subject and object labels and positions,considering the problem that the performance difference of the same predicate may be large,this thesis proposes a predicate recognition algorithm based on multi-feature joint statistical prediction.The algorithm considers a variety of information to jointly predict,combines visual features,location features and semantic features to obtain joint features,and then combines statistical dependencies for joint prediction.In the joint statistical forecasting,the existing statistical-based method does not take into account the importance of different statistical components.Therefore,thisthesis makes a distinction between the importance degree of different components,uses different components as different channels,and uses 1×1 convolution,which can make channel fusion to incorporate this idea into the network.This thesis also uses the method of joint training of cross entropy loss and center loss,which aims to make the difference of joint features between different predicates larger,and the joint features of the same predicate are more similar.Through experiments and comparisons on the public dataset,the results show that the Recall@100 index of the method can reach 85.82%,which is 3.92%higher than the existing DR-Net algorithm,indicating that the method can get better result of predicate recognition task.2)A visual relationship detection algorithm based on object pair sifting and joint predicate recognitionFor the phrase detection and relationship detection tasks in visual relationship detection,considering the phenomenon that there is no relationships between many detected object pairs in the picture,this thesis proposes a visual relationship detection algorithm based on object pair sifting and joint predicate recognition.The visual relationship detection is divided into three stages-object detection,object pair sifting and predicate recognition stage.Firstly,the objects in image are detected,and then the pair sifting model is used to determine whether there is a possible relationship between two obj ects.The pair sifting model is mainly based on the relative position information and semantic information of the obj ect pair.Finally,the proposed predicate recognition algorithm is used to identify the predicate of related pairs to determine the relationship between them.This thesis firstly conducts experiments and evaluations on the pair sifting model,and then tests the overall visual relationship detection algorithm and tests it on the visual relationship detection public data set.By comparing with the existing methods,it can be found that the algorithm can achieve certain effect improvement.