Scene Graph Generation Based On Image Sequence And Deep Learning

The scene graph is a graph structure,which contains the major information of the image.The scene graph can not only include object and relationship information,but also include attributes and caption information.Because of the graph structure,which the scene graph is,the store space can be reduced with the primary information of the image stored.The graph structure is more compatible for computer to retrieval in semantic search tasks.The primary researches of scene graph are focused on the generation,and they only focus on the object and relationship information in single image.The primary targets of the article are showed in the below:1.Extending the semantic information of scene graph,and generate the scene graph which can contain object,relationship and attribute information.2.Extending the source input of the network from single image to sequence images.In the scene graph generation task from single image,we extend the scene graph generate network,and add the attribute detection branch to it.We use VGG-16 as our base model,and use RPN with classifier for object detection.We also use classifier for predicate detection in relationship detection task.Then,we use a GRU-based model for attributes detection.With the help of the three-branch design of our model,we can generate a scene graph with object,relationship and attributes information in the end.Moreover,we use a message passing structure and refine the features between each branch,then we improve the performance in object,relationship and attributes detection tasks,respectively.We compare our method with other existed methods in three sub-tasks of scene graph generation task,our method work well on these tasks and our method is better than other methods.We also do ablation studies on attributes detection task,and the result shows that the performance is improved with the help of our combination of different types of features and the feature refinement struct.In the scene graph generation task from image sequence,we further extend the source input from single image to image sequence.We use a flownet-based method to trace object boxes between image frames.Then,we can determine the same object instance from different predicted boxes in different image frames.We also design two feature refinement structures between frames,which are named message-passing-based method and auto-encoder-based method.Our refinement structure can use the features from different view of the same object to refine the features,and improve the performance in object,relationship and attributes de-tection.To analyze our refinement structure in frames and between frames,we design four variety models and evaluate the performance in object,relationship and attributes detection tasks.We find that all the feature refinement which we designed can improve the perfor-mance in the three tasks,and our final design(switch on feature refinement in frames and between frames)is the best one.We also find the auto-encoder-based method is better than the message-passing-based method,when consider recall and precision simultaneously.