| The temporal action localization task focuses on predicting the actions in the video and predicting the start and end temporal of this action.The dense video captioning task focuses on coherent descriptions of the multiple events in the video using natural language.The use of graph convolutional networks can filter the irrelevant backgrounds,so this thesis uses graph convolutional networks approach around the tasks of temporal action localization and dense video description,respectively.For the temporal action localization task,this thesis proposes a Fusion Graph Convolutional Network(FGCN),in which there is foreground information,and background information in the video,and previous work tends to ignore part of the background information,and a part of the background information of the video can help predict the foreground information,such as the tennis court in the playing tennis video,this information is called semantic information in this thesis.The method proposed in this thesis is to use the temporal information while keeping the semantic information in the features.In practice use convolutional neural networks to extract temporal information features in videos,construct semantic graphs through the relationship between frames,and use graph convolutional networks to aggregate semantic information features in the video,and finally fuse the two features together to improve the accuracy of the model for event prediction.Relevant validation experiments are conducted on THUMOS14 and Activity Net1.3 datasets,where the ablation experiments prove the effectiveness of the module proposed in this thesis,and the final results of the experiments also show that the model in this thesis can improve the accuracy of event prediction.and outstanding results are achieved in metrics such as m AP@0.5,m AP@0.6 and m AP@0.7 of THUMOS14.For the dense video captioning task,Full Information Graph Convolutional Network(FIGCN)is proposed in this thesis.The current common approach for the dense video captioning task mainly uses convolutional neural networks to extract video features,generate multiple event proposals,and then provide textual descriptions of the highly rated proposals,which ignores the linguistic information,such as word lexicality and phrase structure grammar and generates a raw textual description of the video.In this thesis,we use a two-dimensional convolutional neural network to extract visual information and use linguistic information as input and generate n-grams to enhance the fluency of the generated sentence.For the event proposal network in this thesis,a visual semantic candidate proposal generation network is constructed,and this network is used to generate past and future candidate proposals.A graph is constructed based on the above event proposals,and this information is aggregated using a graph convolutional network,and the aggregated features are fused with the originally generated features and input to the encoder-decoder,which will eventually generate the output of textual description results.In this thesis,extensive experiments were done in the public datasets Activity Net Captions and You Cook-II to validate the effectiveness of the model in this thesis,and the results of the model were evaluated in the latest evaluation metrics,and competitive results were obtained in several metrics of BLEU@N,METEOR,CIDEr and ROUGEL. |
