| Video captioning aims to automatically generate text describing the content of the given video.To accomplish this task,the model needs to understand the visual content of the video,organize these visual information reasonably,and generate a smooth and grammatical descrip-tion.Video captioning has important applications,which can be used for video retrieval,security monitoring,helping people with visual impairment and so on.Most of the existing models ignore the important relationship between object features and video visual features and motion features.In addition,the existing models use dense frame sequences to simulate the original video.Such data processing will cause the model to extract a large number of redundant features,which seriously affects the quality of the description generated by the model.In response to the above problems,our model has made the following improvements:First,a graph neural network is used to explicitly model the association between object features and action features,visual features.Then the object features of each frame are fused into the action and visual features.The above method can capture long-distance dependencies between features,and obtain action features and visual features enhanced by object features.Secondly,the enhanced video features are mapped to the latent semantic space,the number of nodes in the semantic space is much smaller than the number of nodes in the original feature space.Message propagation is only carried out between nodes in the semantic space,which can effectively reduce the redundancy of feature vectors while preserving video feature information.Thirdly,a selection module is constructed,which will select the feature with the highest score at each time step of LSTM for feature decoding through the scoring mechanism.The model proposed in this thesis has been trained and tested on two datasets,MSVD and MSR-VTT.The test results show that the video captioning model proposed in this thesis can ac-curately describe the objects and their behaviors in the video.In terms of quantitative indicators,the performance of this model on the MSVD dataset is ahead of most mainstream models,espe-cially in ROUGE and CIDEr,which reach 73.91 and 95.99 respectively.Subsequent ablation experiments also demonstrate the effectiveness of each module. |
