| With the rapid development of the Artificial Intelligence(AI)video captioning has become a hot research topic of AI.Video captioning is a task for automatically describing the content of a given video with a correct and coherent sentence.It involves both the Computer Vision(CV)and the Natural Language Processing(NLP),and has huge application prospects in life.For example,video captioning can be used to explore the semantics of videos to facilitate the quality of video retrieval.With the success of Convolutional Neural Networks(CNN)and Recurrent Neural Networks(RNN)in the CV and NLP,respectively,the CNN-RNN-based "Enocder-Decoder"structure is widely used in video captioning.However,the Encoder-Decoder structure can only use the forward information which flows from video content to language description,while ignores the backward information which flows from language description to video content.In order to utilize the dual information,a neural structure named"Encoder-Decoder-Reconstructor(RecNet)"which is based on"Encoder-Decoder"structure is proposed in this paper.Specifically,the encoder extracts CNN features for each frame in a video clip,the decoder dynamically assigns weights to each CNN feature using a soft attention mechanism and predicts a word at each step,eventually joining a sentence to describe the video clip.Two types of reconstructor are designed on the top of decoder to reconstruct the global features and local features of the video clips from the hidden state sequences of the decoder,respectively,respectively.Subsequently,the backward information is learned by the reconstructor and provided to the encoder-decoder.Besides,we propose a strategy for fusing the two types of reconstructor,which can reconstruct the global and local semantic information simultaneously.The reconstructor can further model video content and language information to improve the performance of video captioning.The qualitative and quantitative experimental results on three large-scale databases,including MSR-VTT,MSVD and ActivityNetl.3,demonstrate that the proposed RecNet can enhance the performance of video captioning.In addition to the traditional training strategy,this paper also introduces the reinforcement learning algorithm(REINFORCE)to directly optimize the metrics of sentence,such as CIDEr,which further proves that the method generalization on different datasets and can be adapted to different training strategies. |
