Automatic Commentary Generation For Snooker Game Videos Based On Deep Learning

The continuous development of Internet has promoted the prosperity of video websites.More and more people like to freely watch their favourite programs on the video website.Sports videos are widely enjoyed by the audience and thus are important for industrial applications.However,compared to other types of videos such as movies or TV series,the viewing threshold for sports videos,especially strategic sports videos,are usually high due to their professionalism.Obviously,the use of natural language processing technology to automatically generate sports commentary for sports videos can not only provide professional help to ordinary users,increase the fun of watching videos,but also attract traffic to sports video websites,and has broad application prospects and theoretical value.In recent years,with the rapidly development of deep learning,it has made great progress on image/video understanding and natural language text generation,and at the same time,studies on video captioning have also made certain progress.However,there is still not much research on sports video commentary generation.Moreover,compared to the video caption,the sports video commentary has its particularity: 1)The generation space of sports video commentary is more divergent than that of video description,and the topics involved are more various.The ”correct” text for the corresponding video is not only diverse in expression,but also have many possibilities in semantics.2)Commentaries of sports videos are more professional than descriptions of open domain videos,and the expression of professional terms requires prior knowledge.3)In contrast to describing a video,generating a commentary for a video needs to logically infer some facts from the contextual video contents,which is particularly evident in generating commentaries for strategic sports such as snooker and nine-ball.These features bring great challenges to the automatic generation of sports video commentary.In this study,we use snooker video commentary as an entry point of the exploration of strategic sports video commentary generation.Specifically,the main contributions of this thesis are:(1)We build the first Chinese snooker video commentary dataset.The dataset contains single shot snooker video clips,and corresponding Chinese commentary annotations from professionals.Besides,we further analyze and label the types of snooker commentaries.The Chinese Snooker Commentary Dataset provides data basis for follow-up research.(2)We perform a study on the automatic commentary generation for snooker videos,and propose a systematic algorithm framework of this task.The framework is composed of visual feature extraction module,strategy prediction module and text generation module.Under this framework,we study the various feature representation of snooker video clips,construct a snooker strategy prediction model,and propose two automatic commentary generation methods for snooker game videos——the constrained commentary generation method based on Metropolis Hastings sampling(CCG-MH)and the end to end commentary generation method based on deep learning(SCN).A variety of visual feature representations provide different views of input for commentary generation;the strategy prediction model makes the generated comment not limited to objective descriptions,but can make certain predictions;the CCG-MH method can generate various commentaries that satisfy various constraints on the basis of semantic keywords,and the SCN method conducts joint learning on semantic features and visual features,reducing the training difficulty caused by the opening of the commentary generation space.The experimental results show that the snooker video commentary generation framework proposed in this paper can generate fluent and reasonable commentaries for single shot snooker videos,and is superior to several baseline methods.(3)We implement a B/S based web demo program which is used to display visualization results in the process of snooker video commentary generation,and to compare the results of different commentary generation algorithms.The demonstration program is helpful for researchers to intuitively understand and analyze the generation of snooker commentary.