Research On Deep Neural Network Based Video Temporal Action Localization

Video is a widely used in real life,which is a data carrier for visual information storage.Video understanding is very important to computer vision technologies,while temporal action localization is a challenging problem in video understanding.Temporal action localization can be used in video retrieval on websites,video analysis of public surveillance,video censoring and industrial production supervision.Most existed temporal action localization studies rarely use the action relations contained in videos.This thesis studies the impact of action relations on temporal action localization problem and proposes a new temporal localization model framework of based on deep neural network.First,in order to use relation information in the video snippet features,this thesis designs the self-attention modules based on multi-head attention in the deep network model architecture.The action relations and class-agnostic temporal relations are fused into the features respectively by two branches,so that they can be used in the classification module and temporal attention module.To increase the proposal pool and adjust the temporal attention weights,a hybrid class activation sequence strategy based on adaptive thresholds is proposed during branch merging.Second,this thesis formulates a multiple instance learning based training strategy and trains the model according to the optimization objective.Then,during action proposal generation,this thesis proposes a method to adjust the initial proposals by using the learned action relation weights.After the thresholding strategy and action proposal evaluation,high-quality action proposals are generated,and better temporal action localization results are obtained.Last,extensive comparative experiments on multiple datasets are conducted to evaluate the improvements of the proposed temporal action localization model.Ablation studies are also performed to show the effectiveness of the methods.