Temporal Convolutional Network Based Temporal Action Detection

Due to the continuously booming of videos on the internet,video content analysis methods have attracted wide attention from both industry and academic field in recently years.Video content analysis methods have huge application requirements in video recommendation,video auditing,intelligent monitoring,human-computer interaction,assistant driving and other fields.An important branch of video content analysis is action recognition,which usually aims at classifying the categories of manually trimmed video clips.However,videos in real scenarios are usually long,untrimmed and contain multiple action instances along with irrelevant contents.This problem requires algorithms for another challenging task: temporal action detection,which aims to detect action instances in untrimmed video including both temporal boundaries and action classes.It can be applied in many areas such as video highlight and recommendation,smart surveillance and smart retail.For temporal action detection task,many state-of-the-art methods adopt the “detection by classification” framework: first do proposal,and then classify proposals.The main drawback of this framework is that the boundaries of action instance proposals have been fixed during the classification step.To address this issue,we propose a novel Single Shot Action Detector(SSAD)network based on 1D temporal convolutional layers to skip the proposal generation step via directly detecting action instances in untrimmed video.Then,we propose that the key bottleneck of current temporal action detection methods is the quality of temporal action proposal.Thus,we introduce an effective proposal generation method,named Boundary-Sensitive Network(BSN),which adopts “local to global" fashion.Locally,BSN first locates temporal boundaries with high probabilities,then directly combines these boundaries as proposals.Globally,with Boundary-Sensitive Proposal feature,BSN retrieves proposals by evaluating the confidence of whether a proposal contains an action within its region.Experiments on two challenging datasets show BSN can outperforms other state-of-the-art methods with both high recall and high temporal precision.