Research On Video Semantic Segmentation In Compressed Domain

Semantic segmentation is a basic task in the field of computer vision.It aims to assist the computer to understand the scene by classifying the pixels of the image.It has a very wide range of applications in real life.In recent years,breakthroughs have been made in image semantic segmentation tasks.Image semantic segmentation algorithms based on deep learning methods have made unprecedented progress in segmentation accuracy and inference delay.However,in most application scenarios,the image acquisition unit collects a continuous image sequence,and this sequence contains important space-time correlation information.Directly using image semantic segmentation algorithms to process image sequences will ignore this correlation,bring huge computing requirements and high inference delay,and affect the application of semantic segmentation algorithms in practical problems.Aiming at the problem that processing the original image sequence directly leads to the complexity of the algorithm calculation,this thesis proposes an algorithm for video semantic segmentation in the video compression domain.The algorithm is based on video compression stream,and the semantic segmentation results are obtained by partial decoding of video stream.According to the characteristics of video stream structure,the video frame is divided into key frame and non-key frame,and different methods are adopted.When processing the key frame,the corresponding video stream is fully decoded to obtain the original image,and then the semantic segmentation network is used to segment the image.For non-keyframe,this thesis obtains the motion vector from the code stream,and generates the non-keyframe feature image by feature propagation combined with the keyframe feature image,and finally generates the corresponding segmentation image based on the nonkeyframe feature image.Since the motion vector obtained from the video bitstream contains noise,this thesis uses the method of median filtering to process the motion vector.In order to verify the performance of the proposed compressed domain video semantic segmentation method,experiments carried out on Cityscapes dataset and Cam Vid dataset.The experimental results show that,compared with the frame-by-frame model,the method proposed in this thesis has a large increase in reasoning speed without a significant decrease in the segmentation accuracy.Compared with DFF,which adopts optical flow to carry out feature propagation,the proposed method has performance advantages in both precision and speed of segmentation.The experimental result strongly demonstrate that the proposed method can utilize video compressed domain information to realize semantic segmentation of video reasonably and effectively.