Video Semantic Segmentation Based On Bidirectional Optical Flow Fusion And Image Depth Estimation

Semantic segmentation is one of the most challenging image processing tasks in recent years,which requires a corresponding label for each pixel in the image.With the rapid development of deep learning and the maturity of full convolutional neural networks applications,semantic segmentation tecnologies are making breakthroughs.The accuracy are constantly improving.Video semantic segmentation not only needs the segmentation accuracy ensurence,but also requires the segmentation speed guarantee under real-time condition,which is more difficult to implement.The thesis is focus on video semantic segmentation technology.The video sequence has the characteristic of strong inter-frame information correlation.Traditional image segmentation methods analyze the image frame by frame,which makes insufficient use of the relationship between frames and result in expensive computation.This thesis pay more attention to the motion changes of the key parts in the video sequence,by means of introducing optical flow information into the video semantic segmentation task,which represent the motion relationship of pixels between frames.The information transmission between video frames is realized by optical flow,which reduces the redundant calculation based on the independent image segmentation network and improves the video segmentation speed.To solve the problem of existing overlaps in the optical flow mapping,we propose a bi-directional optical flow fusion module by taking advantage of the symmetry feature of optical flow.Experiments show that this module can effectively eliminate the overlapping problem in the process of optical flow transmission,and can improve the optical flow disappearance caused by the target occlusion and the target static,as well as accuracy enhancement.The test results in the DAVIS single target video segmentation dataset show that our segmentation method of bi-directional optical flow fusion is more accurate than Siam Mask,FAVOS,FEELVOS,RGMP and other classical algorithms,and has significantly better rapidity than OSVOS,PREMVOS and STM algorithms under close accuracy.Optical flow mapping in multi-target sequence will result in pixel position conflict,To solve the problem,a depth estimation network is used to estimate the depth information of images.The images are layered deoneding on the target categories,then are fused and superimposed according to the depth information.These series of operates effectively improve the uncertainty in the process of optical flow mapping and enhance the segmentation accuracy in multi-target image sequences as well.Finally,by means of combining the image semantic segmentation network PANet and bi-directional optical flow fusion module,we realize high quality semantics transferring between frames by only calculate the semantics of a few frames in video semantic segmentation.In addition to ensuring the accuracy of segmentation,we improve the overall speed of segmentation,and realize efficient and fast video semantic segmentation.