Research On Binocular Scene Flow Calculation Technology Based On Semantic Segmentation

3D scene flow is defined as an instantaneous three-dimensional motion vector of visible pixels on the surface of the scene in the real world.Compared with the optical flow representing the two-dimensional motion vector,the scene flow additionally provides the third-dimensional depth change information,which not only contains the geometric information of the object in the scene,but also can establish the correlation of the dynamic scene in the time domain.The purpose of researching scene flow calculation is to recover the geometric shape and the three-dimensional motion in the dynamic scene through the stereo image sequence.Therefore,the 3D scene flow calculation and related technology research have been widely concerned,and have become the hot issues in the fields of computer vision,pattern recognition and image processing.The research results are widely used in many areas such as UAV navigation and the system of takeoff and landing,autonomous driving,virtual reality and augmented reality,medical image analysis,and robotic technology.At present,the accuracy and reliability of 3D scene flow calculation have been greatly improved.However,when the image sequence contains difficult motions and complex scenes such as motion occlusion,large displacement,and illumination changes,the scene flow estimation results often have edge blurring phenomenon,resulting in that scene flow has poor robustness.Aiming at the above problems,this paper mainly studies the binocular scene flow calculation technology based on semantic segmentation.Firstly,the semantic information is used to model the different semantic regions in the image scene and optimize the segmentation boundary.Then,the occlusion pixels are extracted according to the semantic label,and the initial motion parameters are optimized in segments,which effectively improves the accuracy and robustness of the scene flow calculation in the motion edge and the occlusion region.The main research contents and innovations of this paper include:1.This paper proposes an initial segmentation model of scene flow based on semantic segmentation.Semantic segmentation divides the image into different regions,and solves the semantic segmentation optical flow of different regions according to the semantic information.We use the semantic segmentation optical flow as the initial scene flow input,which not only can reflect the motion form of different semantic segmentation regions,but also can obtain better object segmentation boundaries.2.This paper constructs an optimized energy function based on semantic segmentation constraints.The optimization energy function is mainly optimized for the motion parameters of the initial scene flow results.This paper adds semantic segmentation in the optimization energy function,which can play the role of motion constraint.The motion inside the same semantic label tends to be consistent,which can protect the motion edge information better.3.This paper proposes a method based on semantic label for occlusion detection and optimization.In this paper,occlusion reasoning is divided into two steps.The first step is to judge whether the pixel is occluded by comparing the semantic labels of the pixels in different views.If the semantic labels are different,the occlusion is imposed,and the semantic occlusion penalty is applied.If the semantic labels are the same,the second step is performed;In the second step,the occlusion is determined according to the assignment relationship of the pixels.If the assignment relationship of the pixels in different views changes,indicating that occlusion occurs and the internal occlusion penalty is applied.If the assignment relationship remains unchanged,no occlusion occurs.In this paper,the occlusion inference process of image pixels can better deal with the occlusion problem caused by the position change of three-dimensional objects.4.Our method and representative methods are compared by using the standard testing image sets provided by KITTI and MPI Sintel databases.Experiments show that the proposed method can effectively deal with difficult motions and complex scenes such as motion occlusion,large displacement and illumination changes.The proposed method can also effectively protect the edge of image motion and has better accuracy and robustness of scene flow estimation.