Research On Multi-view Foreground Human Instance Segmentation Method Based On Spatial Consistency Constraints

Image segmentation is one of the most significant research areas in computer vision,which is now widely used in multiple relevant applications.As a pixel-wise classification task,it is a technology that classifies pixels with similar features together.Multi-view segmentation serves as an important procedure in 3D applications,for both the precision and the consistency among viewpoints contributes to the subsequent 3D display quality.Apart from the performance of single-view segmentation,the relations of the segmentation results in three-dimensional space also need to be introduced.Large amounts of redundant pixels of the‘background'class that lie within the instance mask contour,in especial,add to the problems to be solved.The spatial consistency constraint is proposed in this work to further improve the segmentation performance and to contribute to the 3D display quality of foreground objects.The thesis focuses on the multi-view foreground segmentation task constrained by spatial consistency,proposes a brand-new methodology,and completes a series of research and related work.The main contributions of this paper are presented as follows:1)Establishing multi-view image sequence refocusing algorithm based on multi-view projection relation and camera parameters obtained from SfM.Using the TensorFlow framework to build a two-stage instance segmentation network.Training the network on the challenging MS COCO dataset till convergence,and obtained an end-to-end network for both object detection and instance mask output during the prediction phase.The refocused multi-view images are as input,and the output of the two branches is innovatively used as prior for final work-up.The network prediction presents satisfactory results and is set as the baseline.2)As for the redundant background pixels inside the mask,based on the deep learning segmentation output,a GMM segmentation method is proposed.The first experimental set utilizes the bounding-box as prior information,and the other adopts the output of both two branches.Compared to the baseline,the two GMM segmentation experiments with deep learning pre-segmented prior introduced have gained increase of 5.78%and 7.71%respectively on mean mask IoU,which proves that the proposed method is capable of removing certain portion of the background pixels that are misclassified as foreground,thus improving the multi-view segmentation quality.Using the newly-captured 48-viewpoint dataset for 3D display quality validation.Processing the segmented foreground instance with multi-view algorithm to obtain sub-image array.The lens array screen shows that the displayed instance is precise and intact with clear boundaries,thus proving the effectiveness of the proposed methodology.