Pixel-Affinity-Aware And Semantic-Aware Image Instance Segmentation

Image instance segmentation is one of the fundamental tasks in computer vision.Given an image,instance segmentation aims to distinguish different semantic instances among given classes in a pixel level.Existing instance segmentation methods can be divided into two categories,proposal-based methods and proposal-free methods.Pro-posal based methods first try to locate an object with a box and then find a mask within the box.On the other hand,proposal-free methods utilize the network to predict the pixel-level instance information and then cluster the pixels in the image into different classes.It is the key point to find the instance-level representation in proposal-free meth-ods.For proposal-based methods,on the one hand,as the mask are generated within a proposal,the feature learning procedure are constrained with the box.On the other hand,with the limitation of the resolution in mask head,the mask results are usually coarse.Therefore,to solve these problems and improve the performance of instance segmentation methods,we propose to introduce the pixel affinity and semantic context explicitly into instance segmentation task.Our main contribution can be concluded as follows:Firstly,we propose a proposal-free instance segmentation method based on graph merge algorithm.We take the whole image as a graph and the convolutional neural network is utilized to predict the semantic information and pixel affinities as the edges of the graph.We propose the graph merge algorithm to cluster the pixels in the graph into different sets and each set would be an instance result.This is the first method to combine the pixel affinity with the semantic information,and develop the instance segmentation results with graph merge algorithm.Experimental results show our graph merge algorithm can generate acceptable instance segmentation results and compared with conventional methods,our instance mask is more precise.Secondly,we propose an instance segmentation method based on global context.We first take the semantic segmentation as an auxiliary supervision in the network train-ing to help us learn features with global contexts.Meanwhile,we make a constraint on different branches in our network.We impose that the output of mask head and the output of semantic branch should be consistent.Then,to make sure that position in-formation in a box could be learned for a better prediction of the mask,we introduce the position encoding into the mask head.Experimental results show that the algorithm could boost the performance with no extra computational cost.Thirdly,we propose to make a refinement on the instance segmentation method based on the pixel affinity.For a proposal-based method,a mask result is obtained by an interpolation on a feature map with a fixed size.The given mask is usually coarse,and the boundary is not aligned to the objects.To this end,We introduce the pixel affinity into proposal-based instance segmentation methods as an auxiliary loss,which would help the network to learn a better feature.Furthermore,with the semantic information and pixel affinity predicted,we introduce the graph merge algorithm into proposal-based methods.Utilizing the predicted affinities and the prior from mask head,we refine the mask results obtained from proposal-based methods to get more precise masks.Exper-imental results show that we can improve instance segmentation methods on different datasets and more precise masks are obtained.