Multi-level Image Segmentation With Information Complementarity

Image segmentation is one of the fundamental tasks in computer vision.According to the level of granularity,image segmentation can be subdivided into several categories,including two-class segmentation tasks(e.g.saliency segmentation and edge detection)and multi-class segmentation tasks(e.g.semantic segmentation,instance segmentation and panoptic segmentation).Saliency segmentation is to segment the pixels which belong to visual salient regions.Semantic segmentation is to segment the pixels on the basis of semantic categories.Instance segmentation is the instance-level pixel-wise image classification which is beyond semantic segmentation.We focus on multi-level image segmentation,going forward one by one,investigating saliency segmentation,seman-tic segmentation,and instance segmentation.Information complementarity is the key to our work.To achieve precise saliency location and mask,we utilize both high-level global context and local cues obtained in feature residual,since global context and local cues are complementary.To improve the well-trained semantic segmentation network,we use Teacher's knowledge and the ground truth of samples to train Student jointly,which is our improved network when the training is over.Due to high latency in recent instance segmentation methods,we propose an efficient single-stage framework to gen-erate instance mask by employing box information,semantic segmentation results and pixel affinity jointly with a post processing method.Our main contribution can be concluded as follows.Firstly,we propose a residual refinement network for precise saliency prediction.Starting from a common encoder-decoder architecture,we enhance a residual refinement network with feature decoupling for better saliency estimation.To this end,we improve the global knowledge streams with intermediate supervisions for global saliency estimation and design a specific feature subtraction module for residual learning,respectively.On the basis of the strengthened network,we also introduce an attribute encoding sub-network with a grid aggregation block to guide the final saliency predic-tor to obtain more accurate saliency maps.Furthermore,the network is trained with a novel constraint loss besides the traditional cross-entropy loss to yield the finer results.Extensive experiments on five public benchmarks show our method achieves better or comparable performance compared with previous state-of-the-art methods.Secondly,we propose to improve the well-trained network for semantic segmentation.Due to extremely high performance of the supervised semantic segmentation,we propose a new problem with practical significance,namely,how to improve a well-trained semantic segmentation network in order to correct those unsatisfactory results of hard examples without bad influence on common examples—or rather,easy examples.This technique can be used to improve commercial semantic segmentation model.Inspired by knowledge distillation,we employ the well-trained network whose param-eters are fixed as Teacher,meanwhile,employ a duplicate of Teacher as Student.The proposed redistribution algorithm are conducted on the classification scores of Teacher and Student,respectively.Student is trained jointly by the supervision from Teacher as well as the ground truth of training samples.As a result,the trained Student is our improved network.Experimental results demonstrate the feasibility of our method,in addition,our method achieve great efficiency and speed.Thirdly,we propose a single-stage instance segmentation framework with subgraph merge algorithm in order to achieve fast speed for instance segmentation.The single-stage object detectors achieve remarkable performance with faster execution and higher scalability.Inspired by this,we propose a single-stage framework to tackle the instance segmentation task.Building on a single-stage object detection network in hand,our model outputs the detected bounding box of each instance,the semantic segmentation result and the pixel affinity simultaneously.After that,we generate the final instance masks via a fast post processing method with the help of the three outputs above.As far as we know,it is the first attempt to segment instances in a single-stage pipeline on challenging datasets.Extensive experiments demonstrate the efficiency of our post processing method and the proposed framework obtains competitive results as a single-stage instance segmentation method.Furthermore,the accelerated framework can offer satisfactory speed-accuracy trade-off on MS COCO validation set.