Semantic Image Segmentation With Feature Fusion And Hard Example Mining

Semantic image segmentation,the basis of visual semantic understanding,has always been a concern in the field of computer vision.The task here is to assign a unique label to every single pixel in the image and finally get a dense prediction the same size as the input image.Recently,very deep convolutional neural networks(CNNs)have shown outstanding performance in object recognition and have been the first choice for dense classification problems such as semantic segmentation,obtaining better results than traditional methods.However,semantic image segmentation with CNNs is facing two challenges: the first challenge is that repeated subsampling operations in deep CNNs will lead to a significant decrease in the initial image resolution losing much of the finer image structure.The second challenge is that most of the semantic image segmentation datasets have the problem of data imbalance.Assigning same weight to all pixels is not conducive to the classification of hard samples.Based on the mentioned challenges,we target at learning the fine segmentation at the small objects and edges and mining the hard examples in the training set during training.In this paper we firstly introduce the background and significance of semantic image segmentation.Then we address the task of semantic image segmentation with fully convolutional network with feature fusion and hard example mining and make two main contributions that are experimentally shown to have substantial practical merit.First,we design an end-to-end fully convolutional semantic image segmentation network named ResSegNet based on multiscale feature fusion.We decompose semantic image segmentation into preliminary segmentation and segmentation refinement.The preliminary segmentation result is obtained with the fully convolutional coarse segmentation network.A segmentation residual network based on multiscale feature fusion is proposed to extract the multiscale segmentation residuals.And a segmentation corrector is proposed to fuse the two parts and finally get the fine segmentation result.Second,to solve the problem of both inter-and intra-class imbalances,we propose a hard example mining method that goes beyond conventional cost-sensitive learning that allows us to re-weight the contribution of each pixel based on their observed loss focusing more on underperforming classification results based on Focal Loss in object detection tasks.Mining the hard examples can enhance the ability to segment complex objects and imagesFinally,the detailed experimental evaluation verifies the effectiveness and the superiority of the proposed approach,comparing with other state-of-art approaches in semantic image segmentation.Also,we summarize the main research and contributions,analyze the weakness of the proposed approach and discuss the future work.