RGB-D Indoor Scene Semantic Labeling Based On Temporal-Spatial Superpixel Context

Image-based scene understanding is an important research work in the field of computer vision,and has always been hotspot and difficult problem in the field.Indoor scene semantic labeling is one of the core contents of image scene understanding research.Its basic goal is to densely provide a predefined semantic category label for each pixel in a given image(or a frame in an indoor image sequence).Indoor scene semantic labeling usually considers pixel or superpixel as basic unit,and finally a semantic label is assigned to each pixel.Considering that superpixel has clear semantic and benefit for improving computational efficiency,this thesis selects superpixel as the basic unit of labeling.During the indoor scene image acquisition process,limitation of the camera's field of view,camera's fixed viewpoint,and mutual occlusion between or among scene objects result in partial projection of scene object,which would achieve compensation through sequentially captured images based on continuous viewpoints.So,this paper studies utilization of temporal context for indoor scene semantic segmentation.In addition,a large amount of research works have shown that certain semantic similarity exists between adjacent pixels or superpixels.Considering that,this paper studies utilization of spatial context for indoor scene semantic segmentation.In addition,indoor scene labeling can benefit from using multiple context information definitely.Based on that,this paper studies joint use of spatial and temporal context information,and proposes an RGB-D indoor scene semantic labeling method based on superpixel spatiotemporal context.Details are summarized as following:(1)Indoor scene semantic segmentation based on superpixel temporal context.Firstly,the image to be labeled and its previous and latter frames are segmented into set of superpixels,and feature of each superpixel are calculated based on kernel descriptor.Secondly,corresponding superpixels are computed according to optical flow between labeled image and it's neighboring frames.Feature of the superpixel to be labeled and its corresponding superpixels are concatenated,that is,superpixel temporal context feature.The gradient boosting decision tree(GBDT)is then used to semantically classify superpixels based on superpixel temporal context features to obtain indoor scene semantic labels.(2)Superpixel CRF model based on spatial context.Based on the superpixel segmentation tree,considering the superpixel(node of the tree),adjacent superpixel(node on same layer of the tree),and context association between different scale superpixels(subtrees with a height of 1),a new superpixel CRF is proposed.The model considers three types of spatial contexts and it's objective function of the superpixel CRF contains three energy terms?Unary energy term represents semantic probability value of the superpixel,pairwise energy term measures semantic difference between any two superpixels lying on same layer on the segmentation tree,and the higher-order energy term measures semantic differences among superpixels sharing same father node on the segmentation tree.Minimizing value of the objective function would solve final annotation result.(3)Indoor scene semantic segmentation based on superpixel temporal-spatial context.In the aforementioned superpixel CRF model,superpixel semantics based on the superpixel temporal context used as unary energy term to compute semantic labeling.Proposed methods are tested on the open indoor scene datasets NYU-Depth v1 and NYU-Depth v2.Experimental results show that proposed methods achieve better semantic labeling.