RGB-D Semantic Segmentation Jointed With Deep Learning And Transfer Learning

In recent years,unmanned aerial vehicle,unmanned vehicle and intelligent robots have developed rapidly and gradually become a main hi-tech industrial chain that can develop state economy and improve people's livelihood.More accurate semantic information will be prerequisite to implement autonomous obstacle avoidance and navigation,as well as intelligent path planning for these agents.In complex indoor scenes,however,the semantic segmentation method on RGB color image has rough segmentation error or misclassification error due to the imbalanced illumination,complex occlusion and repetitiveness of color texture,thus leading to less-than-accurate understanding of environmental semantic information.The semantic segmentation method on both RGB image and depth image can utilize spatial information contained in the depth image,which is less affected by illumination,and reflects the position relationship between objects.As a result,the method can achieve a higher accuracy of indoor scene semantics segmentation.In this paper,RGB-D semantics segmentation is studied,and theoretical research as well as its application are carried out.The main work and contributions are described as follows:1.To improve segmentation accuracy of existing RGB semantic segmentation method in complex indoor scenes,a RGB-D semantic segmentation network is proposed in this paper.By visualizing the feature maps of two modality extracted from the network,this paper discusses and analyses the fusion location and fusion method of depth image and RGB color image,and innovatively proposes a fusion-branching structure with a novel feature sifting mechanism.The structure uses a set of learnable penalty factors to weigh the RGB-D features and complete the feature selection process.Experiments show that the proposed fusion structure and feature selection structure can improve the accuracy of intersection over union by up to 5.7%in RGB-D semantics segmentation tasks.2.Aiming at the problem of depth holes(invalid and missing values)in depth images,this paper proposes a depth inpainting algorithm based on fast marching method.The algorithm uses the correlation between the pixels in RGB image to estimate the depth value of the corresponding pixels in the depth image.The algorithm takes full account of the corresponding relationship between the depth image and RGB image,optimizes the depth estimation function and modifies the weighting function of reference pixels,so that the final depth image restoration results can obtain more sharp and accurate edges.Experiments show that the accuracy of RGB-D semantics segmentation model is improved by 2.1%using the restored depth image3.In this paper,it is found that when the trained RGB-D model is applied to the practical scene,the model still has rough segmentation error or misclassification error,which is caused by the distribution difference between the training data and the actual scene data.To solve this problem,this paper proposes a RGB-D semantic segmentation network based on joint transfer learning.The network introduces the adaptation layer and complements the loss function with the maximum mean discrepancy.By continuously optimizing distribution discrepancy of high-dimension features computed from two data domains,the model finally achieves better segmentation results on unsupervised small datasets.This paper collects some image pairs form laboratory environment,constructs a small-scale semantically labeled dataset and carries out optimizing experiments on the dataset.The experimental results show that the accuracy of the semantic segmentation model joint with transfer learning is improved by 1.4%in the actual scene dataset.