Indoor Scene Semantic Segmentation Based On Color-Depth Image Information

Image semantic segmentation has always been a research hotspot in the field of computer vision.Its purpose is to give different semantic category labels to objects by segmenting different objects,so that the computer can comprehensively obtain scene semantic information.But the current research mainly focuses on the use of color image data as training for outdoor scenes and single task semantic segmentation.In practical application,the indoor scene is more complex than the outdoor scene.In addition to the semantic category information of the object,the robot also needs to know the location information of the object in order to complete more complex visual tasks.At the same time,the indoor scene lighting and other factors will affect the production of color images.In view of the above problems,this paper takes the complex indoor scene as the research object,uses RGB-D image information,and based on the deep learning and convolution neural network algorithm,carries out the research of multi task semantic segmentation of joint target detection.The main research work is as follows:(1)The establishment of the indoor scene database based on Kinect.Through Kinect's relevant built-in parameter matrix,the position relationship between the camera's color camera and the depth camera is determined to ensure that the color collected by the camera corresponds to the depth image one by one.Using Kinect to get the color and depth images of indoor scene,the database of indoor scene image is constructed.(2)The semantic segmentation of RGB indoor scene based on convolutional neural network.Based on the improved FCN semantic segmentation model,the model is trained with the idea of migration learning,the depth optimization algorithm is introduced to improve the training speed and convergence speed of the network,and the training model based on color image in indoor scene is established.The validity of the algorithm is verified by the semantic segmentation prediction experiment.(3)Multi-task semantic segmentation of indoor scenes based on RGB-D images.Based on the improved Faster-RCNN algorithm,a multi-task semantic segmentation model for joint target detection is constructed,which enables it to simultaneously implement semantic segmentation,target classification,and detection of multiple visual tasks.At the same time,through the fusion of color and depth images,the introduction of RoIAlign,improved NMS algorithm and other series of operations to improve the performance of the model.Aiming at the problem of semantic segmentation of indoor scene,the model is trained by using color,depth and fusion image data respectively.The feasibility and accuracy of the multi task semantic segmentation model in this paper are verified by many experiments.The accuracy of RGB-D fusion image training is 2.650% and 17.675% higher than that of color and depth image respectively.