The Research Of RGB-D Semantic Segmentation Based On Geometric Deformable Convolution

Semantic segmentation is a fundamental task for scene understanding and plays an important role in automatic driving,geological survey,precision agriculture and other fields.With the availability of RGB-D sensor and geometric structure from depth images,RGB-D semantic segmentation immediately becomes a new hotspot.This dissertation focus on incorporating geometric information from depth images into Convolutional Neural Network(CNN)to augment standard convolution.The resulted geometric deformable convolution improves the performance of segmentation.The main work is as follows:(1)Prior based geometric deformable convolution is proposed.Due to fixed grid kernel structure of standard convolution,CNN has limited power for modeling geometric structure of target objects.To solve this problem,assuming pixels with similar depth value have similar contribution to output pixel,this dissertation augments standard convolution with depth similarity term.The resulted prior based geometric deformable convolution improves the accuracy of segmentation.(2)HHA data based geometric deformable convolution is proposed.The geometric similarity term proposed in prior based geometric deformable convolution is handcrafted and hard to adapt to complex scenes.To solve this problem,HHA data based geometric deformable convolution proposes a new data-driven framework that incorporates geometric information into the learning of deformable kernel.The accuracy of segmentation improves further.(3)Pseudo-Li DAR based geometric deformable convolution is proposed.The geometric features in HHA data based geometric deformable convolution are single-scale,which restricts the learning of offsets.To solve this problem,this dissertation proposes HPNet module to extract geometric features from Pseudo-Li DAR data.In the meantime,HHA data based geometric deformable convolution is refined,which boosts the performance further.Based on extensive experiments on benchmark datasets such as NYUv2,SUN-RGBD,the effectiveness and efficiency of the proposed methods are verified.