Research On Self-supervised Learning Based Weakly-labeled Point Cloud Semantic Segmentation

As a fundamental task of computer vision,semantic segmentation still is an unsolved well problem.Compared with semantic segmentation of 2D images,3D point cloud broadens the dimension of perception and point cloud semantic segmentation plays an essential role in automatic driving,robots and smart cities,etc.However,fully supervised point cloud semantic segmentation requires large-scale labeled samples,which results in time-consuming,labor-intensive,and costly manual work.We focus on point cloud semantic segmentation with weak labels,which promotes segmentation performance with a low labeling cost.But research on this topic is still in its infancy.If we apply directly the fully supervised method to the weak label setting,the performance will be seriously degraded due to insufficient label information.Self-supervised learning can improve the representation ability of the model in an unsupervised learning paradigm.Therefore,this dissertation aims to learn prior knowledge from external data or the data in itself by utilizing the self-supervised learning theory to improve the weakly supervised semantic segmentation performance.Specific research contents and innovations include:Firstly,a weakly supervised point cloud segmentation method based on generative self-supervised learning is proposed,which aims to introduce additional supervisory information using external data.We construct a pretext task,i.e.,point cloud colorization,with a self-supervised training manner to transfer the learned prior knowledge from a large amount of unlabeled point cloud to the weakly supervised network.Moreover,a regularization scheme is designed for the point cloud colorization to ensure that the learned knowledge can be transferred to the weakly supervised task.Furthermore,to make full use of unlabeled data,an efficient sparse label propagation is proposed for supervised unlabeled point learning to improve the representation ability of weakly supervised network.Experimental results on three large-scale point cloud datasets,including indoor and outdoor scenes,show the effectiveness of the proposed method.Secondly,we present a perturbed self-distillation framework based on self-supervised learning.We study self-supervised learning to mine the properties of the data in itself to improve the performance of weakly supervised point cloud semantic segmentation.However,for large-scale point clouds with sparse labels,the network is challenging to extract discriminative features for unlabeled points,resulting in incorrect segmentation results of classes with similar structures.Inspired by self-supervised learning,we construct the perturbed branch and enforce the predictive consistency among the perturbed and original branches,which construct auxiliary supervision.In this way,the graph topology of the whole point cloud can be effectively established by the introduced auxiliary supervision,such that the information between the labeled and unlabeled points will be propagated.Besides point-level supervision,we present a well-integrated context-aware module to explicitly regularize the affinity correlation of labeled points.Therefore,the graph topology of the point cloud can be further refined.The experimental results evaluated on three large-scale datasets show the large gain(3.0%on average)against contemporaneous weakly supervised methods and comparable results to some fully supervised methods.Thirdly,we present a cross-point cloud consistency contrast self-supervised learning framework.We study a more flexible approach for weakly supervised point cloud semantic segmentation,which avoids the manual designed techniques such as external data or point perturbation introduced by self-supervised learning,and we give a theoretical explanation based on Expectation-Maximization algorithm(EM).It alternately refines pseudolabel(E step)and optimizes network parameter(M step).Specifically,in E-step,we propose a pseudo label selecting method based on cross sub-cloud consistency to improve the credibility of selected pseudo labels explicitly.In M-step,a cross-scene contrastive regularization is present for reducing the fitting of pseudo label noise.We evaluate our method on four challenging datasets,where experimental results demonstrate that our method significantly outperforms contemporaneous state-of-the-art weakly supervised competitors and even achieves comparable performance to the fully supervised RandLA-Net.Fourth,this dissertation proposes self-supervised exclusive learning based multimodality unsupervised domain adaptation semantic segmentation.Even existing methods achieve considerable improvements by performing multi-modality alignment in a modality-agnostic way,they fail to exploit modality-specific characteristics for modeling complementarity.We generalize self-supervised learning theory from weakly supervised tasks to unsupervised domain adaptive semantic segmentation tasks,and it no longer requires the assumption that training data and test data are identically distributed.Specifically,we introduce a mixed domain by mixing the patches of the source and target domain samples.On the basis of the mixed domain,two self-supervised tasks are proposed,i.e.,reconstructing the spatial information of the point cloud using the image features,and reconstructing the regular image texture information using the point cloud features.The former helps the 2D network branch improve the perception of spatial metrics,and the latter supplements structured texture information for the 3D network branch.The modalityspecific exclusive information can be effectively learned,and the complementarity of multi-modality is strengthened,resulting in a robust network to domain shift.To further obtain domain-invariant features,we propose category-aware adversarial learning with category-specific discriminators by constructing the category prototypes and pseudo category prototypes.We evaluate our method on various multi-modality domain adaptation settings,where the result significantly outperform both uni-modality and multi-modality contemporaneous competitors.