Research On Image Semantic Segmentation Based On Weak Supervision And Domain Adaptation

As the key technology of scene understanding,image semantic segmentation has always been a very active research direction in the field of computer vision.It is widely used in automatic driving,object detection,human-computer interaction,gait recognition and video surveillance.With the development of deep learning,semantic segmentation has achieved great success with the backbone of fully convolutional networks.However,training deep neural networks usually requires a large amount of annotated training data.Obtaining pixel-level annotations for semantic segmentation are prohibitively expensive and time-consuming.Therefore,in recent years,an increasing number of researchers have turned their attention to get rid of large-scale pixel-level annotations,so as to reduce the annotation burden of semantic segmentation task.The research on image semantic segmentation based on weak supervision and domain adaptation has become a new hot direction.Image semantic segmentation has been explored in this dissertation from the per-spectives of weak supervision learning,domain adaptation and one-shot learning.Aiming at the problem of insufficient target object mining and incomplete localization of class ac-tivation maps in weakly supervised tasks,algorithms of non-salient region object mining and saliency guided inter-and intra-class relation constraints are proposed,respectively;aiming at the problems of feature distortion and classifier overfitting in domain adapta-tion tasks,an enhancing feature space adversarial learning method is proposed;aiming at the problem of network overfitting resulting from features lacking semantic information in one-shot semantic segmentation task,a semantically meaningful class prototype learning method is proposed.The main research content of this dissertation includes:(1)A non-salient region object mining approach is proposed for weakly supervised semantic segmentation to discover more objects outside the salient areas.A graph-based global reasoning unit is introduced to strengthen the classification network’s ability to capture global relations among disjoint and distant regions.This helps the network activate the object features outside the salient area.Then the self-correction ability of the segmentation network is further exerted to mine non-salient region objects.Specifically,a potential object mining module is proposed to reduce the false-negative rate in pseudo labels.Moreover,a non-salient region masking module is proposed for complex images to generate masked pseudo labels.The non-salient region masking module helps further discover objects in the non-salient region.Experiments on PASCAL VOC 2012 and MS COCO datasets reveal the superiority of the proposed approach.(2)A saliency guided inter-and intra-class relation constrained method is proposed for weakly supervised semantic segmentation to assist the expansion of the activated object regions in class activation maps.Specifically,a saliency guided class-agnostic distance module is proposed to pull the intra-category features closer by aligning features to their class prototypes.Further,a class-specific distance module is proposed to push the inter-class features apart and encourage the object region to have higher activation than the background.After strengthening the capability of the classification network to activate more integral object regions in class activation maps,an object guided label refinement module is introduced to take a full use of both the segmentation prediction and the initial labels for obtaining superior pseudo-labels.Experiments on PASCAL VOC2012 and MS COCO datasets verify the effectiveness of the proposed approach.(3)An enhancing feature space adversarial learning method is proposed for domain adaptation of semantic segmentation to alleviate the feature distortion and classifier over-fitting problem.First,a classification constrained discriminator is proposed to balance the adversarial training and alleviate the feature distortion problem.The classification constrained discriminator can help the feature generator extract domain-invariant fea-tures that are useful for segmentation rather than just ambiguous features to fool the domain discriminator.Next,to alleviate the classifier overfitting problem,self-training is collaboratively used to learn a domain robust classifier with pseudo labels selected from target domain noisy predictions.Moreover,an efficient class centroid calculation module is proposed,and the domain discrepancy is further reduced by aligning the feature cen-troids of the same class from different domains.Experiments on GTA5→Cityscapes and SYNTHIA→Cityscapes tasks demonstrate the superiority of the proposed approach.(4)A semantically meaningful class prototype learning method is proposed for one-shot image segmentation to alleviate the problem of network overfitting resulting from features lacking semantic information.The multi-class label information is leveraged during the episodic training for encouraging the network to generate more semantically meaningful features for each category.After integrating the target class cues into the query features,a pyramid feature fusion module is proposed to mine the fused features for the final classifier.Furthermore,to take more advantage of the support image-mask pair,a self-prototype guidance branch is proposed for the segmentation of support image.It can constrain the network for generating more compact features and a robust prototype for each semantic class.For inference,a fused prototype guidance branch is proposed for the segmentation of the query image.Specifically,the prediction of the query image is leveraged to extract the pseudo-prototype which will be combined with the initial proto-type.Then the fused prototype is utilized to guide the final segmentation of the query image.Experiments on PASCAL-5~iand MS-COCO-20~idatasets reveal the effectiveness of this approach.Research on image semantic segmentation based on weak supervision and domain adaptation is conducted in this dissertation.Three different tasks(i.e.,weak supervision learning,domain adaptation and one-shot learning)are explored,and methods are pro-posed accordingly.Comprehensive experiments verify the effectiveness of each algorithm.