Semantic-based Image Panoptic Segmentation

In the past decade,with the great improvement of computer computing ability and the rapid development of artificial intelligence technology,many image-based cognitive applications attracted more and more attention by both academic and capital fields.The most typical landing scenes are intelligent surveillance cameras,self-driving cars and drones,etc.In these scenarios,semantic segmentation,instance segmentation and panoptic segmentation play as an important role to perceive the external environment for smart agent such as autonomous cars.Among these sub-directions of image segmentation,panoptic segmentation is the most complicated and most meaningful for commercial purposes.It requires that each pixel in the image be assigned a unique semantic category and instance identification number.In this thesis,we perform theoretical research and practical development for panoptic segmentation based on the convolutional neural network architecture.In order to depict the contour boundary of the instance object,two mathematical models are proposed in this thesis,which are based on the angle and distance under the polar coordinate system,and cubic spline curves under the Cartesian coordinate system to describe the outline of the instance boundary,respectively.For contour boundary description in polar coordinate system,a point inside the instance is pre-defined as the origin,then rays with interval rotation are shot from the origin,the intersection points of these rays and instance boundaries are sampled as ground truth.For cubic spline curves based contour boundary description in Cartesian coordinate system,the boundary points are sampled with equal arc length.Once these boundary-sampling points are obtained,cubic spline could exploited to fit instances' boundary.In order to verify the difference between the two instance boundary modeling,a panoptic segmentation network based on these two different instance boundary modeling methods,are built in this thesis.It consists of semantic segmentation and instance segmentation sub-branches in parallel.The instance segmentation sub-branch does not require predefined anchors,it directly regresses the defined distance vector in polar coordinate system or these cubic spline curve coefficients in Cartesian coordinate system.Compared with the anchor-based instance segmentation method,the network complexity and the amount of model parameters are greatly simplified and reduced.In the panoptic segmentation architecture,we customize the U-Net network as its backbone for feature extraction,stitching the feature maps of the same scale in the down sampling and up sampling process in U-Net,aiming to effectively excavate the input features under different abstract dimensions.In order to combine the output of both semantic segmentation and instance segmentation,a simple and effective fusion algorithm is proposed.Based on Tensorflow framework,the panoptic architecture is implemented and different loss functions for different classification or regression tasks are designed.The neural network are trained and validated based on Ali cloud ECS elastic computing platform,with 9,600 images from Baidu's Apollo Scape Street View dataset as its training and validation set.Different measuring metrics are defined to verify the network performance,which includes the average precision of instance segmentation,average recall rate,and panoptic quality(which consists of segmentation quality and recognition quality).Experimental results demonstrates that network performance could be significantly improved by raising the sampling granularity of the instance boundary in both polar coordinate system and Cartesian coordinate system.But when the sampling granularity reaches more than 72 points,the effect of further improving the sampling granularity is not obvious,at which time the performance is limited by the inherent capability of the network model and the characteristics of the instance boundary modelling method.At the same time,the thesis also compares the panoptic quality and model parameters of different backbones,the customized U-Net proposed in this thesis achieves encouraging performance with the smallest amount of parameters cost.In addition,the panoptic performance of polar coordinate system and Cartesian coordinate system are taken into consideration for horizontal comparison.Results show that the contour representation method based on the cubic spline uses more parameters than the polar coordinate system,which has obvious advantages for instances with more complex boundaries such as bicycle,pedestrian and cyclist.Meanwhile,the performance of the two methods are quite close for large instance objects such as cars and buses.