Semantic Segmentation Of Street Scenes By Appearance And Geometry Features

Digital Images have been widely used in our life as carriers of information, and one of the grand goals of computer vision is to interpret a scene semantically given an input image. Street scenes are closely related to people's daily life, but these images are complicated to process, challenges and opportunities exist at the same time to semantically segment such images with good performance.Current segmentation algorithms could only deal with images of some application field, which contain a few classes. Besides, general image segmentation algorithms are sensitive to the changes of rotation, scale and intensity and so on, which often fail to produce semantic regions identical to the objects. Recently, combining object segmentation and recognition has been particularly active and notable progress has been made, but challenges remain for getting fine object boundaries, when large number of classes occur simultaneously and objects vary dramatically in size and shape. On the contrary, street scenes consist of many classes simultaneously, which makes it complex to process, so semantically segmenting them means very much not only for research field but also for application field.Regarding the problems mentioned above, in this paper, we propose a method that combines the image segmentation, which uses space extreme points as the seeds of watershed segmentation and recognition, which is based on the conditional random field model to inference the semantic labels of each object class, in a frame.Firstly, according to the frame, a Gaussian pyramid of input image is built, then space extreme points of the image are extracted to feed the watershed segmentation algorithm as seeds. Based on the basic rules of image segmentation, noisy regions are removed and regions satisfying the similarity measurement are merged together to get the unsupervised segmentation results.Current research results show that using appearance features only fails to get good object boundaries, so apart from the broadly used appearance, shape and context features, we add five geometry features, height above the camera, closest distance to camera path, surface orientation and track density, which contain rich structure information, to our frame work. More than that, we introduce the high order term to Markvo random field, which uses the unsupervised segmentation results as input, to boost the edge responses between semantic objects. The main research points are: (1) how to acquire the unsupervised segmentation results that are in accordance with the perceptual characteristics to some extent; (2) apply three visual features and the five geometrical features mentioned above in the frame work to detect object boundaries and recognize the semantic classes; (3) add high order term to the Markvo random field to strengthen the semantic segmentation results.The experimental results show that, the unsupervised segmentation has good robustness, and it does not change with the image scale. Besides, after adding the high order term to the conditional random field, better object boundaries are acquired and the segmentation results are basically identical to the objects' semantics.