Spatial Layout Estimation Of Indoor Scene Using Informative Edges And Multi-modality Features

Spatial layout estimation of indoor scenes in high-level visual image scene understanding,has become one of the challenging research hotspots in recent years.By providing effective 3D spatial structure information,spatial layout estimation of indoor scene can be applied to the fields of smart home,virtual reality,mobile robot navigation,etc,which is beneficial to transfer the traditional grid monitor mode into the video surveillance fusing with real-time monitoring mode in the 3D space.However,there exist several problems such as multiple semantic classes,mutual occlusion,weak discrimination for low-level visual features and unblanced illumination in the spatial layout estimation process.To resolve these problems,this paper proposes a spatial layout estimation approach of indoor scenes consisting of coarse-to-fine layout candidate generation with informative edges and multi-modality heterogeneous featurs fusion.Traditionally,layout candidate generation of indoor scenes depends on the increase of the frequency of the sampling rays from the given vanishing points to improve the accuracy of sampling in spatial layout estimation.To solve this problem,the coarse-to-fine layout candidate generation with informative edges is designed in this paper.First,Canny edge detection with adaptive thresholds is adopted to obtain the straight lines in the indoor scenes.In terms of the directions of straight lines,the RANSAC strategy can be applied to estimate the vanishing points in the input image.The whole image is divided roughly with uniformly-spaced rays from the given vanishing points in the horizontal and vertical directions.Then,the informative edges maps in indoor scenes are obtained with the VGG-16 convolutional neural network.The rough divided regions with high energies will be selected respectively from horizontal and vertical directions with the given informative edges map as a priori,and the layout candidates are generated by resampling the just chosen regions.Experiments on Hedau and LSUN datasets show that,compared with traditional methods,the informative edges map can benefit to effectively determine possible regions of the spatial layout to obtain the fine-grain possible sampled regions,which can reduce the calculation of layout candidate generation and obtain the given layout candidate with high accuracy.In traditional candidate layout models,the unary potential energies have the limited ability of visual representation and the relationships between two neighbour candidate polygons are not enough considered in the pairwise potential.The thesis proposes a spatial layout estimation method of indoor scenes with multi-modality heterogeneous features fusion and structured prediction.First,the spatial layout estimation model for three-dimensional box room is constructed with the parameterization by the angles between the given rays from the obtained vanishing points in indoor scenes.Second,the spatial multi-scaled VGG-16-based convolutional neural network is adopted to obtain the surface direction of the normal vector and geometric depth features with respect to the given indoor scene images,and low-level structured visual features from the line group memberships are attained by the canny edge detection with adaptive thresholds,while the geometrical contextual features with semantic attributes are obtained by VGG-16-based fully convolution neural network.Then,integral geometry-based accumulation technique is applied to merge the given different modual features and generate the unary occurance potential from the regional features in the polygons of layout candidates in the given scene,while the pairwise smooth constraint terms in the probabilistic graphic model are represented by the location relationships between the given neighbour polygons and the angles between their geometric centers in the assumed layout candidate.Finally,the Cutting-plane structured optimization is applied to learn the given model parameters.By the mechanism of maximizing the layout candidates' s scores,the candidate of the given spatial layouts with highest score is selected by ranking the overall scores of the candidates to attain the given indoor spatial layout.Experiments on Hedau and LSUN datasets show that,compared with the traditional methods,the proposed technique performs better in improving the completeness of the feature map in the given polygons of spatial layout candidate to obtain more accurately the final layout of indoor scene by the given model inference.