Research On Parallel Mosaicing Technology For Street Scenes Visualization

Street visualization is one of the key problems in the field of the Virtual Reality. The streets have been an important part of the city. With rapid development of the urbanization and the virtual reality, the demand for street visualization is more and more urgent in modern society. The street visualization widely applies in virtual battlefield environment, automobile guidance, urban planning, virtual traveling, games and heritage protection.The technologies of 2D and 3D street visualization have their own advantages and disadvantages in different application areas. Multi-perspective panorama (MPP) is an important form of 2D street visualization which is usually obtained through the parallel mosaicing. Mosaicing in depth-changing region of the street scene and real-time processing are recognized as the most difficulties in existing methods. 3D street visualization has the advantages of stronger reality and immersion. Even though many 3D reconstruction methods has been proposed, a 3D modeling method with compact form, small data amount and low hardware requirement has also being the research focus in this field. Street image is the main source data of the street visualization, but it often contains various obstacles. Therefore it is an important issue in street visualization that removes the obstacles by using a computer vision method. In this paper, by analyzing the problems of the exiting methods as well as taking the characteristics of urban visualization into consideration, we propose 2D parallel mosaicing technology, stereo parallel masaicing technology and spatial-temporal image completion technology respectively for the 2D street visualization, 3D street visualization and street image completion. Our work mainly includes:1. 2D Parallel Mosaicing technology researchMulti-perspective panorama is a corridor-shape 2D image and covers the main scene information of street side, which supports the user continually browse the streetscape. Compared with the 3D urban visualization technology, MPP have the advantages of lower amount of data, simpler data structure and more convenient browsing. By analyzing the exiting parallel mosaicing methods, we take real-time process and overcoming the depth-change problem as the goal to propose two parallel mosaicing methods. First we propose an adaptive Pushbroom model based parallel mosaicing method which achieves real-time stitching and improves the visual effect of the MPP. In addition, we propose a global optimization based parallel mosaicing method. The method can get an optimal mosaic effect for depth-changing region by automatically choosing the Pushbroom model or multi-perspective model for different street geometry.2. Stereo Parallel Mosaicing technology researchUnder the condition of advanced network and computing platform, users are eager to obtain more realistic and more immersed sense again, so the 3D street visualization technology has become the hotspot. We take a compact, lightweight, and good rendering effect 3D street visualization technology to the research aim, and then present the concept of Stereo Parallel Mosaicing (SPM) which is a kind of parallel mosaicing technology with structure information. First we propose dense-depth information based stereo parallel mosaicing technology, whose outputs are a MPP and a depth MPP. By using the two kinds of MPP, new virtual view can be synthetized. Next we propose a Piecewise Continuous Fa?ade (PWCF) based SPM. The PWCF is used to simplify the urban geometry and as a projection surface for a MPP which will be used as the texture. The SPM is an extension of 2D parallel mosaicing in depth, whose theoretical basis and development can learn from the 2D parallel mosaicing technology directly. The SPM has the inherent advantages of compact, lightweight, and high quality of visual effect.3. Spatial-Temporal Image Completion for street imagesThe street scene contains the permanent objects such as buildings and the temporary objects such as trees, vehicles and pedestrians which are called obstacles. Only the permanent objects are concerned in the SPM, by which it is conducive to real street scene descriptions and visualization. The obstacles have also brought the difficulty for the three dimensional modelling, and have affected the rendering effect. Moreover, the pedestrian in the street image may cause the privacy flaw. In this thesis, we use spatial-temporal image completion technology to remove these obstacles in street images. We propose a disparity assisted spatial-temporal image completion method for the dense depth information based SPM technology. Target and reference regions are labeled through disparity segmentation; and the pixel-wise relationship is also built based on the disparities. Secondly, a global optimization based Spatial-temporal image completion method is proposed for the PWCF based SPM technology. The corresponding relationship between the target and reference regions is built by using structure from motion algorithm. Through the reasonable definition of Markov Random Field (MRF), label set and energy function, the method has higher robustness and integrates the spatial and temporal completions together into one process. The two methods can effectively improve the rendering of the SPM, and compared to existing normal-image completion methods has stronger universality and robustness.