Research And Applications On Establishing Correspondence Between Images And 3D Shapes

In the context of explosive data growth,images and 3D shapes ser-ve as popular visual mediums which respectively carry rich color appearance and geometric struc-ture information of the represented objects.Better mining and processing information characteristics expressed by these two visual mediums is an important research topic in many relevant fields.Because of the complementarity of information expressed by images and 3D shapes,some practical applications can be realized effectively by taking advantage of the cor-respondence established between them,such as image based 3D model retrieval,single image depth estimation,image/3D model segmentation,etc.In this paper,we take a single image(or a single 3D model)as input,by using existing 3D model datasets to explore the 3D model retrieval of complex indoor scenes and the mutually guided seg-mentation of images and 3D models.We also found some applications of these methods in real scenes.Specifically,our work is divided into the following aspects:1.Image-based 3D model retrieval for indoor scenes by simulating scene context.For an input indoor scene image,this proposed method returns an aligned 3D model which is most similar to the input scene object from the ShapeNet[1]model reposi-tory by simulating the input scene context.Firstly,this method renders 3D models at different viewpoints and represents them as view-dependent visual elements.Subse-quently,with the help of estimated occlusion relationships between target objects in the input scene,the rendered models are placed at corresponding positions to simulate the context of the input scene.Finally,by matching simulated indoor scenes with the in-put image,the most similar 3D model which owns the corresponding pose is retrieved.Experimental results on public datasets demonstrate the effectiveness and robustness of our method,especially in several challenging scenarios,such as clutter backgrounds and severe occlusions.Extensive experiments on large number of synthetic images show that the retrieval accuracy of our method is better than the state-of-art methods.Moreover,an effective greedy algorithm is proposed to reduce the model retrieval time significantly.It's worth mentioning that the flexibility of this method is better and the user only needs to drag a few semantic boxes for query objects.2.Bidirectional guided segmentation between images and 3D models via label transfer.This method can realize mutually guided segmentation between images and 3D models through label transfer,and the segmentation accuracy is better than other guided segmentation methods which can only be performed in single direction.At first,we retrieve a small number of annotated models whose rendered images are similar to the input.Then,with the aid of the proposed method named local correspondence en-coding,the coarse correspondence between images and 3D models is obtained.Finally,label transfer problem is formulated as a Conditional Random Field model(CRF)[2],in which the likelihood of labels is represented by local correspondence encoding term.The accurate label transfer result and fine-scale correspondence between images and 3D models can be acquired through iterative optimization.In the meantime,the pur-pose of bidirectional guided segmentation between images and 3D models is achieved.Our label transfer method can be realized bidirectionally,which is suitable to image-guided model segmentation and 3D model-guided image segmentation.Meanwhile,the fine-scale correspondence between images and 3D models can be applied to some prac-tical scenes,such as image depth estimation,2D sketch segmentation,and image-based mesh colorization.The effectiveness,generalization and superiority of our approach is demonstrated by extensive experimental results on public datasets.