Image-based Modeling And Rendering Of Transparent Objects

Numerous techniques have been proposed for image-based modeling and rendering of opaque objects in the past decades.However,none of them can be directly applied to transparent objects,because of the complex light refraction and view dependent effects,which will apparently violate the Lambertian hypothesis of traditional methods.Therefore,specifically for transparent objects,we start from data acquisition,as well as explicitly considering light transport properties,by capturing ray-ray correspondences and ray-pixel correspondences under different camera views,we will discuss full 3D reconstruction of transparent objects and deep view synthesis for transparent objects,respectively.The main contributions can be summarized as follows:1)Automatic data acquisition system.We design a fully automatic data acquisition system for collecting the refraction of light from transparent objects at different viewing angles of the camera,i.e.,the correspondences between incident and outgoing rays(rayray correspondences for short),in order to assist the full 3D reconstruction and deep view synthesis for transparent objects.Note that,a special feature of this acquisition system is the utilization of a turntable.By positioning a transparent object on the turntable,while fixing the camera,rotating the object to different viewing angles,then it is possible to acquire the light information for different surfaces of the object through the camera,thereby calculate ray-ray correspondences as mentioned previously.Besides,according to environment matting,object masks(silhouettes)and light attenuation maps could also be obtained at the same time.2)Full 3D reconstruction of transparent objects.Based on the collected data,starting from an initial rough model generated from space carving,the proposed algorithm progressively optimizes the model under three constraints: surface and refraction normal consistency,surface projection and silhouette consistency,and surface smoothness.Experimental results on both synthetic and real objects demonstrate that our method can successfully recover the complex shapes of transparent objects and faithfully reproduce their light refraction properties.3)Deep view synthesis for transparent objects.The problem of view synthesis,i.e.,interpolating and generating novel views from sparsely sampled images,has recently attracted a lot of attention.In this work,we are exploring a challenging view synthesis problem that specifically aims for transparent objects.That is,different from existing approaches that mainly focus on Lambertian surfaces without considering specularity or transparency,we propose to explicitly take light transport properties and complicated view-dependent effects into account.We use deep convolutional neural networks to learn light transport matrix at each novel view,instead of directly synthesizing target images with ad-hoc approaches.Therefore,our method performs much better on novel views synthesis,and can seamlessly composite target objects into any new backgrounds at each view.As a side contribution,we have collected and will share a benchmark dataset,the first one to our best knowledge,that contains 8 categories of synthetic models and 6 real transparent objects for view synthesis training and testing.Experimental results demonstrate that our method can nicely predict complex light transport behaviors involved in the refraction of transparent objects at novel views.