| With the wide availability of consumer RGB-D cameras, casual users can easily acquire the geometry of various objects at home nowadays. However, the geometry alone is often not sufficient to faithfully convey the realism of a captured object without a realistic material appearance. Unfortunately, acquiring general material appearance with hand-held consumer RGB-D cameras is difficult for casual users, due to the inaccuracy in reconstructed camera poses and geometry, as well as the unknown lighting that is coupled with materials in measured color images.To tackle these challenges, we present a novel technique, called Simultaneous Localization and Appearance Estimation (SLAE), for estimating the spatially varying isotropic surface reflectance, solely from color and depth images captured with an RGB-D camera under unknown environment illumination. To use this technique, users only need to take a video of the object with a RGB-D camera, and use the acquired RGB images, obj file and camera poses provided by KinectFusion as the input.The core of our approach is a joint optimization, which alternates among solving for plausible camera poses, materials, the environment lighting and normals. To refine camera poses, we exploit the rich spatial and view-dependent variations of materials, treating the object as a localization-self-calibrating model. To recover the unknown lighting, measured color images along with the current estimate of materials are used in a global optimization, efficiently solved by exploiting the sparsity in the wavelet domain. The reconstructed material appearance is expressed as the form of point-wise specular/diffuse albedo, roughness and normal.We demonstrate the substantially improved quality of estimated appearance on a variety of daily objects. Since both geometry and appearance can be acquired with high fidelity, our system is potentially useful for many applications, such as digital content creation by nonprofessionals in e-commerce and games. |
PDF Full Text Request