Reconstructing Real Object Appearance With Virtual Materials Using Mobile Augmented Reality

Inserting separate and independent virtual objects such as 3D shapes and animation into a realistic environment may not be a complicated matter.By contrast,modifying the appearance of real objects using AR technology requires more effort.Thus,virtual textures should look wholly glued to the surface of the real object,requiring accurate tracking of the target object.The material must also be glued to a mesh matching the real object and the loaded virtual materials should truly reflect the lighting and shadow conditions of the actual environment in which they are located.In this paper,we introduced an improved method for real-time light estimation of environmental lighting in AR applications,tracking the movement of physical objects,and making the corresponding virtual objects reflect the lighting and shadows of the actual environment.First,we proposed an effective method for estimating light in the AR scene by training a deep neural network once on different scenes from the input of RGB-D images to determine the direction of the brightest light.This study relied on estimating the light source by a neural network independent of the camera position and then calculating the transformation to real-world space after the light estimation process.This problem was solved by calculating the brightest light source aligned with the camera position in the coordinate space.Thus,the direction of the brightest light was modeled by relative angles in the coordinate system.To make the light estimate free from the camera’s position,these angles are computed from the camera’s coordinate space.The brightest light direction’s coordinate representation was translated into vectors,which are used for rendering in the AR system.In this study,we proved through tests that the real data did not perform as well as did the synthetic data.The proposed neural network was trained on only synthetic data due to the poor performance of real data in training and testing cases.Second,we experimented with advanced methods of detecting and tracking different types of real objects(flat surfaces,simple geometries,and complex geometries)for visualizing virtual materials on the top of these surfaces,where virtual materials will acquire the conditions of the real place in terms of lighting,reflection,and shadows depending on our proposed method for light estimation.Our method of light estimation was tested via experiments on three types of real objects.Results indicate that our method achieves higher resolution and lower error rate compared with other work.Our methods have achieved good results in terms of detection accuracy,tracking accuracy,realistic visualization of materials,matching real shade,and lighting conditions.