Shadow Generation In Augmented Reality

For general augmented reality applications,the reality of the composite image output determines the quality of the augmented reality application and has a crucial impact on the user experience.The illumination consistency of the composite image has a very important effect on the realism of the composite image.The realism of synthetic images is mainly reflected in the consistency of space and illumination between virtual objects and the real world.Spatial consistency mainly reflects whether the relative rigid body transformation and perspective relationship between the virtual object and the real world are correct,and lighting consistency is mainly reflected in whether the shadows,shading effects of the virtual object match the lighting conditions in the real world.In contrast,the problem of lighting consistency is more complicated and changeable due to the impact of the shooting scene,which is difficult to solve.Since the human eye's judgment of the realism of the synthetic image depends to a greater extent on the shadow of the virtual object,it is important to generate a plausible shadow of the virtual object for augmented reality applications.Existing methods for generating virtual shadows follow the physically based rendering theory,and need to estimate the real world's geometry,lighting,reflectance,and material properties.After acquiring these properties,virtual objects are rendered to generate shadows.However,the inverse rendering of these properties is a very challenging task in the field of computer vision,which often depends on a huge amount of computation and is difficult to perform in real time.In addition,incorrect estimation results will cause subsequent rendering to generate virtual object shadows that do not match the real world lighting conditions.This makes the inverse rendering based methods consume a lot of manpower and material costs,and then is severely limited in practical augmented reality applications. In view of the above key issues,this paper studies the method of direct generation of virtual objects shadows based on monocular vision from the perspective of image generation in deep learning,with the goal of general and convenience.The main research contributions of this paper are summarized as follows:(1)Construct the first large-scale augmented reality shadow-generating image data set.This data set is very different from the existing shadow-related image data sets,and not only contains the corresponding relationship of virtual object shadows from scratch.The shadows in the real world and the corresponding occluders that produce them are also labeled.These annotations serve as important clues that can better guide the generation of virtual object shadows.(2)Propose an end-to-end trainable generative adversarial network called ARShadow GAN.ARShadow GAN has the ability to directly generate shadows for inserted virtual objects after training on the above-mentioned augmented reality shadow generation dataset,which is suitable for real-world scenes containing a single domain light source.Different from the general image-to-image translation framework,ARShadow GAN makes full use of the attention mechanism,pays attention to the shadows and corresponding occluders in the real world,and uses them as clue information to better guide the generation of virtual object shadows,achieving directly modeling the mapping relationship between the shadow of the virtual object and the real world without any inverse rendering steps such as explicitlighting estimation.(3)Combine theory and practice,successfully transplanted and deployed the optimized ARShadow GAN network model on the Huawei Atlas 200 AI developer kit,and realized the execution of inference steps for virtual objects shadow generation at a near real-time speed on a single developer board.The research content of this paper is tightly centered on the topic of shadow generation of virtual objects in augmented reality,which involves the construction of shadow datasets,the structural design of deep neural network models,the training and evaluation of network models,and the migration and deployment of network models.The method of this paper can avoid the complicated and costly inverse rendering process,and realize simple and convenient direct shadow generation.Extensive experiments confirm that the method proposed in this subject can be effectively applied to general augmented reality applications.