Research On Visual Effects Of Projection Mapping Oriented To Augmented Reality

Projection mapping is a common method in AR(Augmented Reality)technology,which project virtual image onto real objects to mix them for the purpose of reality enhancement.Projection mapping is widely used in fields of education,entertainment,military and medicine,etc.General application of projection mapping usually projects contents onto arbitrary planes or target objects with 3D geometric shape.One of the goals in projection mapping is sense of reality,which is closely related to the correctness of projection contents.As we want the projection contents as correct as possible,it is necessary to obtain accurate material information and precise geometric shape of projected objects.In previous researches on projection mapping,researchers seldom consider the actual material of projected objects.Some of them directly use assumed texture without consideration on the rationality of lightening effects,which causes large deviation of vision reality between the projected result and the actual scene should be.Another common method is to apply simple Lambertian reflectance model with manual parameters and textures,which is time-consuming and manpower-consuming.The material properties are completely dependent on the subjective judgment of human.The sense of reality of the projection effect cannot be well guaranteed.As to the recovery of geometric shape,there have been many researches with impressing results at present.Such as designing prior models according to target objects and using the prior models for subsequent operations.Another popular way is to gain geometric shape of target object from depth information.The method of reconstructing geometric models using depth information is more versatile.However,since the depth image captured by depth camera always has some error,the reconstructed 3D model usually has the problem of rough surface and incomplete edge.Considering the above problems about the sense of reality of material and the precision of geometric reconstruction in projection mapping technology,we propose a reconstruction algorithm of material and geometry of the target object in real scene based on deep neural network.The algorithm reconstructs the material information and geometric shape of target object in real scene from a single RGBD image.Besides,we also smooth the depth image and refine the edge part of the triangular mesh model of target object.It helps to mitigate the problem of rough surface and partly-missing edge in projection results.On the basis of above algorithm,we design a projection mapping system based on material and geometry reconstruction.It uses the results of reconstruction to render projection contents and make the projection result more real.The main research results of this thesis are as follows:(1)A material and geometry reconstruction algorithm based on deep neural network is proposed.A neural network is used to learn from a single RGBD image of the scene under specific illumination and predict the material and shape information of target object.The material is represented by parameters of Bidirectional Reflectance Distribution Function(BRDF)while the geometric shape is in the form of normal map.This algorithm reconstructs the non-homogeneous surface of target object by pixel and also handle the scene with several materials.Moreover,the geometric result of the algorithm is smoother by preprocessing the depth image.(2)A projection mapping system with high precision based on material and geometry reconstruction is designed and realized.We design both software and hardware architecture of the system.Such system reconstructs the material and geometric shape of target object in the real scene and projects the results rendered by recovered information onto target object.It makes the obtained projection consequence much closer to real scene.(3)A dataset of RGBD images used to train the network is established.The dataset is a set of color images and depth images of the scene under known directional illumination.It is used in the training process of above network for the goal of scene reconstruction.(4)The effectiveness and correctness of the algorithm and system proposed in this thesis are evidenced.Comparing to hand-crafted method and testing the training strategy of neural network,it is proved that our algorithm and system can realize the reconstruction of material and geometric shape of target object in projection mapping and simulate virtual scene.The visual effect of projection results is impressed for sense of reality.