Research On Virtual And Real Fusion Algorithm Based On Illumination Consistency

Augmented reality is a technology that combines real scene and virtual information,and presents the obtained virtual-real fusion scene to users to enhance their understanding and perception of the real scene.At present,augmented reality technology is widely used in military,medical,entertainment,education and other fields.The research goal of augmented reality is to realistically fuse virtual objects into real scenes,and meet the requirements of geometric consistency,illumination consistency and time consistency.There is still a lack of systematic research on the problem of illumination consistency in the current solutions,but reconstructing the interaction between the light sources in the real scene and the virtual objects is the key to improve the realism of the virtual objects.In this thesis,four algorithms for illumination estimation and joint estimation of illumination and material are proposed for virtual-real fusion in augmented reality.The main contributions and innovative work of this thesis are summarized as the following four parts:1.Illumination estimation algorithm suitable for multiple reflection phenomena.Aiming at the problem that the Lambert illumination model is not suitable for the phenomenon of specular highlights in the scene,an algorithm for illumination estimation is proposed,which takes the light source position obtained from the specular highlight information as a priori knowledge and then combines with the Lambert illumination model.By detecting the specular highlights of the input images,the images with and without highlight are filtered out.Based on the generation principle of specular highlight point phenomenon,the highlight point light sources are estimated.Taking the highlight point light source as the constraint condition,the local illumination model is inversely used to optimize the position of the light sources to be estimated.The effectiveness and feasibility of the proposed method are proved by the captured image datasets.This method has the following advantages: for the real scene with specular highlight,compared with two illumination estimation algorithms that only consider diffuse reflection and only consider specular reflection,this method will obtain more accurate estimation results by considering both reflections at the same time.Moreover,this method has wide applicability.This method can also obtain accurate illumination estimation results for scenes with only diffuse reflection.2.Illumination estimation algorithm based on a global illumination model.Aiming at the local illumination model that only deals with the influence of light emitted from the light source on the surface of the objects,without considering the problem of reflections of light between objects,an algorithm for illumination estimation based on photon mapping of the global illumination model is proposed.By establishing a photon emission hemispherical model that conforms to optical principles,photon tracing under multiple light sources is performed.By utilizing multiple photon maps,the information of multiple reflections of photons in the scene under different light sources is recorded.The photon information collected in the ray tracing process and the input images form an objective function,and the objective function is optimized to obtain the light source intensity value on the hemisphere model.According to the estimated light source result,the virtual object is rendered,and a realistic virtual-real fusion image is output.This method has been experimented on both the virtual datasets and the captured image datasets.Compared with the method of using the local illumination model to estimate the illumination,this method obtains more accurate estimation results based on the physical law of light propagation.3.Virtual-real fusion algorithm beyond Lambertian assumption.Aiming at the problem that the caustics of specular objects and transparent objects in real scenes will affect the inserted virtual objects around them,an algorithm for jointly estimating the illumination and the material of specular objects and transparent objects is proposed.Using the depth inconsistency in the multiview depth images captured by the depth camera,the specular object and transparent object are recognized at the same time.Based on the different interaction properties between light and the surface of objects,a method to distinguish the specular object and the transparent object according to the initial estimated light source is proposed.In order to estimate the material parameters of transparent objects more efficiently,the visual characteristics of the human eye are combined with the mathematical model to derive the minimum change in refractive index of the transparent object when the human eyes can perceive the change in caustics of the transparent object.This minimum change in refractive index is used as the step size to estimate the refractive index in the optimization algorithm.Several experimental results show that the joint estimation of illumination and material can not only obtain the estimation results of light source and material,but also make the estimated light source position more accurate.Combining the estimation results of illumination and material with differential rendering technology,the influence of caustics produced by specular objects and transparent objects on virtual objects is also solved,and a more realistic virtual-real fusion effect image is obtained.4.Virtual-real fusion algorithm for scenes of different roughness transparent objects.For a real scene where there are different roughness transparent objects,inserting virtual objects behind the transparent object will cause different occlusion effects on the virtual object.A joint estimation algorithm of illumination and the material of the transparent objects with different roughness based on inverse path tracing is proposed.Due to the complexity of the interaction between different roughness transparent objects and light,a two-step optimization algorithm is designed.In the first step,a hemispherical area illumination model is established to distribute the light sources to be estimated.The light source and opaque objects are jointly optimized,and the output results are used as the input of the next stage.Then in the second step,the transparent object is added to the optimization algorithm.When the result of the light source in the first step is known,the material parameters of the transparent object can be estimated more accurately.The virtual-real fusion results and the recovery different roughness transparent object images prove the effectiveness and feasibility of this method.Compared with the current methods,this method can insert virtual objects into the real scene of different roughness transparent objects more realistically.