Sampling Theory For Plenoptic Function Based On Geometric Information

Image-based rendering(IBR)has been presented as an alternative to conventional 3D computer graphics.The technique of IBR has been extensively applied to three-dimensional television(3DTV),free viewpoint TV,and immersive communication systems.Instead of representing a scene via geometric models,the scene can be represented using a set of multiview images that compensate for the lack of geometric information.Then,arbitrary novel views can be rendered from these multi-view images.Clearly,sampling the scene using multiview images is a fundamental step in IBR.However,the sampling problems of the IBR remain widely unresolved in theory and practice.To investigate the sampling problem,we can determine the minimum number of capturing cameras that are needed while achieving an antialiased IBR.Furthermore,the IBR systems are comprised of an acquisition component via which the image information is captured using a given geometric configuration of the camera(GCC,i.e.,the position and shooting direction)and a component that renders novel views.In this work,we are also interested in the optimization acquisition component.To study the sampling theory of the IBR,a multi-view image set can be considered as a collection of light rays emanating from a scene.A plenoptic function(POF)can be applied to describe the light ray's captured position,viewing direction,the wavelength,and at a certain time.Thus,the IBR problem can be treated as an application of the sampling theory to the POF.Firstly,the spectral support of the POF depends on the properties and the scene depths.Currently,existing techniques have studied the influence of the following factors on the plenoptic spectrum: the minimum depth and the maximum depth,non-Lambertian reflections and occlusions,whether the scene surfaces are flat,maximum frequency of painted signals and the angle of a slanted plane.Additionally,N.Do also analyzed the relationship between a regular curvature of the scene surface and the spectrum of the POF.Nevertheless,we determine that the scene geometry for arbitrarily irregular shapes is not completely described only by the regular curvature of the scene surface.Generally,the scene geometry for an irregular shape is unknown,its quantification is extremely complicated,such as the leaves of a tree,and its mathematical analysis appears futile.Therefore,compared to these prior studies,we examine the spectral properties of the POF for arbitrary irregularly shaped scenes.Our analysis reveals that as the surface geometry shape becomes more complicated,the plenoptic spectrum will broaden.Based on this characterization,the plenoptic sampling theorem can be derived.Secondly,plenoptic sampling theory is presented under some assumptions,such as Lambertian reflections,camera moved along a line.Because the assumptions underlying plenoptic sampling theory are not fully met in practice,some aliasing is always presented with the reconstruction.To reduce aliasing of the rendered images,we can take more multi-view images or increase the geometric information for the 3D scene density beyond that indicated by the theory.However,for complex foreground objects reconstruction often does not improve,while leading to more redundant multi-view images.Additionally,choosing the appropriate amount of geometrical information we can have alias-free rendering,and that no further improvement will be gained by adding extra geometry.Therefore,it is very important for users to be able to seek the most economical balance between image samples and geometrical information for a given reconstruction quality.We show that,in many complex scene cases(e.g.,occlusion,irregular shape and non-Lambertian),little amount of geometry simplification is challenging topic from dense range scans.The reason is that the scene geometry for irregular shape is unknown and its quantification is very complicated.Fortunately,the IBR technique no needed accurate geometry information.In this report,consider different surface shapes,the pivotal geometrical information of a complex scene can be described using some single salient points.Our approach is not dependent on dense accurate geometric reconstruction;instead we compensate for sparse 3D information using single salient points.Particularly,we study the plenoptic sampling theory based on the pivotal geometrical information.Thirdly,for the spectral analysis of the POF,an important issue is estimating the frequency of the POF.Generally,frequency domain methods can be applied to effectively estimate the frequency of the POF,such as studies that employ the spectrum of the POF and the bandwidth of the POF.However,a mathematical framework has not been fully developed for studying the sampling problems in the POF.In the time domain methods,one of the most effective approaches to estimate the frequency of signals is the autocorrelation function(ACF).This approach is employed because the ACF of a periodic signal is a periodic function,and its cycle is the same as the periodic signal based on the Wiener-Khinchin theorem.Considering that the POF is a periodic function,we can derive the ACF of the POF to estimate the maximum frequency of the POF.Our analysis reveals that the ACF will be more complicated if the variations in the scene geometry become more complex or if the depth of the scene increases.Given this characterization,the spectral support of the POF is bounded by the maximum frequency.Based on the spectral support,an essential bandwidth of the plenoptic spectrum is determined and applied to determine the minimum sampling rate of the POF.Fourthly,the success of plenoptic sampling theory depends on the assumption of a bandlimited light field signal,which is difficult to obtain for complex objects that exhibit occlusion and non-Lambertian reflection.Therefore,in most real-world scenarios,alias-free rendering is challenging,even only from the point of view of plenoptic sampling theory.Additionally,the image information that is captured is closely related to the GCC.For example,the acquisition information for a scene can be changed by using different camera positions and directions.Thus,the quality of the reconstruction is also related to the position and direction of the cameras used to capture the images.Capturing a set of multi-view images of a scene is a fundamental issue;therefore,simultaneous optimization of the positions and directions that compose the GCCs to improve the rendering quality of the virtual views is an important research topic for IBR.To be able to optimize the GCC dynamically,a mathematical model,called the non-coverage field(NCF),is proposed to quantify the relationship between the rendering quality and the GCC.We give an equation to express the rendering quality based on the NCF area,and the rendering quality of virtual views is assessed using the NCF.Then,the NCF is used to devise an algorithm to optimize the GCC to improve the rendering quality.The optimization algorithm is based on modified fuzzy c-means(MFCM)clustering.Finally,the conclusion and further works are presented.