| Light field imaging is becoming more and more popular due to its capability to record hunderds of views in a single shot,enabling novel imaging applications such as digital refocusing,tomography and all-in-focus imaging.It has achieved significant improvements over previous imaging methods in many fields,e.g.,life science,experimental physics and astronomy.However,the existing light field acquisition devices are fundamentally limited by the imaging trade-off,which results in huge amount of data and low effective resolution.Because the spatial and angular samplings have not been fully decouplied theoretically,existing processing techniques are time-consuming and suffer from low accuracy.In this dissertation,based on the two-parallel-planes parameterization in light field,we study the properties of the light field imaging in both spatial and angular dimensions,which support the new insights for several traditional applications or new questions.The main contributions include:(1)The spatio-angular consistency in 4D light field is derived.By analyzing the two-parallel-planes parameterization in light field,it is found that the light field is redundant and there is a similarity between the spatial and angular dimensions,which is called the spatio-angular consitency.Furthermore,there is an occlusion consistency in occlusion boundary areas,i.e.,the boundaries between the occluder and object in the spatial dimension are consistent with the boundaries between the occluded and un-occluded views in the angular dimension.Specifically,for a real light field camera which samples discrete rays,the effective resolution is determined by the scene structure and the spatial and angular resolutions,and can be higher than the spatial resolution.(2)An anti-occlusion depth estimation algorithm is proposed.Based on the spatioangular consistency in occlusion area,we construct the light field occlusion model.With the guidance of such a model,the distribution of the occluded and un-occluded views in the angular dimension can be derived from the patch analysis in the spatial dimension.Then,a novel anti-occlusion energy function is constructed based on the un-occluded views.Experimental results demonstrate the advantages of the proposed algorithm over existing methods especially in occlusion areas.(3)Based on the spatio-angular consistency,we propose the light field superpixel(LFSP)for sparsely representing the light field and a novel segmentation algorithm.The light field superpixel is defined as the ray set emitted from a proximate,similar and continuous surface in 3D.This definition not only decreases the light field redundancy,but also eliminates the segmentation ambiguities for defocused or occluded regions introduced by traditional 2D superpixel segmentation.By building a clique system containing 80 neighbours in a light field,a robust refocus-invariant LFSP segmentation algorithm is developed.Experimental results on both synthetic and real light field datasets demonstrate the advantages over current state-of-the-arts in terms of both of traditional evaluation metrics and light field metrics,i.e.,the refocus-invariant and full-sliced properities.(4)An efficient and anti-occlusion algorithm for full view optical flow estimation of a light field video is developed.Combining the spatio-angular consistency and the LFSP,the problem of full view optical flow estimation is converted to recovering the motion matrix of the LFSP in 3D space.So,we model the LFSP as a slanted plane in3 D.By optimizing the optical flow between two central view images,a common motion model is then fitted and employed to all the views for full view optical flow recovery.As the functions of slanted planes varies from superpixel to superpixel,the sharpness of boundaries in the reconstructed optical flow field is guaranteed.The utilizability and efficiency of our method is verified using synthetically generated image sequences and real-world imagery captured using a Lytro Illum camera.(5)A learning-based method for light field super-resolution is proposed.The spatioangular consistency indicates that the effective sampling resolution is generally higher than the number of micro-lenses in a light field camera.This contribution makes it theoretically possible to super-resolve a light field.Further optical analysis proves the “2D predictable series”nature of a 4D light field,which provides new insights for analyzing light field using series processing techniques.To model this nature,a specifically designed EPI based CNN-LSTM network is proposed to super-resolve the spatial and angular dimensions simultaneously.Rather than leveraging semantic information,our network focuses on extracting geometric continuity in the EPI.This gives our method an improved generalization ability and makes it applicable to a wide range including previously unseen scenes.Experiments on both synthetic and real light fields demonstrate the improvements over state-of-the-arts,especially in large disparity areas where the EPI line is discontinuous. |
