Research On Light Field Reconstruction Algorithms Using Deep Convolutional Neural Networks

With the wide application of light field imaging in the fields of anti-terrorism monitoring,industrial inspection,microscopic diagnosis,virtual reality,and satellite reconnaissance,its research has received more and more attention.Unlike traditional 2D imaging,which integrates light intensity from different directions into a single pixel,light field imaging records the intensity of light rays in each direction separately,thereby providing additional angular information for 3D scenes.However,existing light field acquisition devices can only acquire sparse angular signals,and the resulting angular undersampling greatly limits its development.Light field reconstruction aims to generate a densely-sampled light field by synthesizing novel views from the sparsely-sampled one.Instead of building complex optical systems at a high cost,it uses mathematical modeling and algorithms to improve the angular resolution of the light field.In recent years,deep convolutional neural networks have shown great advantages in the field of computer vision,and light field reconstruction algorithms based on them have also achieved higher performance than traditional algorithms.Nonetheless,it still faces many problems.Firstly,it is difficult to take into account the reconstruction quality of texture and occlusion regions by simply using methods that are independent or dependent on scene depth information.Secondly,affected by the complexity of texture and occlusion in the scene,the synthesized image is prone to distortion in its near-edge regions.Finally,existing algorithms can only adapt to specific sampling patterns and quantities,limiting their generalization ability in practical applications.To solve the above challenges,the paper based on the convolutional neural network to analyze and study the light field reconstruction algorithms,and obtained the following research results:1.An algorithm named Light Field Reconstruction via Attention Maps of hybrid Networks(LFRHN)is proposed.It contains two parallel sub-networks that realize non-depth-based and depth-based light field reconstruction,respectively.Through attention map-based fusion,the algorithm can exploit the characteristics of the two types of light field reconstruction methods from a complementary perspective.Through quantitative and qualitative evaluation,it is proved that the image synthesized by the algorithm has satisfactory results no matter the texture areas or the occlusion regions.2.To repair the distortions of the reconstructed light field image in its near-edge regions,this paper proposes an Edge-guided Inpainting Module(EGIM).In addition,a near-edge(NE)loss function is proposed to further improve the similarity of the near-edge regions.By designing two light field reconstruction algorithms using EGIM and applying them to large disparity datasets and real-world datasets,the experimental results show that,compared with the current state-of-the-art algorithms,the synthesized images by them perform better in near-edge regions.3.To solve the problems of limited sampling mode and multiple training of the network in light field reconstruction,this paper proposes an algorithm named Light Field Reconstruction with Flexible Sampling and Arbitrary Angular Resolution(LFRFSAAR).It aims at reconstructing a light field with arbitrary angular resolution from an arbitrary number of randomly sampled views.It encodes the positions of the input and target viewpoints into the plane sweep volume(PSV)and uses it as input,which effectively solves the problem of limited sampling mode.In addition,to fully explore and fuse the features of any number of input views,the algorithm proposes an extended SE(Extended Squeeze-and-Excitation,ESE)attention mechanism based on the SE(Squeeze-and-Excitation)attention mechanism.Comprehensive experimental evaluations show that the algorithm can accurately estimate the depth map of challenging regions,maintaining high reconstruction quality.The versatility of the algorithm is improved through flexible sampling mode,as well as flexible input and output angular resolutions.