Deep Learning-based Methods For Light Field Images Super-Resolution

Light field image super-resolution is crucial in the research of light field image processing.Unlike traditional 2D images,a single exposure of light field camera can record the structural information of the entire four-dimensional light field,including the intensity and direction of light,which supports multiple computer vision tasks.However,due to the limited resolution of light field camera sensors,the spatial resolution of the light field sub-views are too low to limit the application of the LFI.At the same time,the decomposed sub-views of the LFI have the problems of low brightness and high noise,which affect the quality of the LFI.To this end,this paper proposes two spatial super-resolution algorithms for LFIs.The main contents include:1)A spatial super-resolution algorithm for light field images that fuses global and local features is proposed,which improves the ability to model the global and local relationships of light field sub-views.In this network,an improved 4D zero-reference depth curve estimation network(4D Zero-DCE-Net)is first utilized to improve the brightness of the LFI.Aiming at the low spatial resolution of light field images,a GAN-based spatial super-resolution network model for LFI is proposed.The generator consists of three parts.the first part is a network structure that combines Transformer and 4D convolution in parallel,which can capture the global and local details of the image in a shallow network layer.The second part is an interactive fusion attention module IFAM,which can effectively fuse the global self-attention and local detail information obtained by the above two branches.The third part is a reconstruction module PS-PA that enables the network to focus on useful information to improve the spatial resolution of the entire light field.Finally,the relative discriminator is used to guide the training of the generator.Extensive experiments prove that the algorithm can effectively improve the resolution of real-world LFI.2)A real-world light-field image spatial super-resolution algorithm via a hybrid imaging system is proposed,and DSLR images are introduced to provide abundant high-frequency detail information for light-field images.The hybrid imaging system is composed of a light field camera and a Canon 70 D camera.The algorithm explores and exploits the abundant structural information of LFI and the high-frequency detail information of aligned DSLR images.The network mainly consists of three steps.First,the RANSAC-Flow method is adopted to align the DSLR image to each upsampled sub-views of the corresponding light field.Then a 4D residual channel attention information aggregation module is proposed to extract the structural features of all sub-views in the light field.At the same time,a 3D residual dense Swin Transformer is adopted to extract the high-frequency details from the aligned DSLR images.Finally,the reconstructed information of all sub-views of the light field and the high-frequency information of the aligned DSLR images enter the coupled feedback module(CFB)for fusion to reconstruct high-resolution light field sub-views.Furthermore,a dataset of real-world light field and DSLR image pairings is also proposed,and the proposed neural network is trained through an unsupervised learning strategy.Extensive quantitative and qualitative evaluations on real-world LFI demonstrate the effectiveness of the proposed algorithm.