| As the core technology of next-generation virtual reality,digital light field has received more and more attention.However,limited by hardware technology,the imaging resolution of current light field imaging devices in the dimensions of time(frame rate),space(subaperture image resolution),and angle(sub-aperture array)is still low.Therefore,highresolution light field rendering has always been a hot spot in the field of computational imaging.Combined with the latest progress in deep learning,this paper focuses on the superresolution reconstruction method of light field images in the spatio-temporal-angular dimension.Although many deep learning-based light field super-resolution methods have been proposed,the existing models still have many limitations: 1)model complexity is coupled with resolution,and increases rapidly with the increase of resolution;2)only super-resolution reconstruction with limited fixed magnification can be achieved;3)super-resolution methods for light-field video have not been effectively explored.Aiming at the above limitations,this paper proposes a light field super-resolution method based on local implicit functions.Using the properties of implicit functions,the decoupling of model complexity and resolution can be achieved.In theory,super-resolution reconstruction at any magnification can be achieved.The main work and contributions of this paper are as follows:1)In view of the limitation that the existing model can only achieve spatial superresolution reconstruction of light field with limited magnification,we proposes a superresolution model LFSSRNet based on spatial local implicit function.It uses the spatial local implicit function to establish the mapping relationship between the continuous image representation and the pixel coordinates,and learns the spatial local implicit function from the light field image implicit code and its pixel coordinates through a multilayer perceptron(MLP),so that it can predict the color information of any pixel coordinates,and the light field super-resolution reconstruction of any spatial resolution can be realized.In order to learn the spatial local implicit function more effectively,this paper uses cascaded Transformer modules to extract the implicit code of light field images.Experiments show that LFSSRNet achieves high accuracy on multiple datasets and has good generalization performance on multiple resolutions.2)Aiming at the limitation that the existing model can only achieve limited magnification of light field angular super-resolution reconstruction,we proposes a light field angular superresolution model LFASRNet based on the spatial-angular local implicit function.The method takes Epipolar Plane Image(EPI)as input,constructs a three-layer scale EPI pyramid,and sends them to the feature extraction module to obtain the deep semantic features of multiscale fusion.Finally,the spatial-angular local implicit function is used to establish the EPI pixel implicit code and the mapping relationship between its coordinates and pixel values,so as to realize the super-resolution reconstruction of light field with any angular resolution.Experiments show that LFASRNet achieves SOTA accuracy on multiple datasets and can output reconstruction results with arbitrary angular resolutions.3)In view of the limitation of existing models that are difficult to achieve light-field video frame interpolation,we proposes a temporal-angular super-resolution model LFATSRNet based on hybrid local implicit functions.The method firstly establishes the mapping relationship between the mixed latent coding of the previous and next frames to the pixels of the synthetic video frame through the shuffling,convolution and attention mechanism based on macro pixels.Then,the internal structure information of macro-pixels and the correlation between adjacent macro-pixels are learned through the angular detail refine module to realize end-to-end light field video simultaneous frame interpolation and angular super-resolution reconstruction.Experiments show that LFATSRNet has a good generalization effect on scenes with different motion amplitudes. |
