Research On Hyperspectral Image Super Resolution Based On 3D Multi-scale Neural Network

Hyperspectral images not only contain the spatial information of the scene but also contain rich spectral informatioin.Hyperspectral images are widely used in many fields.However,due to the hardware limitation of the hyperspectral imaging sensor,while the hyperspectral image contains rich spectral resolution,its spatial resolution is very low.When the hardware limitations cannot be solved in a short time,it is particularly important to improve the spatial resolution of hyperspectral images by studying the super-resolution algorithms of hyperspectral images.In addition,the super-resolution method of single-image hyperspectral images does not require any auxiliary images and has good flexibility.This paper proposes two super-resolution algorithms of single image hyperspectral image based on three-dimensional multi-scale neural network combined with deep learning.The main contributions of this article are as follows:（1）we propose a multi-scale feature fusion network with 3D（MFFA-3D）convolution by cascading the MFFA-3D block.The MFFA-3D block includes a group multi-scale feature fusion part and a multi-scale feature aggregation part.In group multi-scale feature fusion part,a novel group multi-scale feature fusion method is proposed.Group feature fusion module and two-step multiscale module are proposed in multi-scale feature aggregation part.In order to prevent spectral distortion,a spectral gradient loss function is proposed and combined with the mean square error loss function to form the final loss function.Since the proposed super-resolution network is a full 3D convolutional network,our method can perform direct super-resolution transfer even if the number of the bands of test images is different from that of the training images.This super-resolution method can save a lot of time and improve efficiency.（2）we propose a single hyperspectral image super-resolution method based on multi-network ensemble.Our method includes single network super-resolution part and multi-network ensemble part.In single network super-resolution part,we construct two 3D multi-scale mixed attention networks by cascading 3D multi-scale mixed attention block to restore high-resolution hyperspectral images.3D multi-scale mixed attention block consists of the 3D Res2net module and the mixed attention module.3D Res2net module is a simple and effective multi-scale method.The mixed attention module is proposed by combining the first-order and second-order statistics of features.In addition,we use the mutual learning loss between two 3D multi-scale mixed attention network so that they can learn from each other.In ensemble part,the fusion module is proposed to merge the output of each 3D multi-scale mixed attention network.