| Spectral image is rich in image information and spectral information and plays an irreplaceable role in the field of computer vision.However,due to hardware limitations and other reasons,the spatial resolution of the spectral image collected by the sensor is low,which is not conducive to subsequent analysis and processing As a branch of image fusion technology,spectral pan-sharpening technology is considered to be one of the most important techniques to improve the spatial resolution of spectral images.The technology can fuse the spatial information of panchromatic image with the spectral information of spectral image to obtain high resolution spectral image to meet the subsequent needs.In recent years,deep learning technology has become another important means in the field of computer vision.Among them,convolutional neural network,as a new research branch,has achieved significant breakthroughs in many research fields.Based on convolutional neural network,this paper focuses on end-to-end spectral pan-sharpening technique based on deep learning.Its main contents are as follows:(1)To solve the problems of spectral distortion and missing spatial details,a panchromatic and multispectral image fusion method based on 3D multiscale attention is proposed in this paper.In this method,based on U-Net network,3D convolution is introduced to maintain the spectral features of each pixel in view of the spectral continuity of multispectral images.Attention mechanism is used instead of jumping connection to connect context information,which makes the network focus on the spatial details of feature map.Furthermore,a new multi-scale loss function is proposed based on the scale invariance of image and the multi-scale extraction characteristic of U-Net,so that the network can better extract multi-scale spatial information of image.(2)To solve the problems of the existing hyperspectral image pan-sharpening network is too large and spatial spectral features are difficult to learn,a panchromatic and hyperspectral image fusion framework based on deep mutual learning strategy is proposed in this paper.The framework introduces a deep mutual learning strategy,which consists of three parts: preprocessing,network structure and information exchange.In this framework,spectral network and spatial network are designed to learn spectral and spatial features.In the information exchange section,a set of hybird loss functions based on mutual learning strategies are proposed to transfer information with different features.In addition,based on Res Net,this paper designs a multi-scale residual network to extract spatial features through grouping convolution method.Experiments show that the deep mutual learning strategy is effective in image fusion.The data show that the proposed framework has good effects on spatial information enhancement and spectral information retention.In addition,the proposed multi-scale residual network is also more competitive than other methods. |
