Research On Deep Learning Based Hyperspectral Image Fusion Methods

The hyperspectral image(HSI)with the characteristics of very high spectral resolution and synchronous acquisition of images and spectra,has been widely applied in civil,military and many other fields.With the constant increase of demand,plenty of remote sensing applications require images with both high spectral resolution and high spatial resolution.However,in practice,the HSI often exhibits a relatively low spatial resolution,which limits the precise interpretation and application effect of HSI.The panchromatic(PAN)sensor can provide PAN images with high spatial resolution.Therefore,to effectively improve the spatial resolution of HSI,it is particularly urgent and important to fuse the HSI with the PAN image through efficient and reliable hyperspectral image fusion methods.Focusing on the core issue of how to use the PAN image to enhance the spatial resolution of HSI and maintain its spectral information simultaneously,this thesis thoroughly investigates deep learning based hyperspectral image fusion methods to fully exploit and combine the advantages of PAN and hyperspectral images,and to break through the limitations of traditional linear models.In this thesis,we overcome the problems of poor spectral fidelity,spatial information loss,high complexity of models,limited discriminative ability of networks,and inaccurate detail injection in existing fusion methods,and explore and realize high-accuracy,high-efficiency and high-intelligence methods for the fusion of hyperspectral and PAN images.The main contributions of this thesis are summarized as follows:(1)Research on hyperspectral image fusion based on guided filter and deep residual networkCompared with PAN and multispectral images,the HSI usually possesses lower spatial resolution and wider spectral range.Thus the fusion of hyperspectral and PAN images is more prone to suffering from spatial blur and spectral distortion.Aiming at this problem,we propose a hyperspectral image fusion method based on guided filter and deep residual network.First,in order to generate the initialized fused image with spectral information preserved,the guided filter is performed on the upsampled HSI with the edge detail enhanced PAN image serving as the guidance image.Then,to further improve the quality of the initialized HSI,a deep residual network for boosting the fusion accuracy is designed.The experimental results of both ground-based and airborne hyperspectral datasets show that the proposed method enhances the edge details effectively and improves the fusion accuracy significantly.(2)Research on a deep residual spatial attention network for hyperspectral image fusion The existing methods do not fully consider the spatial structure information of HSI and the different importance of features extracted from different image regions,leading to low fusion accuracy.To solve this problem,we propose a deep residual spatial attention network for hyperspectral image fusion,which not only considers the spatial information of both PAN and hyperspectral images simultaneously,but also adaptively learns more useful features of spatial locations for details enhancement of the fused HSI.Firstly,the spatial information of PAN and hyperspectral images is extracted by using structure tensor and guided filter,respectively.Then,the obtained complete spatial information of both PAN and hyperspectral images is fed into the constructed deep residual spatial attention network to map the residual HSI,where the spatial attentional features that are more useful for the fusion are fully extracted and utilized.Experimental results of both airborne and spaceborne hyperspectral datasets demonstrate that compared with the current algorithms,the proposed method can achieve superior results in terms of both spatial enhancement and spectral preservation.(3)Research on hyperspectral image fusion based on deep prior and dual-attention residual networkTo address the problems of ineffective restoration of detail information during upsampling the HSI and limited discriminative ability of networks,a hyperspectral image fusion method based on deep prior and dual-attention residual network is proposed in this thesis.First,we upsample the low-resolution HSI through the deep hyperspectral prior algorithm,which can effectively improve spatial resolution and better preserve spectral information without large amount of data for training.The upsampled result is then concatenated with the PAN image along the spectral dimension to form the input of the dual-attention residual network,where several channel-spatial attention residual blocks are stacked to adaptively highlight more important features of spectral channels and spatial locations for boosting the fusion performance,while suppressing trivial ones simultaneously.The experimental results of both simulated and real hyperspectral datasets show that the proposed method can further improve the quality of fused images and has strong generalization capability.(4)Research on an adaptive feature modulation network for hyperspectral image fusion To further overcome the problems of insufficient utilization and inaccurate injection of PAN details,and unsatisfactory balance between spectral and spatial quality in the fusion process,we propose a hyperspectral image fusion method based on an adaptive feature modulation network in this thesis.The overall architecture of this method follows the detail injection model,which has a clear interpretation.First,the comprehensive spatial detail information is extracted directly and effectively from the high-frequency features of the PAN image via an octave convolution unit.Meanwhile,the spatial and spectral separable 3D convolution units with multiple kernel sizes are designed to efficiently capture multiscale spatial-spectral features of the HSI.Subsequently,a detail injection scheme based on adaptive feature modulation is proposed.Taking the PAN details as prior and performing affine transformation on the HSI features,the proposed scheme enables not only to enhance the spatial information effectively,but also to adjust the injected details adaptively to ensure the spectral fidelity of the fused HSI.Experimental results of both simulated and real hyperspectral datasets demonstrate that the proposed method obtains much improvement in terms of both quantitative and visual evaluations.