Research On Hyperspectral And Multispectral Image Fusion

Hyperspectral image(HSI)contains abundant spectral information,which enables accurate identification of materials.Therefore,they find broad application in earth observation,military detection,environmental monitoring,geological exploration,and precision agriculture.However,there is the trade-off between the spatial resolution and the spectral resolution for the imaging sensor,that is,HSI with high spectral resolution suffers from low spatial resolution,which limits the application of HSI.The imaging sensor can acquire the multispectral image(MSI)of much higher spatial resolution.Hence,fusing low-resolution HSI with high-resolution MSI of the same scene has become an effective way to obtain high-resolution HSI(fused HSI).This paper has summarized and analyzed the existing HSI-MSI fusion methods in detail,and focuses on the aforementioned main problems and challenges in HSI-MSI fusion,that is accurate representation and efficient fusion.The proposed methods can take full advantage of the spectral-spatial characteristics in HSI and have superior performance.The main contributions of this thesis are as below:(1)To accurately model the complex spatial-spectral structures of HSI,we establish a sparse tensor factorization model for the HSI-MSI fusion.The matrix factorization based fusion methods is very popular in HSI-MSI fusion,where the matrix data representation may be hard to fully make use of the inherent structures of three-dimensional HSI.To solve the problem,the proposed method decomposes the fused HSI as dictionaries of the three HSI modes and a sparse core tensor based on the Tucker decomposition,in which dictionaries of three modes represent information of each mode,and core tensor models the correlations of three modes.The similarities in spatial and spectral modes result in sparse core tensor.This decomposition incorporates the information of three modes into an unified framework.Based on the established model,we propose coupled sparse tensor factorization method and non-local sparse tensor factorization method for HSI-MSI fusion,and they are suitable for the non-blind fusion case and semi-blind fusion case,respectively.Experimental results show that the proposed method can effectively preserve the spatial and spectral structures of HSI and improve the fusion performance.(2)To precisely extract the spatial and spectral features of HSI and MSI,we establish a low-rank tensor model for HSI,and propose a low-rank tensor representation based HSIMSI fusion method.Firstly,to learn the similarities in the spectral mode,the fused HSI is decomposed as the spectral subspace and coefficients.The spectral subspace represents the spectral information of the fused HSI,and we learn it from the observed HSI via the truncated singular value decomposition.The coefficients represent the spatial information of the fused HSI,and we estimate the coefficients by using the spatial similarities.We firstly cluster the high-resolution MSI patches as many groups based on their similarities,and the patches of coefficients are also clustered according to the learned cluster structure in the MSI patches.The patches of the coefficients in each group are much similar to each other and can constitute a three-dimensional tensor,whose three modes are highly correlated.Therefore,we impose the low-rank tensor constraint on these tensors,which can effectively model the spatial similarities.We formulate the estimation of the coefficients as tensor rank regularized optimization problem,which is solved via the alternating direction method of multipliers.Experimental results show that the proposed method decreases 9.9% compared with the state-of-the-art methods in terms of spectral angle mapper.(3)Focusing on solving the problem of low computational efficiency of the model optimization based fusion methods,we present a deep convolution neural network(CNN)for HSI-MSI fusion with high efficiency.The proposed method combines the imaging models and the image prior learned via the convolution neural network.Specifically,we firstly initialize the fused HSI from the imaging models via solving a Sylvester equation to improve the quality of input HSI.Then,we map the initialized fused HSI to the reference fused HSI via deep residual learning to learn the image priors.Finally,the learned image priors are combined with the imaging models to further improve the quality of the fused HSI.Experimental results show that the computational efficiency is five times higher than the that of competitors.(4)To solve the problem that demanding training data and weak generalization ability for deep learning based fusion methods,we propose a novel HSI-MSI fusion method,which is based on subspace representation and deep transfer learning.Our method only needs to train the CNN on the more accessible gray images and can be directly used for any HSI and MSI datasets without re-training.Firstly,to exploit the high correlations among the spectral bands,we approximate the desired fused HSI with the low-dimensional subspace multiplied by the coefficients,which can not only speed up the algorithm but also lead to more accurate recovery.Since the spectral information mainly exists in the low-resolution HSI,we learn the subspace from it via truncated singular value decomposition.Due to the powerful learning performance of CNN,we use the well-trained CNN for gray image denoising to regularize the estimation of coefficients.Specifically,we plug the CNN denoiser into the alternating direction method of multipliers algorithm to estimate the coefficients.Experimental results show that our method has good generalization ability and markedly improve the fusion performance of images with different spatial and spectral resolutions.