Spatial-spectral Hyperspectral Unmixing Based On Conditional Generative Adversarial Network

Hyperspectral image(HSI)has been widely used in mineral detection,environmental disaster monitoring,military,and many other fields,and the research works on it attract considerable attention.HSI is obtained by imaging spectrometers,having dozens to hundreds of narrow band spectral information in each pixel,and the abundant spectral information significantly enhances its ability to detect material attribute information.Nevertheless,as the low spatial resolution of HSI,each pixel typically is a mixture of several pure substances.These mixed observations hinder the application and development of HSI analysis technology.Therefore,hyperspectral unmixing(HSU),which aims to estimate the pure substances(endmembers)of the mixed pixels in HSI as well as their fractional abundances,is a significant and challenging task in the area of hyperspectral remote sensing.In this paper,the deep learning method is used to provide a new perspective for solving the unmixing problem,and we proposed a conditional adversarial network as the unmixing framework.The main contributions of this paper are as follows:1.Existing unmixing methods utilize prior knowledge about the abundance distribution to solve the regularization optimization problem,where the difficulty lies in choosing appropriate prior knowledge and solving the complex regularization optimization problem.This paper proposes a conditional generative adversarial network based on convolutional networks(sc GAN)as a generic framework for unmixing,which is applicable to the unmixing problem where the abundance vector of partially mixed images is known,based on the following assumptions: the unmixing process from pixel to abundance can be regarded as a transformation of two modalities with an internal specific relationship.The proposed scGAN is composed of a generator and a discriminator.The former completes the modal conversion from mixed hyperspectral pixel to abundance,avoiding solving the sophisticated regularization optimization problem.The latter is used to judge whether the distribution and structure of generated abundance are the same as the true one,eliminating the manual effort for the introduction of prior knowledge and regularization terms under different unmixing scenarios.Generator and discriminator train each other with adversarial strategies.In addition,HSI has certain spatial structure information.In order to make full use of the spatial information of HSI,the HSI patches with spatial information are input into the generator,and the convolution network is adopted to extract the joint spatial spectral features of the HSI patches to generate the abundance of the central pixel,achieving better unmixing performance.Then,this article proposes a data synthesis method based on simple linear iterative clustering and random splitting to synthesize hyperspectral data with spatial structure to verify the effectiveness of the proposed unmixing method.For the hyperspectral dataset without known abundance,such as the unmixing problem for Cuprite dataset,we randomly select some pixels from the mixed pixels and unmix them with classical and effective methods.The unmixing result is regarded as the approximation of the real abundance.Then we train the network and unmix the remaining mixed pixels through the network.2.The structure of HSI patches is multivariate and the correlation between the domain hyperspectral pixels and central hyperspectral pixel in different HSI patches is different,which means that the fused weights should be adaptive.To utilize the hyperspectral spatial information more precisely,based on the first point,this paper designs a conditional generative adversarial unmixing network based on Patch Transformer as an improved algorithm,where the generator is composed of the Patch Transformer module for abundance generation.The sub-attention head of multi-head attention in Patch Transformer focuses on the abundance characteristic information of each endmember subspace,and calculates the adaptive attention score of inter-domain pixels to capture the internal pixel correlation of HSI patches,which acts as the fusion feature weight to distinguish effective spatial information from interference information.Patch Transformer leverages the collaborative spatial-spectral information in a fine-grained way to achieve optimization of the unmixing process.3.The above two methods are mainly aimed at the unmixing problem for the hyperspectral datasets with known abundance labels of partially mixed pixels.However,it is difficult to obtain the abundance labels of some hyperspectral datasets.Most of the existing autoencoder unmixing methods are based on minimizing reconstruction error of manual design to train the network and the original pixels with noise and outliers will participate in the calculation of reconstruction error,which may result in meaningless unmixing results in the case of limited hyperspectral training samples.Therefore,this paper proposes a conditional generative adversarial network unmixing algorithm based on an autoencoder,which solves the unmixing problem without known abundance labels in the case of given endmember by using the structure of autoencoder network.The algorithm adopts discriminator network to judge whether the reconstructed pixels are similar to the original pixels,and the more abstract hyperspectral pixel features learned by the discriminator are used as the reconstruction errors of the autoencoder,which alleviates the noise interference in the reconstruction process to a certain extent,achieving better unmixing performance.