Pansharpening Via Super-resolution Iterative Residuals Network With A Cross-scale Learning Strategy

Visible spectral remote sensing has become an important technical means for practical applications such as national security,environmental protection,national defense supervision and management,due to its wide range,fast speed,low cost,intuition,clarity,and easy interpretation.And the multi-spectral(MS) images with high-spatial-resolution(HR) are specially needed in many fields.However,due to sensor constraints,satellites usually fail to capture HR MS images,replacing by the HR panchromatic(PAN) images and the corresponding low-spatial-resolution(LR) MS images in the same coverage area.Therefore,the pansharpening technology,which exploits HR PAN images to restore the spatial resolution of the corresponding LR MS images producing fused images HR MS images,is particularly important for practical application.Recently,many approaches based on convolutional neural networks(CNNs) have been put forth for the pansharpening task,but most of them remain limited:(1)Most methods adopt the bicubic interpolation as up-sampling preprocessing and use the up-sampled LR MS image as the input of the network,which results in the incomplete input-information.(2)The majority of these methods regard the DL model as a black box problem and treat different characteristics equally.In fact,in a simple stacked convolution structure,information transmission is noneffective,thus hindering the representation ability of the network.(3)The supervised learning strategy based on Wald Protocol neglects that the mapping relationship of different scales is similar but unequal.Therefore,sometimes it may cause some scale-related problems.For address the above issues,we propose a pansharpening method via super-resolution iterative residual(SRIR) network with a cross-scale learning strategy.The major contributions are threefold.(1)For address incomplete network input-information caused by the traditional upsampling preprocessing,we built an Integrated Network.That is we introduced the Sub-Pixel Convolutional Layer to construct Sub-Pixel Convolutional Layeral network(SPCNN) for the super-resolution of LR MS images,replacing the up-sampling preprocessing and ensuring the integrity of network input information.(2)Aiming at the issue of simple stacking of convolution structures,we constructed an iterative residual network(IR-Net) in which we constructed a new spatial information injection module by branch residual network(BR-Net) and adopted the original spectral information injection module to inject spatial&spectral information in IRNet continuously,and mapped the residuals between the approximation HR MS based on guidance filter(GF) processing and the reference HR MS by the overall network SRIR,to speed up training and improve the mapping accuracy.(3)For scale effects caused by supervised learning strategies,we put forward a crossscale learning strategy in which the finer-scale unsupervised fine-tuning loss function was proposed to train the model in two-scales separately.We constructed an finescale unsupervised fine-tuning loss function based on non-reference quality evaluation indicators(QNR),by which we refined the network,after deep prior in the coarser-scale,in the finer-scale.Experiments show:(1)SRIR-based pansharpening method can obtain the best result at lower-resolution scale;(2)The scale-effect is negatively correlated with the depth of the network,that is,the deeper the network,the stronger the robustness to scale effect;(3)The cross-scale learning strategy can widely improve the performance of CNNs-based pansharpening methods in full-resolution;(4)Our method can gain the best result at fullresolution scale than other traditional and deep learning methods.The research of this paper can effectively obtain high-quality fusion HR MS images,which is very practical for follow-up applications,such as land cover monitoring,crop classification mapping and anomaly change detection.