| With the advent of the era of high-resolution remote sensing(RS)big data,how to efficiently mine the rich information from massive RS data for land cover classification has become a research hotspot.However,due to the type complexity and ambiguity as well as size difference of land objects in high-resolution RS images,synergistically using the complementary information between heterogeneous RS images,which is one of the main ways to improve the performance of collaborative land cover classification(CLCC).Recently,deep learning progressively mines deep features to solve the heterogeneous gap of high-resolution and heterogeneous RS images,which has become a mainstream framework for the CLCC.However,under the framework of deep learning,how to extract,optimize and fuse multimodal remote sensing data and then obtain more discriminative complementary information for land cover classification,it is a new challenges and has not been well resolved.We focus on deep learning to carry out systematic and in-depth research on the CLCC methods under the conditions of complete heterogeneous data and lack of heterogeneous data in test sets,and then propose novel solutions and processing methods from three aspects:heterogeneous feature association,heterogeneous feature selection,and hetero-geneous feature semantic consistency construction.The main work and contributions he completed are as followsFor the correlation between the optical and SAR features,a Heterogeneous Compact Bilinear Fusion Network(HCBFNet)is proposed,which realizes heterogeneous bilinear correlation between the optical and SAR features.First,to avoid mutual interference of the optical and SAR feature extractor,the pseudo-siamese network is used as the feature extractor.Then,the Second-Order Information-based Feature Selection(SOI-FS)module constructs the second-order statistics between the global mean and maximum information to guide the learning of compact bilinear fusion features,which alleviates the problem of reducing the performance of fusion features and increasing the parameters of the network model.The experimental results show that the proposed method can effectively construct the bilinear correlation to improve the CLCC accuracy.For the selection between the optical and SAR features,a Collaborative Attention-based Heterogeneous Gated Fusion Network(CA-HGFNet)is proposed.First,Multi-stages Feature Extractor(MS-FE)consists of pseudo-siamese and siamese components,pseudo-siamese component overcomes the heterogeneity of the optical and SAR data,and the siamese component guides semantically consistent representation of heteroge-neous features.Then,Multi-modal Collaborative Interaction Attention(MCIA)module efficiently realizes the interaction between the optical and SAR features.Finally,Het-erogeneous Gated Weighted Fusion(HGWF)realizes adaptive weighted fusion between optical and SAR features for the CLCC.It is proved that the CA-HGFNet effectively al-leviates semantic gap of heterogeneous data through the secondary interaction of optical and SAR features,meanwhile the CA-HGFNet adaptively highlights the contribution of different source features,which obtains superiority over other CLCC methods.A Multi-modal Semantic Consistency-Based Fusion Architecture Search(M~2SC-FAS)model is proposed to address the shortcomings of the hand-designed convolutional neural network as remote sensing feature extraction.The M~2SC-FAS uses neural archi-tecture search strategy to automatically learn optical-and SAR-specific feature extraction networks.Furthermore,with the increase of the number of interactions between the opti-cal and SAR features in above researches 1 and 2,the performance of the CLCC is also improved.The M~2SC-FAS introduces a heterogeneous semantic consistency constraint based on the representation dissimilarity matrix to guide the optimal dense fusion archi-tecture,which avoids the subjectivity of the artificially set dense fusion of heterogeneous features.The experimental results show that the proposed method is significantly better than the pre-trained model-based CLCC model in terms of accuracy and universality.In practical CLCC task under the lack of heterogeneous data,a Heterogeneous Salient Privileged Distillation Network(HSPDNet)is proposed,which efficiently implements the privileged information of training samples to guide the CLCC with missing data.With the help of the attention mechanism,Dynamic-Hierarchical Attention Distillation(DH-AD)module not only highlights key information from feature map,and then avoid noise and useless information to interfere the transfer of privileged information,but also dynam-ically assigns the contribution of different layers of privileged information in different training stages to improve“students”model.Experimental results show that the proposed HSPDNet can make use of the optical and SAR data to assist the CLCC of missing data,and approach the performance of deep CLCC method using complete multimodal data.The above research content 4 ignores the problem that each component of privileged information interferes with each other,a Dense Adaptive Grouping Distillation Network(DAGDNet)is proposed.Firstly,a Interactive Gated-based Feature Grouping(IG-FG)module not only gradually realizes the information interaction of the optical and SAR features,but also decouples the optical and SAR features to obtain various features com-ponents,which provides a prerequisite for the realization of feature grouping distillation.Then,a Multi-stage Adaptive Distillation Learning(MS-ADL)adaptively highlights dif-ferent layers of privileged information and each components of the same layer of priv-ileged information,adaptively,which can guide the learning of the“student”.Experi-mental results show the proposed method can effectively overcome mutual interference of components in the process of privileged information distillation,and further improve the CLCC accuracy under the absence of a certain source data in the test set. |
