| Autonomous vehicles are often equipped with multiple sensors to acquire multimodal data.Combining different modal data can capture more spatial and contextual information,thereby helping the vehicle to perceive the surrounding environment robustly and accurately.Especially in recent years,with the rapid growth of data resources and computing power,the development of deep multimodal fusion models has been promoted.However,while multimodal data provides rich information,it also brings complexity and redundancy.How to use the complementary information between multimodalities as much as possible to obtain the optimal joint representation and improve the performance of the perception system is still unresolved.This paper uses visible and thermal images as multi-modal sources to study the two key perception tasks of pedestrian detection and scene segmentation in autonomous driving scenarios.The main work of this paper is as follows:(1)A pedestrian detection method based on the fusion of visible and thermal images is proposed.Considering the problem of spatial misalignment of data in the data set,an alignment complementary fusion module is designed.Before fusing the complementary information of visible and thermal images,deformable convolution is used to automatically deform the features in space to eliminate the misalignment problem.Secondly,due to the different contributions of different modalities to pedestrian detection under different light conditions,the introduction of a light-aware module makes more effective use of the differentiated advantages of visible and thermal images.The experimental results show that the method of this paper achieves the best detection effect on the improved public Kaist test set compared with the advanced methods in recent years.(2)A scene segmentation method based on the fusion of visible and thermal images is proposed.At first,atrous convolution is introduced to increase the field of view,and a multi-scale parallel module is formed without adding additional parameters,which frequently appears in the deep layers of the model encoder.Secondly,a feature transformation module is constructed to automatically learn the translation and scaling factors required for transformation,reducing the data distribution difference between thermal and visible features,thereby promoting the fusion efficiency between different modalities.The encoder features are fed into the feature transformation module in the decoder by skip connection,which is conducive to the information transfer within the model.Finally,the method shows significant advantages over other methods on public multispectral segmentation datasets.Starting from the characteristics of visible and thermal images,this paper further designs a multi-modal pedestrian detection method and a scene segmentation method according to the needs of different sensing tasks and the actual distribution of public datasets.Corresponding modules are designed to effectively alleviate the problem of spatial misalignment of multimodal data and imbalanced distribution of multimodal features.The proposal of these methods and modules has positive significance for the field of multimodal data fusion perception. |
