Research On Pedestrian Detection Based On Multispectral Information Fusion

Pedestrian detection based on multispectral information fusion is the core technology in all-weather applications such as unmanned driving and smart video surveillance.It means that for the same scene,both infrared and visible sensors are used to obtain multispectral(infrared and visible)images as input sources.Then the complementarity of the infrared and visible can be used to distinguish pedestrian and backgrounds,complete the positioning of pedestrian targets,and make up for the disadvantages of traditional pedestrian detection that only uses visible images as the input source,to deal with changes in illumination and bad weather.Since the performance of multispectral pedestrian detection largely depends on the pros and cons of the infrared and visible fusion mechanism,and the infrared and visible belong to two modalities,when they are fused,the contribution of each modality to the pedestrian detection network cannot be divided solely on the basis of day and night.Multispectral pedestrian detection not only has practical application value,but also faces many challenges,which has become a hot and difficult point in academic research.This thesis mainly focuses on designing a fusion mechanism based on infrared and visible modalities in terms of selecting the appropriate fusion timing and fusion method,to fuse information of infrared and visible modalities and provide complementary information for pedestrian detection,so as to improve the performance of pedestrian detection.The main research contents and results are as follows:(1)Multispectral pedestrian detection network under multi-stage feature fusion information reusing mechanism.In view of the fact that some multispectral pedestrian detection algorithms only choose to fuse on a single feature layer,there is a waste of feature information,resulting in pedestrian missed detections and false detections,a multispectral pedestrian detection network based on multi-stage feature fusion method was proposed.First,the paired infrared and visible images were extracted through the two-stream VGG16 network to get the features of the middle layer for early features stacking,and the features fused to obtain the early fusion features.Then the fusion features were used to generate pedestrian proposals.Unlike the single-stage fusion method that directly used the early fusion features to complete the subsequent detection tasks,the multi-stage feature fusion strategy mapped each pedestrian proposal back to three multi-modal features for multi-feature pooling,then performed weighted fusion of high-level pooling features,and used high-level feature pooling strategy to complete the combination of high-level and low-level features.The pooling features were finally sent to the fully connected layers to complete the detection task.The experimental results indicated that the average precision of the Kaist multispectral pedestrian detection dataset reached 76.24%,and the log-average miss rate drops to 27.63%.(2)Multispectral pedestrian detection network under modal adaptive weight learning mechanism.A pedestrian detection network based on the weight learning of fusing multi-modal information was developed to address the issues of the pedestrian detection method based on infrared and visible modal fusion in adapting to changes in the external environment.First,unlike the fusion method used in several recent studies in which two modalities were stacked directly,the weight learning fusion network reflected different contributions of the modalities to the pedestrian detection task under different environmental conditions.The differences between the two modalities were determined through dual-stream network learning.Next,based on the current characteristics of each modal feature,the weight learning fusion network assigned the corresponding weights to each modal feature to generate the fusion feature by performing weighted fusion autonomously.Finally,a new feature pyramid based on the fusion feature was generated,and previous information about the pedestrian was improved by changing the size and density of prior boxes to complete the pedestrian detection task.The experimental results indicated that the average precision of the Kaist multispectral pedestrian detection dataset reached 76.55%,and the log-average miss rate drops to 26.96%.(3)Pedestrian detection network under multi-modal cross-guided learning mechanism.Aiming that most of the current generation modules based on infrared and visible modal features are independent of each other,lack long-term dependence between the modalities.The large differences in different modal features will affect the fusion result and easily lead to the problem of target false detections and missed detections,a pedestrian detection network with multi-modal cross-guided learning was proposed.First,the paired multi-modal images were sent to the feature generation module to generate high and low layer features.Starting from the middle layer feature,the paired multi-modal features were sent to the weight-aware module.Unlike some current fusion mechanisms only used to generate fusion features,the weight-aware module also outputted the weighted features of each mode.Then the weighted features of each modal were returned to the feature generation module of another modal,so that the weighted information is gradually transmitted to the next layer in a joint cross-guidance manner to establish long-term dependence between modalities.At the same time,the fusion features were also input to the weight-aware module of the next stage to strengthen the connection between the fusion features at different stages and obtain more discriminative features.Finally,the feature layers of different scales were selected and sent to the detection layers to generate the position and score of the pedestrian target.The experimental results indicated that the average precision of the Kaist multispectral pedestrian detection dataset reached 77.16%,and the log-average miss rate drops to 25.03%.