A Study On Multispectral Pedestrian Detection And Segmentation Based On Deep Convolutional Neural Networks

Pedestrian detection and segmentation is an essential and significant task in video surveillance,advanced driver assistance systems and so on.With the big success of deep convolutional neural networks(CNNs)in machine vision,pedestrian detection and segmentation based on CNNs has also attracted the attention of scholars.On the other hand,pedestrian detection and segmentation based on visible images usually fail in the nighttime due to the bad visibility.Combination of visible and far infrared images has become an effective technique for all-day pedestrian detection and segmentation.Based on the success of deep convolution networks and the introduction of large-scale multispectral pedestrian detection database KAIST,this paper aims to explore effective ways to fuse visible and thermal images in CNN-based pedestrian detection and segmentation methods.Detailed work are as follow:First,based on a deep and thorough understanding of the CNN-based pedestrian detectors,two fusion architectures are developed based on the advanced SSD framework.By a proper network architecture design,information from visible and thermal images is effectively fused and the performance of pedestrian detection is improved.Second,the performance of multiple pixel-level image fusion methods in CNN-based pedestrian detectors is tested.Combination of pixel-level methods and CNN fusion architectures is also studied.Extensive experiments show that the combination of pixel-level fusion methods and CNN fusion architectures can usually further improve the pedestrian detection performance.Especially,the combination of late-fusion architecture with the joint bilateral filter method can significantly reduce the log-average miss rate by around 11%in the nighttime.Finally,based on a deep and thorough understanding of the CNN-based image segmentation frameworks,fusion architectures for multispectral pedestrian segmentation are designed and implemented based on the PSPNet framework.Experimental results show that the developed networks can effectively explore the information in visible and thermal images and improve the segmentation performance.Overall,to effectively explore the information in visible and thermal images,different image-fusion methods and CNN-fusion architectures are studied in this thesis for pedestrian detection and segmentation.Extensive experiments have been carried out to propose effective strategies to achieve all-day pedestrian detection and segmentation.The conclusions are valuable for the future applications of vision-based pedestrian detection and segmentation.