Joint Semantic Segmentation Algorithm For Visible And Infrared Images

With the continuous development of deep learning algorithms,significant progress has been made in semantic segmentation algorithms based on visible images.However,due to the poor imaging quality of visible images under adverse lighting conditions,the semantic segmentation results are not ideal.Recently,a joint semantic segmentation algorithm based on visible and thermal infrared images is being studied.The joint semantic segmentation algorithm combines thermal infrared images to solve the semantic segmentation problems in adverse environments such as low light,night,fog,and backlight.The main research areas of this thesis are:Firstly,a high-precision joint semantic segmentation network for thermal infrared and visible images named RFSSNet(Stepwise Segmentation Network based on Residual Fusion)is designed.In order to effectively balance the importance of thermal infrared and visible images features and filter out unique noise from both modalities,a feature residual fusion method is designed.To improve the detail accuracy of semantic segmentation prediction result,an encoder-decoder structure is used and the information from each level in the encoder is linked to the decoder with skip connections.This method effectively enhances the detail information.Furthermore,a multi-level mixed loss function is designed to guide the decoder to construct semantic labels in a step-by-step manner.Experimental results show that the design of feature residual fusion for thermal infrared and visible images,as well as the multi-level mixed loss function at the decoder end,are effective and play a key role in improving the algorithm performance.Secondly,a real-time joint semantic segmentation method for thermal infrared and visible images,called Fast Mix Seg(Fast Mix Semantic Segmentation)is proposed.Fast Mix Seg aims to address the problem of large network parameters and slow inference speed in current joint semantic segmentation algorithms for thermal infrared and visible images.A lightweight dual-branch joint semantic segmentation algorithm is proposed based on the Mobile Net V3 network.In the case of a limited amount of data for joint semantic segmentation algorithms,the concept of transfer learning is used to initialize the network with pre-trained parameters from Mobile Net V3.In order to better integrate infrared and visible image features,a soft connection fusion method is used to strengthen the feature fusion of thermal infrared and visible images.Experimental results show that the network achieves lightweight joint semantic segmentation while ensuring accuracy.Fast Mix Seg can achieve an inference speed of 97 frames on the Ge Force RTX 2070 Super GPU platform.Finally,a dual-vision system for thermal infrared and visible images is built.To address the registration problem of thermal infrared and visible images,the direct linear transformation(DLT)is used.In the end,RFSSNet and Fast Mix Seg are deployed on the NVIDIA Jetson AGX Xavier.A joint semantic segmentation algorithm system for thermal infrared and visible images is achieved.The experimental results show that both RFSSNet and Fast Mix Seg have certain scene transferability.