Research On Information Fusion Algorithms Of Multimodal Images

Thermal infrared image,also known as long-wave infrared image,is imaged by receiving the thermal radiation energy emitted by the target object,and the objects are distinguished by the difference of thermal radiations.The acquisition of thermal infrared image does not depend on the external light,and has the characteristics of allweather.It has excellent detection performance on handing the occluded objects in the weak light environment.Due to the random interference of external environment,such as the thermal balance of the scene,the thermal image has strong spatial correlation,low contrast and blurred visual effect.These shortcomings make target recognition and anomaly detection difficult to be applied independently in thermal images.In contrast,visible image can better satisfy the intuitive understanding of human vision.In the same scene,visible image has higher spatial resolution and contrast,and the texture details of the image are clearer.Combining the advantages of the two sources,new images with clear targets and high resolution can be generated to meet the requirements of allweather conditions.To solve the above problems,this paper uses adversarial generation network in deep learning,and proposes a novel double-flow neural network,which integrates complementary information from thermal infrared and visible images.The main work is as follows:1)A thermal to visible image style transfer algorithm based on progressive GAN network is designed.We propose a style transfer method which applies progressive generation strategy,aiming to convert low resolution thermal face image into high resolution visible face image in the same identity.We use two kinds of image data,both paired and unpaired,to improve the generalization performance of the model and avoid the generation of normalization.Perceived loss is used to constrain the generated image to preserve the details of the source image at the feature level.Experimental results show that the incremental generation strategy is beneficial to the stability of training and the performance of generation is improved 15%,when compared with other methods.2)A fusion algorithm of thermal and visible image based on double-flow neural network is proposed.Due to the lack of a large number of fusion results as reference data,the fusion process lacks constraints and guidance.The usual method is to use only visible images for training,use the same feature extraction operation for different source images in the test stage,or only focus on the fusion of some salient information of the images,and ignore the differences and connections between dual-light images In order to solve these problems,the dual-stream network proposed in this paper provides independent channel for visible image and thermal infrared image,so that the model can learn two types of feature simultaneously in the training stage,and guide the generation of fusion image by combining the respective features of the two kinds of images.In this way,an end-to-end model consistent with the network structure in the training stage and the test stage is formed while both the dual-light images participate in the training stage.The algorithm calculates the multi-level structural similarity and other indicators on the TNO data set,which is 1.39% higher than the similar algorithm,and achieves the best accuracy on the data set.The experimental results show that the proposed algorithm can retain the respective features of the thermal infrared image and the visible image,suppress the generation of artifacts,and improve the fusion accuracy.