Research On Non-local Attention Fusion Method For Infrared And Visible Images

Infrared sensor can effectively detect hidden or camouflaged thermal targets and has strong anti-interference ability by perceiving thermal radiation.However,the obtained image has some defects,such as blurred background and missing details.Visible sensor can obtain image with rich scene information,clear texture and high spatial resolution through receiving reflected light,but it is easily affected by variable environment or weather.Due to the strong complementarity of imaging mechanism and characteristic,the image fusion technology aims to synthesize the thermal radiation characteristic of infrared image and the texture information of visible image into a single image.The fusion image retains more prominent target perception and richer scene information,so as to facilitate the human eye observation and subsequent visual processing,which is widely used in remote sensing detection,medical diagnosis,intelligent driving,safety monitoring and other fields.At present,traditional fusion methods rely on prior knowledge to construct transformation models extracting image features,but their designed models are increasingly complex and usually have poor generalization ability.However,the deep learning fusion methods can effectively extract complex and variable features of images by convolution operation.they overcome the shortcomings of traditional inherent mathematical models and have stronger feature extraction capabilities.Even so,the limitations of network depth and convolution kernel size easily produce the lack of deep feature extraction ability,which fail to retain as much multi-scale information as possible.In addition,their methods rely on convolution operation to extract the features of the local neighborhood without considering their long-range dependency,which necessarily losts some essential global context and limits the fusion performance.Towards above issues,infrared and visible image are taken as subject investigated,and deep learning as well as attention mechanism is utilized as the technical means to study non-local attention fusion method for infrared and visible image.To address the problem that convolution operation only extracts local features and neglects their long-range dependency,infrared and visible image fusion based on dense Res2 net and double non-local attention models is proposed.Firstly,dense aggregation block constructed extracts multiscale feature maps with multiple available receptive fields and improves feature coding capability without downsampling operations or variable kernel sizes.Secondly,the fusion strategy with dual non-local attention models is put forward to establish the long-range dependency among multi-level features from the channel and spatial positions.The strategy can obtain essential global context and refine feature maps more focusing on the typical targets and details of source images.The effectiveness and robustness of this method are verified through lots of experiments and our method obtains excellent fusion performance from the subjective and objective aspects.To solve the problem of hand-craft fusion strategy and parameters,infrared and visible image fusion using non-local attention-based generative adversarial networks is proposed.Firstly,in generator,convolution aggregation block is introduced to extract deep features.In addition,infrared and visible discriminators are constructed to establish an adversarial game with generator,which can preserve source image features in a balanced manner.Secondly,A learning nonlocal attention module is designed,which can establish the long-range dependency among different scale and channel features and further enhance the feature coding ability.The method completes the end-to-end fusion task for infrared and visible image.Experimental results of TNO and Roadscene datasets show that the method is superior to other typical methods in subjective and objective aspects.For the practical application of infrared and visible image fusion,an embedded real-time and color fusion platform is developed.In the platform,Linux Ubuntu18.04.3,arm-himix200-linux and HP-KMS4300 intelligent dual-light camera are utilized as the operating system,the cross-compilation tool and the hardware architecture respectively.The fusion algorithm is proposed for Hi3516DV300 processor and it researches infrared and visible video image registration,real-time color fusion scheme and picture-in-picture display technology in the fixed field of view.The algorithm realizes the real-time color fusion in different patterns,which can effectively improve the image details and resolution quality,and enhance the actual detection ability of dual-light imaging equipment.