Research And Hardware Implementation Of Infrared And Visible Image Fusion Algorithm Based On Deep Learning

Infrared and visible light image fusion is a hot topic in the field of image processing.By fusing and processing two types of images after registration,the advantages of both can be integrated to improve the ability of human or machine to perceive and understand scenes,and to more comprehensively identify and detect targets.Therefore,the application-level research on infrared and visible light image fusion system has important theoretical and practical significance.In this paper,we take infrared and visible light images as the research object,expound the basic theory and registration pre-processing of image fusion,and introduce and analyze several classical traditional fusion methods and deep learning-based methods.Among them,the infrared and visible light image fusion framework STDFusionNet defines for the first time the required fusion information as the salient targets in the infrared image and the background texture in the visible light image through the design of specific loss functions.However,it still has problems such as poor edge information extraction ability and insufficient description of fine-grained details.Therefore,this paper optimizes and improves the STDFusionNet by introducing a multiscale residual module to enhance information sharing ability and improve the network’s ability to extract detailed feature information.The Sobel operator is introduced into the structure of the original network residual block to extract fine-grained detail information while strengthening the network’s extraction and reconstruction of image edge texture features and effectively suppressing noise.This paper trains and tests the improved algorithm on the TNO dataset and RoadScene dataset.Compared with the STDFusionNet algorithm,the EN,MI,and VIF evaluation indicators are improved by about 1.5%,2.8%,and 2.3%,respectively.The results show that the fusion image generated by the improved algorithm contains more source image information and is more consistent with human visual observation.Based on the network model proposed in this paper,a hardware system for infrared and visible light image fusion based on FPGA is further designed.The hardware design mainly includes a serial communication module,a data storage module,a top-level control module,and a data processing module.Each module is written in Verilog HDL hardware description language,and then function simulation is performed in Quartus II and Modelsim software to verify the correctness of the system function.Finally,after synthesis,the board-level verification is performed on the DE2-115 development board,and the software and hardware output results are compared to verify the correctness of the hardware system implemented based on the algorithm proposed in this paper.The results show that the hardware system in this paper only takes 48ms to complete algorithm inference and the power consumption is only 4.6W.