Pixel-level Image Fusion Based On Convolutional Neural Networks

Image fusion is an algorithm that extracts and combines the effective information of multiple source images into one fused image,which facilitates the computer processing and human observation.With the development of related technologies and the mass production of images,the fields in which image fusion can be applied are also wide,such as multi-modal fusion of medical images,multi-focus image fusion in photography,and panchromatic and multi-spectral image fusion in remote sensing,etc.Meanwhile,convolutional neural networks(CNN),as the main method in the field of deep learning,solve or improve multiple problems in the image field and obtains good results,such as image classification,segmentation,detection,etc.Nowadays,some image fusion algorithms based on deep learning have obtained good results.They are superior to traditional methods in some indicators,but compared with other fusion algorithms,the lack of the details,edges and texture information in the fused image need to be improved.Therefore,a novel multi-focus image fusion algorithm based on the convolutional neural network(CNN)in the discrete wavelet transform(DWT)domain is proposed.The algorithm combines the advantages of spatial domain-and transform domain methods.DWT is used to decompose the image into low frequency subbands(low frequency images)and high frequency subbands(high frequency images)in different directions.Decision maps take full advantage of the characteristics of frequency subbands.The low frequency subband is used to detect focused areas,which improves the accuracy of the decision map.To preserve more details,the high-frequency subbands are used to select image details,such as edges and textures.In addition,the CNN,which can be seen as an adaptive fusion rule,replaces the traditional fusion rules.The proposed algorithm includes the following steps: first,we decompose each source image into one low frequency subband and several high frequency subbands using the DWT;second,these frequency subbands are used asinput to the CNN to generate weight maps.To obtain a more accurate decision map,it is refined by a series of post processing operations,including the sum-modified-Laplacian(SML)and guided filter(GF).According to their decision maps,the frequency subbands are fused;finally,the fused image can be obtained using the inverse DWT.Subjective and objective evaluation on the experimental results verify the effectiveness of the algorithm.The former refers to evaluating the quality of the fused image through the observation and judgment of the human eye;the latter refers that indicators are used to measure the quality of the fused image.To illustrate the stability of the algorithm,this paper tested 20 groups of multi-focus images,and calculated the mean and standard deviation of the indicator scores.The two evaluations show that our algorithm is superior to other nine comparison algorithms.In addition,the algorithm's strategy for processing three or more source images,CNN used in different transform domains,and the algorithm's runtime are also discussed and analyzed in this paper.