Study On Multi-exposure Image Fusion By Convolutional Neural Network

With the rapid development of digital imaging technology,the demand for digital image quality is increasing day by day,to achieve a more realistic and immersive visual experience.As a technical means to make up the dynamic range gap between the real scene and the image acquisition equipment,multi-exposure fusion(MEF)can improve the quality of the fused image and simulate the real scene more realistically and vividly.It has been widely concerned by scholars and related technology manufacturers in the domestic and overseas.Based on the convolutional neural network(Convolutional Neural Network,CNN),this thesis focuses on the lack of labeling information,the applicability of flexible exposure shots images,and the details preservation of fused image involved in static scene fusion,as well as the inconsistency of spatial information caused by camera shake and foreground movement in dynamic scenes.This thesis studies how to use the convolutional neural network to realize unsupervised learning and training strategies that do not rely on labeled information in static scenes,and deeply explores how to eliminate the ghosts of MEF problems in a dynamic scene.The main work and innovation involved in this thesis can be included in the following aspects:1.Aiming at the lack of labeling of the dataset and the detailed information for fusion images involved in static scene exposure fusion,an unsupervised CNN model based on a non-reference loss function is proposed.By analyzing the coupling relationship between the image quality evaluation index and the loss function,a loss function is constructed,which is guided by quantitative index and does not need any reference information or labeling information.So,the network can be updated and learned in an unsupervised manner,and directly merge the color image using chrominance information,without the need for chromaticity decomposition and multiple spatial conversion.And the fusion results can be viewed directly,which avoids the intermediate process of tone mapping.In addition,through the trapezoidal fuzzy membership function,a gradient fidelity loss term that can recover high-frequency information in the image is constructed.It makes the fused image contain rich details.Each module of the fusion model has serval alternative schemes,which can be replaced and changed according to the fusion scene and data characteristics to achieve better merging results.2.Aiming at the uncertainty of exposure shots and the requirement of fused image detail enhancement in static scene exposure fusion,an unsupervised multi-exposure CNN model based on exposure guidance and a detail enhancement model are proposed.First,an exposed-guided feature merging method is constructed to enhance the flexibility of the network structure to the number of exposure shots and prevent feature degradation.Secondly,a set of pseudo-siamese encoders are used to obtain the exposure information,and the attention mechanism is used to supplement and suppress the exposure information.Then,to make the fusion results more refined,a multi-scale dilation convolutional module is constructed,and a reconstruction network with multi-scale information and global residual is built,which makes the fusion results finer and richer.Finally,through the gradient information and the structure tensor,two different exposed-guided loss functions are constructed to enhance the image details,which makes the fusion results more refined and vivid.3.Aiming at the phenomenon of ghosts in dynamic scene exposure fusion,dominated by the idea of optical flow,a dynamic scene multi-exposure CNN fusion model based on feature image optical flow registration is proposed,which integrates registration and fusion.This model transforms the image optical flow registration into feature optical flow registration,which avoids the computational and structural redundancy caused by multiple coding and decoding structures in the existing networks.Based on the classical optical flow method,a multi-scale feature pyramid is constructed,and the features are registered layer by layer,and the motion information of different scales is gradually eliminated from coarse to fine.To delicately describe the spatial registration relationship,a multi-scale coarse-grained optical flow estimation is constructed,and a fine-grained optical flow correction network constructed by deformable cost volume and functional convolutional modules is constructed.Besides,to reduce the network shock,based on the classical HDR loss,a feature loss function is constructed to restrict the exposure consistency and content consistency of feature maps.4.To solve the problem of geometric deformation due to camera shake or foreground motion in a dynamic scene,a dynamic scene multi-exposure CNN fusion model based on a feature deformable alignment network is proposed to generate ghost-free fusion images.By analyzing the limit of the application of CNN in complex dynamic scenes,deformable convolutional is introduced to increase its geometric deformation ability,and at the same time,the spatial information registration of features is realized.In the model,an adaptive scale convolutional module is constructed,so that the feature fusion can be reduced to a variety of different convolutional modules through parameters learning,and a learning method of network structure is simple realized.Besides,to accelerate the convergence of the network,the spatial registration features are constrained from the aspects of feature intensity and structural similarity.