A Study On Multimodal Multispectral Image Registration And Fusion Using Adaptive Weighted Total Variation

In recent years,with the development of the sensor technology,multimodal and multispec-tral registration and multispectral fusion has become a very active research area.Multimodal and multispectral images usually suffer from an unregistered problem in the multispectral imaging system,such as flash and no-flash images,RGB and near-infrared(NER)image pairs,depth and color images.In eomputer vision,quite a few prior multispectral fusion methods assume that the input image is well aligned,which makes multimodal or multi-spectral images unsuitable for applications to the fusion image.Existing image registration methods can be divided into intensity-based,feature-based and combining intensity-based and feature-based.Feature-based methods rely on extracting features of distinct structures from images,such as points,lines and regions.The intensity-based method compares the intensity patterns in the image pair by a correlation metric.An important part of image regis-tration is the calculation of similarity.Some ways to calculate similarity have been proposed,including mutual information(Ml).Multispectral image fusion is using fusion technology to get a high-quality fusion image via multispectral images is obtained from different sensors in the same scene.Multispectral fusion provides a viable solution to generating high quali-ty photographs in various applications such as remote sensing,modern military,and medical image diagnoisis.The multispectai fusion of RGB and near infrared(NIR)images is actively used to enhance images for video surveillance including object detection and object recog-nition.Image fusion is a difficult problem to handle.The first problem is how to preserve accurate information from multispectral images.Many methods have been proposed which includes principal component analysis,wavelet,intensity hue saturation,and their combi-nations.Based on the previous research,this thesis mainly focuses on the multi-mode and multi-spectral registration and fusion.The main contributions of this thesis are as follows:1.We propose multimodal and multispectral image registration using a dynamic fusion index.The dynamic fusion index represents the fusion degree of two pair images,and we adopt it for image registration.Moreover,adaptive weights represent confidence of pair images structural features of pair images,and we utilize adaptive weights to select reliable components of pair images for image registration.Based on adaptive weights,we present adaptive weighted total variation to calculate the similarity between misaligned image and reference image.In the adaptive weighted total variation framework,we first get the dy-namic fusion index by fixing transformation parameters,and then estimate transformation parameters by fixing the dynamic fusion index.Finally,we determine the transformation parameters for image registration by iterating up to the convergence,and align the target im-age based on them.Experimental results demonstrate that the proposed method outperforms state-of-the-art registration ones in terms of visual quality and quantitative measurements.2.We propose multispectral fusion of color(RGB)and near infrared(NIR)images using adaptive weighted total variation.The RGB image has advantage of color information with severe corruption by noise,while the NIR image has advantage of clear textures without color information even in low light condition.We adopt adaptive weighted total variation to take the multispectral advantages in fusion.We remove outliers caused by the gradient mismatch between RGB and NIR images to make fusion natural without color shift.Since the NIR imaging depends on the NIR light strength,the NIR values are not reliable out of the range that NIR light can reach.Thus,based on a sigmoid function,we selectively perform total variation regularization according to the NIR values.If they are reliable,we perform the multispectral fusion with the guidance of the NIR image.If the NIR values are not reli-able,i.e.none or too low,we only perform total variation regularization for denoising on the RGB image without NIR gradient transfer.For optimization,we convert the nonconvex to-tal variation regularization into a linear system using iterative reweighted least squares.The proposed method successfully transfer the NIR details into the fusion result while removing noise and keeping colors as well as it produces natural looking fusion results even with un-reliable NIR values.Experimental results demonstrate that the proposed method is superior to state-of-the-art fusion methods in terms of visual quality and quantitative measurements.