Multispectral Registration And Fusion Of RGB And NIR Images Using Deep Learning

In recent years,low level image processing including image registration,denoising,fusion,and enhancement plays an essential role in high level vision systems such as video surveillance and autonomous cars as preprocessing.They have attracted much attention by more and more researchers as hot research topics.The fundamental goal of image registration is to find the spatial transformation relationship between two or among more images of the same scene and perform transformations based on the relationship.This is the operation of image stitching,fusion and reconstruction.The theoretical basis of multispectral images is that objects in natural scenes have different characteristics of absorption,reflection and radiation with different wavelength bands.In video surveillance,registration of images obtained by different sensors is required.The images obtained by different sensors have different characteristics,resulting in the complexity of image registration.Image fusion is able to integrate advantages of multiple sensors,thus generating fusion results that are suitable for human visual perception and subsequent vision tasks.It can overcome the shortcomings of a single sensor by improving visual quality of the resulting image,and gathering its own advantage of each sensor.Thus,it results in accurate,reliable,and informative images.Due to the insufficient light at night,the captured RGB images by visible sensors contain much noise and severe loss of textures.However,near-infrared(NIR)sensors generate clean textures even in low light condition without color.Thus,RGB and NIR images are complementary,which can be effectively fused.In this thesis,we investigate multispectral registration and fusion of RGB and NIR images based on deep learning.To be specific,we propose network architectures for multispectral registration and fusion based on convolutional neural networks(CNNs).We also provide training data generation for them to deal with insufficient or no training data problem.Main contents of this thesis are as follows:1.A multispectral image registration method based on CNN is proposed.Most existing image registration methods are based on key point detection and matching.However,the traditional registration method is mainly to register images generated by similar sensors,and the deformation model is mainly linear transformation.Although the registration results by the proposed method contain fuzzy details,the proposed method does not need a deformation model as priori information.Therefore,we propose a multispectral registration method based on CNN to make full use of multispectral advantage.2.A multispectral image fusion method of RGB and NIR images based on CNN is proposed.Due to the insufficient training data and inaccurate registration,the fusion of RGB and NIR images is required.In this thesis,we propose an end-to-end fusion network for RGB and NIR images through normalized gradient graphs.We present a new strategy of training data generation for multispectral fusion,which uses normalized gradient maps of NIR images as input,instead of feeding NIR images directly into the network.We adopt a gradient guidance strategy for multispectral registration because gradient information is robust to multispectral paired images.We generate normalized gradient maps from clean RGB images(i.e.,ground truth),and then synthesize inputs from clean RGB images(noisy RGB images and normalized gradient maps)for network training.The training data generation successfully deals with the problem of insufficient training data and transfers the NIR texture information to the fusion result.The proposed fusion network consists of three sub-networks: denoising,fusion,and enhancement.We use residual dense blocks for the denoising subnetwork,multi-scale guidance structure for the fusion subnetwork,and multidepth selection structure for the enhancement subnetwork.