A Study On NIR Colorization And Long Range Imaging Based On Convolutional Neural Networks

With the development of the sensor technology,the fusion of multi sensor data has received much attention by researchers.It is certain that multi sensor data has significant advantages over single sensor data.Visible color(RGB)images have rich visual features such as colors and textures,which have been commonly used in many vision and surveillance applications.However,color images are easily interfered by weather conditions such as clouds and fogs.According to the Rayleigh scattering model,the intensity of light scattered by particles(clouds and fogs)in the atmosphere is inversely proportional to the wavelength of the incident light,resulting in blurry or lost details.Compared to visible light,many objects are more sensitive to near-infrared(NIR)light and thus the intensity of reflected NIR light is higher.Therefore,NIR cameras can capture distant objects clearly.The sensitivity of NIR to atmospheric scattering is low,and infrared light can directly reach the image sensor,thus producing nearly fog-free NIR images without colors.Since RGB and NIR images are complementary,their fusion provides a solution to high quality color imaging.Although NIR images are of grayscale,they contain rich textures and details even in harsh conditions or at a long distance.To obtain a color image with clear details and rich textures,I first attempt to colorize grayscale NIR images.Then,I try to achieve long range imaging by fusing RGB and NIR images.Convolutional neural networks(CNNs)have a good ability of learning features by a deep representation,and thus I adopt them for NIR colorization and long range imaging.The main scope of this thesis is as follows:1.I propose deep NIR image colorization with semantic segmentation and transfer learning.I use a convolutional layer to establish a relationship between a single NIR image and a three-channel color image rather than mapping to Lab or YCb Cr color space.I use semantic segmentation as global prior information to remove needless noise in smooth regions caused by error prediction.I use a color divergence loss to further optimize NIR colorization results with good structures and edges.Since the training dataset is not enough to capture rich color information,I adopt transfer learning to obtain color and semantic information.2.I propose long range imaging using multispectral fusion of RGB and NIR images,called LRINet.I adopt unsupervised learning for the fusion to solve the absence of ground truth.LRINet is an end-to-end network based on CNNs that consists of three steps for the multispectral fusion: feature extraction,fusion and reconstruction.To treat the discrepancy between RGB and NIR images,I construct Warping Net to warp NIR features,and add it into the feature extraction.Since dark channel prior(DCP)provides distance from the camera by light transmission degree,I combine it with structure loss as the weight for fusion.To ensure the color fidelity,LRINet operates the fusion on the input of the RGB luminance channel and NIR image.