A Study On Multi-Sensor Image Fusion And Its Applications To Depth Super-resolution And Multispectral Image Fusion

With the development of different kinds of image sensors,more and more image information is recorded in various data formats.In addition to traditional RGB color cameras,there are depth cameras that record depth information,as well as near infrared(NIR)cameras use non-visible bands for imaging.Due to the unique advantage of each sensor,they are fused in various applications.For example,self driving car uses depth and RGB cameras,while video surveillance systems use NIR and RGB cameras.Since one type of sensor usually has its own advantage,image fusion from multiple sensors is helpful for producing higher quality images and more favorable images for subsequent decision making.With the development of image processing and sensor technology,the fusion of multi-sensor images has become a research hotspot.In this thesis,we investigate multi-sensor image fusion to maximize their own advatages,including color-guided depth super-resolution and RGB-NIR image fusion.The main contents and contributions of this thesis are as follows:1.An alternately guided depth super-resolution algorithm by using weighted least squares(WLS)and zero-order reverse filtering is proposed.Due to the structural inconsistency between color and depth,texture copying and edge blurring artifacts appear in color-guided depth super-resolution.In this thesis,we propose alternately guided depth super-resolution using WLS and zero-order reverse filtering.We adopt WLS for alternating guidance,and alternately use color and depth as guidance in WLS to suppress texture copying artifacts.Since color guidance causes edge blurs in depth due to the mismatch between color and depth,we apply zero-order reverse filtering to depth images to alleviate edge blurring artifacts.Moreover,WLS is a global optimization-based filter and thus it is effective in removing depth noise.2.A scale-aware multispectral fusion algorithm of RGB and NIR images based on alternating guidance is proposed.In low light condition,color(RGB)images captured by camera contain much noise and loss of details and color.However,near infrared(NIR)images are robust to noise and have clear textures without color.In this thesis,we propose an alternating guidance based scale-aware fusion of RGB and NIR images with WLS.Low light RGB images provide big-scale image structure and color information,while NIR images offer clear textures.Since they are complementary,we adopt alternating guidance for fusion of RGB and NIR images based on WLS.First,we perform the first guidance for denoising on noisy RGB image to get base layer.Then,we conduct the second guidance for scale-aware detail transfer on NIR image to get detail layer.Finally,we combine base and detail layers to produce a fusion image.We maximize the advantage of multispectral input(RGB and NIR)for fusion based on alternating guidance.3.A multispectral fusion algorithm of RGB and NIR Images using WLS and convolution neural networks is proposed.In low light condition,acquiring clean RGB images is a difficult task due to the lack of light.The usual approach is to increase the camera's ISO for color image acquisition.However,RGB images obtained by increasing the camera ISO can contain much noise and loss of details.NIR images taken under the same conditions contain clear details with little noise.Therefore,the fusion of RGB and NIR images can improve the imaging quality in low light conditions.In this thesis,we propose multispectral fusion of RGB and NIR images to improve imaging quality in low light conditions by using WLS and convolution neural networks(CNNs).The fusion of RGB and NIR images is divided into three modules: RGB image denoising,image enhancement and NIR detail transferring.For RGB image denoising,the WLS filter uses RGB and NIR images together as weights to filter out noise in RGB images.In image enhancement,we built an image enhancement network to enhance the color of RGB image.In NIR detail transfer module,we build a detail transfer network based on CNN to transfer clear NIR details to the fused image.