A Study On Multi-spectral Image Fusion And Single Image Super Resolution Based On Convolutional Neural Networks

Due to the increasing demands on visual surveillance,people have higher and higher re-quirements on visual quality of images.Up to the present,the image processing technology has received much attention by reasearchers as preprocessing for visual surveillance appi-cations.As a media to convey information,images are informative and entertaining,thus playing a very important role in many aspects of human life.With the development of imaging systems,the quality of images obtained by people far exceeds the past.However,there still needs improvement in visual quality to meet people's expectations.Since im-ages are captured in various conditions such as weather,environment and devices,they have various defects and problems.For example,visible color images(VCI)taken in low light condition contain much noise with severe loss of textures.To overcome this problem near infrared(NIR)images can be fused with low light RGB images to remove noise and recover textures,thereby obtaining high-quality fusion results.Improving image resolution is also an issue that researchers have paid much attention to in recent years.Single image super-resolution(SISR)is to recover a high resolution(HR)image with better visual effects from a low resolution(LR)one.However,SISR is an ill-posed problem that a low-resolution im-age can be obtained from many different high-resolution images.Fortunately,deep learning has achieved a remarkable performance improvement in many computer vision and image processing tasks.In this thesis,we investigate multi-spectral image fusion and SISR based on deep learning,and propose network architectures using convolutional neural networks(CNNs).Main research scope of this thesis is as follows:1.We propose multispectral fusion of low light RGB and NIR images based on a coop-erative CNN.RGB images taken under low-light conditions have much noise and texture loss with color.while NIR,images taken in the same scene contain clear textures robust to noise without color.They are complementary,and we propose a fusion network of RGB and NIR images through a cooperative CNN.To generate training data in low light condition,we use clean RGB images captured in daylight as ground truth and synthesize low light RGB images from them by adding noise and smoothing.We build a cooperative network to fuse RGB and NIR images,and generate clean RGB images without noise.To fuse them without color distortion we choose YCbCr color space.Through this network,we get a fused image that eliminates noise and maintains the original color with good textures.2.We propose edge guided SISR using wavelet-based CNNs.SISR is a challenging task and deep learning is able to improve the performance with the help of big data.Unlike the previous SISR network,we adopt edge maps as a priori information for low-resolution im-ages to guide the super-resolution process,and we add a wavelet transform to the network.We simultaneously restore SR image and its edge map,and use the edge map to guide SR reconstruction process.In practice,low frequency components of LR and HR images are very similar.The main difference between them is the high frequency components,and the key to SISR is the recovery of the high frequency components.Therefore,we intro-duce wavelet transform into CNNs to successfully recover the high frequency components.The proposed SISR network consists of three parts:Wavelet-SRNet,Wavelet-EdgeNet,and Fusion-SRNet.The experimental results show that the proposed method is robust to alias artifacts and successfully recovers high frequency components in SR results.