Removing Rain From Single Images Via Domain Knowledge Driven Deep Learning

Due to the effect of refraction and reflection of rain streaks,objects in rainy images suffer from various degradations,such as ambiguity and deformation.These degrada-tions affects the enjoyment of human eye and the performance of subsequent computer vision systems.Therefore,the research on image de-raining has extremely important theoretical and practical values for image processing and computer vision.At present,the methods of image restoration can be roughly categorized into two types:model-driven and data-driven.The model-driven approach primarily uses the relevant domain knowledge to manually design the algorithm,such as methods based on shallow visual representation models,e.g.,image filtering and sparse representation.Since rain streaks have obvious geometric features,it is easy to be confused with the objects' edges and textures.This makes the handling of rain streaks removal more difficult.On the other hand,the limited model capacity results in unsatisfactory de-raining performance.The data-driven approach is represented by deep learning,which allow the model to learn the mapping relation from the data itself.The performance of such methods depends on the training data used for network training.Since it is difficult to collect rainy-clean image pairs in the same scene from the real-world,most methods use synthetic data to train networks.However,due to the imaging model under rainy conditions are extremely complex,the synthetic method cannot completely simulate real scenes.In other words,there is a certain difference between the synthetic data and the real-world data.Therefore,algorithms designed by synthetic data cannot be well generalized to the real-world data.These algorithms perform well on synthetic data,but not on real-world data.To address the drawbacks of existing single image de-raining methods,in this dis-sertation,a novel method is proposed to utilize domain knowledge of image processing to drive and guide deep learning.By fully taking advantages of both model-driven and data-driven methods,this work opens a new research direction.This work takes single image de-raining as a specific verification object and propose the first deep convolu-tional neural networks based algorithm to solve this problem.The main contents and innovations of this work are as follows:1.This dissertation proposes the DerainNet that only process image high-frequency parts for image de-raining.Through experiments,it is found that when the rainy image and the corresponding ground truth are decomposed into high-and low-frequency parts,the rain streaks is mainly concentrated in the high-frequency part,while the low frequency component basically does not contain rain streaks.Different with existing deep learning based methods,by using the domain knowledge of image processing,this work trains the network in high-frequency domain instead of image domain.The final result is obtained by adding the de-rained high-frequency image with the low-frequency part.On the one hand,the learnt mapping function is simplified due to the sparsity of image high-frequency components,which can boost de-raining performance.On the other hand,not only de-raining but also generalization are improved by the proposed DerainNet.This is because the high frequency components have similarities compared to images.In addition,the low-frequency components contain contrast information,which can be enhanced to further improve the visual quality of de-rained results.To the best of our knowledge,this work is the first to make a big breakthrough in the effect of single image rain removal.It is also the first method to use deep convolutional neural network for single image de-raining.Compared to model-driven algorithms at the same time,DerainNet's SSIM average increased by 8%on the public light rainy dataset and nearly 83%on the heavy one.2.This dissertation proposes the Deep Detail Network by integrating domain knowledge into deep networks for image de-raining.Based on the DerainNet,this dissertation further proposes an end-to-end Deep De-tail Network(DDNet)for image de-raining.The domain knowledge used by DDNet is similar to that of DerainNet.The difference is that DDNet directly integrates the domain knowledge into the deep network,which effectively overcomes the lack of De-rainNet to use the low-frequency information.DDNet' s characteristics are as follows:1)The actual input of the convolutional layer of the DDNet is the same as that of De-rainNet,which is the sparse high-frequency component that can simplify the learning problem.2)The de-rained result is obtained by adding the rainy image and the out-put of convolutional layers through the global skip connection.This connection makes the output of the convolution layer is actually a negative residual,which also contains sparsity and is the same as the output of DerainNet.In addition,DDNet's basic net-work module uses the classic ResNet model.The local skip connection structure of this model can bring two advantages.1)This structure can effectively eliminate the gradient vanishing and build a deeper model.Larger receptive fields and stronger non-linearity can be obtained to improve de-raining performance.2)This structure can propagate the information of the shallow feature map to the deep network without distortion.This corrects the cognitive bias that such tasks can only use shallow models(DerainNet only contains 3 convolutional layers,while DDNet contains 26 convolutional layers).Exper-iments demonstrate that the proposed DDNet can further improve both qualitative and quantitative results with no need to add extra computing resources and data.Compared to DerainNet,DDNet's SSIM average increased by nearly 14%on the public heavy rainy dataset.3.This dissertation proposes the Lightweight Pyramid Network for image de-raining.This dissertation expands the idea of DerainNet and DDNet to simplify the learn-ing process and proposes the Lightweight Pyramid Network(LPNet).By utilizing do-main knowledge to introduce the multi-scale pyramid decomposition algorithm,LPNet decomposes single tough de-raining problem into several easy sub-problems.Com-pared to DerainNet and DDNet with single-scale decomposition,the learning process of each sub-problem of LPNet is greatly simplified.This indicates that the correspond-ing sub-network can learn the mapping relationship with only a few parameters.This" divide and conquer" strategy can greatly simplify the network,which makes it possi-ble to design a lightweight model.Compared with state-of-the-art deep learning based methods,the proposed LPNet reduces the number of network parameters by nearly 98%while still achieves comparable performance on rain streaks removal.4.Extension of the proposed models.Although the main goal of this dissertation is to remove rain streaks from single images,the utilized domain knowledge is derived from the common features in im-ages,such as image high-frequency sparsity and multi-scale decomposition,thus the proposed models are not limited to image de-raining.The proposed models have been extended to image de-noising,super-resolution,JPEG compression artifact removal and Pan-sharpening.The experimental results show that the proposed method has cer-tain comparative advantages.For the pan-sharpening,the proposed model achieves the best performance on both subjective and objective evaluations.Moreover,the proposed model can be well generalized to real-world data.