Research On Deep Learning Methods For Image Processing Based On Synthetic Datasets

Deep learning is a branch of machine learning which uses multilayer neural networks with nonlinear transformations to extract the high-level features of the data.In recent years,with the developments of image datasets and the improvements of computing power,deep learning has been widely applied in various image processing tasks,such as image defog,target detection and image super-resolution.Although deep learning has made great progress in its methods and applications,lacking of training datasets is still a factor hindering its further development: 1.The data collection is quite expensive and time-consuming,it may violate the privacy.2.The cost of labeling the data is also high and there is no guarantee that the labels are right.Furthermore,sometimes human can't even labeling the data.In view of the above problems,this paper explores the idea that whether the deep networks trained on synthetic datasets without any real images can do image processing tasks well.The advantages of this idea are: 1.The training images are generated directly by computers fast and cheap.2.The data labels are generated together with the images by unique algorithm,they are consistency and accuracy.3.It is convenient to transform the synthetic training images for reinforcing some specific functions of the networks.The potential problems of this idea are:Firstly,the imaging model of the synthetic dataset is relatively simple compared with a real camera,and whether the real imaging process from 3-D analog objects to 2-D digital images can be expressed correctly.Secondly,the contents of the synthetic images are relatively simple compared with that of the natural images,and whether the simple contents are enough to simulate the complex contents in the real images.Therefore,it is still uncertain that the network trained on the synthetic dataset can process the real images.In order to explore the feasibility of this idea,we propose four synthetic dataset generation methods and the corresponding networks in the chosen checkerboard corner detection and image super-resolution tasks.After being trained with our synthetic datasets,our networks have achieved satisfactory results.These evidences strongly prove that it is feasible to train neural networks completely on synthetic datasets: Even if relatively simple camera imaging models are utilized to generate synthetic images,the neural networks can eliminate the influences of camera poses,blur and noise well,and then detect very accurate sub-pixel corner coordinates in real images.Even with very simple synthetic images' contents,the network can also learn the key image analysis techniques and successfully super-resoluton the real images.The specific works of this paper are as follows:1.In the field of checkerboard integer corner detection,the accuracy of the manually annotated corner position is not high enough.We plan to train our network with our synthetic dataset whose ground-truth corner positions are known for higher corner detection accuracy on real datasets.Therefore,we propose a synthetic checkerboard image generation method and a simple but effective checkerboard integer corner detection network trained with the synthetic dataset.Our synthetic dataset with ground-truth corner positions can be utilized to directly evaluate different detectors.Experimental results using both synthetic and real data show that the proposed detector significantly outperforms typical methods,including the commonly used Matlab camera calibration toolbox,the Open CV checkerboard corner detectors,and the more recently proposed deep learning-based methods.The work of checkerboard integer corner detection corresponds to Chapter 3.2.In the task of checkerboard sub-pixel corner detection,human cannot manually annotate the sub-pixel corner position in the real checkerboard images.We hope to provide a training dataset for supervised deep learning-based methods and further improve the accuracy of checkerboard corner detection by generating synthetic checkerboard images with ground-truth sub-pixel corner coordinates.We then improve the generation method of the synthetic dataset and propose two new checkerboard sub-pixel corner detectors.Experiment results show that the first detector is more accurate than the latest method based on deep learning in the real testing dataset and the second detector has better performance over the state of the art classical methods and the learning-based method on both synthetic and real testing datasets.The work of checkerboard sub-pixel corner detection corresponds to Chapter 4.3.In the simple image(text and engineering images)super-resolution task,considering that it is quite convenient for us to control the variables in the synthetic training dataset,we plan to generate perfect shapes and sharp edges that do not exist in real images.With thus generated training images,we can strengthen the network's power of keeping the boundary lines continuous and sharpening the edges.We therefore generate a synthetic dataset with only one triangle or ellipse in each image,the dataset simulates the variation of the CCDs' density which is hardly realized in practical applications to obtain both LR and HR images.Besides,we propose a network and train it on our synthetic dataset.In comparison with other state of the arts in the simple image superresolution experiments,the edges in our SR images are the sharpest when the resolution increases and the lines distort the least.The visual quality of our method surpass other methods a lot.The work of simple image super-resolution corresponds to Chapter 5.4.To study whether the synthetic dataset generated with simple shapes of triangles or ellipses can simulate the various textures in the natural images and if the neural network which is trained with the synthetic dataset can work well in the natural image super-resolution task,we propose a synthetic image generation method which can imaging the simple shapes in 3-D space and a network for natural image super-resolution.After being trained with our synthetic dataset,our network generates fewer artifacts in the flat areas,makes the regular structures with a certain size more clear and deblurs the very fuzzy areas during the super-resolution process.After further improvement,comparing with the state of the arts(including the winner of both real-world SR tracks in NTIRE2020),our natural image super-resolution method has the best perceptual quality in the metric of LPIPS,its visual quality has reached the top level.The work of natural image super-resolution corresponds to Chapter 6.