Deep Learning Models For 2D Image Reflection Removal And 3D Point Cloud Analysis

2D and 3D images provide a way of perceiving the world.Nowadays,the demands of high quality digital images are still not satisfied with the development of technology.One often gets low quality images due to occlusion,reflection and low light.Those images are unfriendly to either human visual system(HVS)or some downstream applications such as image classification,image segmentation and object detection.Therefore,it is necessary and meaningful to restore the low quality images and graphics.Similarly,3D images may also have bad quality due to limitation of viewing angles.Sometimes,it is needed to reconstruct 3D shapes from 2D images,thereby facilitating scene analysis.Hence,high quality 2D images are necessary.In short,2D and 3D image analysis are independent and unified.In recent years,deep learning have brought excessive improvement in computer vision and a large number of outstanding works.In this thesis,2D image reflection removal,3D point cloud completion and how to reconstruct 3D point clouds from 2D image sequences are discussed.For 2D image reflection removal,three scenarios are listed as follows:(1)Remove reflections on a single image using deep neural network:when one takes a photo through glass,the image is a linear combination of two layers:transmission layer and reflection layer.In most conditions,the characteristics of reflection layers and transmission layers are similar for the reason they are natural images.Moreover reflections are dominant in some regions,thus the transmitted scene is totally degraded,which make it more difficult to remove reflections.However there is difference between reflection and transmission:when the light of reflected scene arrives in plane of glass,it usually undergoes a series of reflections and refractions.Thus the reflection layer often has ghosting artifacts.In this thesis,we synthesize training data based on observations and achieve reflection removal on a single image using deep neural networks.We introduce internal and external loss for training and experimental results show the outstanding performance of our model.(2)Joint reflection removal and depth estimation from a single image:Reflection caused by glass often degrades the quality of an image and further makes it difficult to estimate depth.Thus,we propose joint reflection removal and depth estimation from a single image.We perform reflection removal(transmission recovery)and depth estimation jointly using a collaborative neural network that consists of four blocks:encoder for feature extraction,reflection removal sub-network(RRN),depth estimation sub-network(DEN),and depth refinement guided by the transmission layer.We achieve collaboration between reflection removal and depth estimation by concatenating intermediate features of DEN with RRN.Since the recovered transmission layer contains accurate edges of objects behind glass,we refine the estimated depth with its guidance by guided image filtering.Experimental results demonstrate that the proposed method achieves both reflection removal and depth estimation even for images with dominant reflections.(3)Siamese dense network for reflection removal with flash and no-flash image pair:When objects are covered by glass,the no-flash image usually contains reflection,and thus flash is used to enhance transmission details.However,the flash image suffers from the specular high-light on the glass surface caused by flash.In this thesis,we propose a Siamese dense network(SDN)for reflection removal with flash and no-flash image pairs.SDN extracts shareable and complementary features via concatenated siamese dense blocks(SDBs).We utilize an image fusion block(IFB)for the SDN to fuse the intermediate output of two branches.Since severe information loss occurs in the specular highlight,we detect the specular highlight in the flash image based on gradient of the maximum chromaticity(MC).Through observations,flash causes various artifacts such as tone distortion and inhomogeneous brightness.Thus,with synthetic datasets we collect 758 pairs of real flash and no-flash image pairs(including their ground truth)by different cameras to gain generalization.Various experiments show that the proposed method successfully removes reflections using flash and no-flash image pairs and outperforms state-of-the-art ones in terms of visual quality and quantitative measurements.Deep leaning can also be directly applied to 3D point cloud analysis.In 2D image reflection removal we use encoder decoder(E-D)to recover a clean image,which inspire us to adopt E-D to point cloud completion:(4)FinerPCN:point completion network using global and local information:3D scanners often obtain partial point clouds due to occlusion and limitation of viewing angles.Point cloud completion aims at inferring the full shape of an object from an incomplete point set.Existing deep learning models either do not consider local information or easily deform the regions associated with the input.In this thesis,we propose a point completion network that generates complete and fine grained point clouds in a coarse-to-fine manner,called FinerPCN.FinerPCN consists of two subnetworks:auto encoder-decoder(AE)for generating a coarse shape and pointwise convolution for refining its local structure.By repeatedly feeding partial input into the second subnet,FinerPCN effectively considers local information and alleviates structural deformation of input while maintaining global shape.Experimental results show that FinerPCN generates finer grained completion results while successfully keeping the shape of the input than state-of-the-art methods.Moreover,we can reconstruct the 3D points from 2D images and utilize a Siamese structure to make diagnosis:(5)Cardiac MRI segmentation and pathology classification usingg deep neural networks and point clouds:Segmentation of cardiac MRI images plays a key role in clinical diagnosis.In the traditional diagnostic process,clinical experts manually segment left ventricle(LV),right ventricle(RV)and myocardium(Myo)to get a guideline for cardiopathy diagnosis.However,manual segmentation is time-consuming and labor-intensive.We propose automatic cardiac MRI segmentation and cardiopathy classification based on deep neural networks and point clouds.The cardiac MRI segmentation consists of two steps:(?)We use a detector based on you only look once(YOLO)to obtain region of interest(ROI)from the whole sequence of diastolic and systolic MRI.(?)We obtain a pixel-wise segmentation mask automatically by feeding ROI into a fully convolutional neural network(FCN).Based on the segmentation results,we reconstruct 3D surface by simple linear interpolation,and randomly and uniformly sample 3D point clouds from the 3D surface.From the cardiac point clouds,we perform cardiopathy classification based on a cardiopathy diagnosis network(CDN).Experimental results show that the proposed method successfully segments LV,RV,and Myo from cardiac MRI images...