A Study On Light Field Depth Estimation And Super-Resolution Using Convolutional Neural Network

Light field cameras,such as lytro camera,is renowned for its ability to refocus after taking a picture.It obtains spatial and angular information at the same time in one photographic exposure,capturing more information about where the light is traveling.With rich infor-mation the light field images provide,it shows improved performance on refocusing,depth estimation,saliency detection,material recognition and the recovery of transparent objects in microscopy.However,there exists some challenging problems in light field depth esti-mation and the light field image itself.First,Although light field depth estimation is easier and more accurate than a binocular stereo depth estimation,it still suffers from occlusions and noise.Second,it has a limited number of training data if we adopt convolutional neural networks to model nonlinear mapping function between light field data and corresponding depth.Third,for light field image itself,because of limited resolution in light field camera sensor,light field images resolution is restricted by the size camera sensor and microlens array.The light field view images exhibit a significantly lower resolution than images from traditional cameras,which restricted the application of light field images.In this thesis,based on the main purpose of light field depth estimation and light field super-resolution,We first used SVM classification and segmentation guided bilateral filtering to solve the occlusion problems.Then,we uses the result of the segmentation guided bilater-al filtering result to generate ground truth for our training data,then uses the convolution neural networks to do light field depth estimation.Finally,we adopt disparity compensat-ed prediction to get super-resoltion light field images.The main work of this thesis is as follows:1.We propose occlusion robust light field depth estimation using segmentation guided bi-lateral filtering.First,we calculate refocused light field images those were virtually focused differently than acquisition camera from light field data using digital refocusing.Second,we perform support vector machine(SVM)classification to classify occluded pixels and unoc-cluded pixels.Third,we conduct depth-cost estimation differently according to occlusions and remove noise in depth by cost volume filtering.Finally,we perform segmentation-guided bilateral filtering to refine the depth map while preserving edges.Experimental re-sults on both synthetic and real data sets demonstrate that the proposed method can achieve better results in occlusion robust and edge preserving.2.To overcome the lack of training data sets,we adopt pixel-wise light field depth esti-mation.Moreover,to learn a good depth representation from the limited light field data,we generate a training data set using the given light field data and ground truth.We propose light field depth estimation based on convolutional neural networks(CNN).First,we calcu-late refocused images from light field data using refocusing and combine them into a multi-dimensional image.Second,our method directly learns an end-to-end mapping between multi-dimensional image and depth map.The mapping is represented as a convolutional neural network.Third,we refine the depth map in Markov random field(MRF)framework using the depth and the confidence of the CNN output.Finally,to eliminate low-amplitude structures in depth caused by MRF,we smooth the depth map based on L0 gradient mini-mization.Experimental results show that the proposed method outperforms state-of-the-art light field depth estimation based on refocus way and have comparable results in detail pre-serving compare with state-of-art methods based on epipolar plane image(EPI).3.We propose light field image super-resolution based on disparity compensated predic-tion.We adopt deep convolutional neural networks perform that learns an end-to-end map-ping between low resolution light field images and high resolution virtual view.We use a warping model based on a correlation layer that captures multi-image correlation for virtual view synthesis.First,we predict vertical and horizonal disparity using a disparity prediction network.Then,we warp the low resolution light field images based on the predicted dispar-ity to synthesize the virtual view.Finally,we employ the efficient sub-pixel convolutional neural network(EPSCN)to achieve super-resolution reconstruction for the virtual view.Ex-perimental results demonstrate that the proposed method produces virtual views with high accuracy and close to the ground truth as well as achieves state-of-the-art performance in terms of PSNR and SSIM.