3D Shape Understanding Based On Deep Learning

With recent developments in computer graphics,3D shape understanding has been a highly relevant and challenging research topic.The central problem of this thesis is how to build intelligent computer algorithms to analyze and generate 3D shapes.To effectively accomplish the above tasks,a key step is to construct 3D shape descriptors.However,traditional methods rely on intuition and knowledge of researchers.These approaches adapt to a single or small collection of 3D objects.In the information era,the amount of accessible 3D models has increased by several orders of magnitude.This significant growth pushes us to redefine 3D shape understanding algorithms from the perspective of data-driven manner.Deep learning has swept the field of computer vision during the past few years.By exploiting the availability of large image datasets(i.e.,ImageNet),deep learning algorithms has achieved better performance than traditional non-learning methods for 2D image understanding.Inspired by the success of deep learning for 2D image understanding,this thesis devotes to designing novel algorithms and frameworks for 3D shape understanding,by following the philosophy of deep learning.In general,3D shape understanding tasks can be classified into three basis categories: 3D shape description,3D shape analysis,3D shape generation.3D model analysis and generation are two key steps of 3D shape understanding,and 3D shape description is the foundation of 3D shape analysis and generation.And 3D shape analysis is composed of 3D shape correspondence ? 3D shape segmentation and 3D shape retrieval.This thesis focus on 3D shape understanding,exploring the description ? analysis and generation of 3D shape,by using deep learning technology.The main contributions of this thesis are as follows:1.For the problem of 3D shape description,we propose an algorithm to learn shape descriptors via deep triplet CNNs.During the process of perception and processing 3D shapes,a key step is to extract or learn effective and robustness shape descriptors.This thesis proposes a novel Triplet CNNs to learn local shape descriptors.It takes as input a large pool of multi-scale features.This thesis also designs a triplet loss function,which guarantees that descriptors of corresponding points are closer than ones of noncorresponding point in the learned deep feature space.The experiments show that the learned local shape descriptors have great discriminative power by using different evaluation metrics.2.For the problem of 3D shape analysis,we propose an algorithm to analyze3 D shapes via deep metric learning.This thesis introduces an EdgeNet,a deep metric learning architecture for learning 3D shape local features.In the deep feature space,this approach considers the feature similarity of point-wise correspondence between different shapes and the one of local adjacent points on the same shape together.Thus,the learned local feature is structure-aware.And EdgeNet directly consumes on 3D coordinates of point clouds.We also present several application algorithms of EdgeNet in different shape analysis tasks,including shape correspondence ?shape segmentation ?shape partial matching and within-domain shape retrieval.3.For the problem of 3D shape retrieval,we propose an algorithm to crossdomain retrieve 3D shapes by using Cycle CNNs.For the problem of 3D shape retrieval,this thesis designs a Cycle CNNs to cross-domain retrieve 3D shapes and 2D sketches.It directly learns the mappings between the feature space of 2D sketches and the one of3 D shapes,without common feature space construction.This method first learns representations of sketches and shapes by using different deep networks respectively.Then,we also design three different loss functions to compute feature similarities in different feature spaces.Finally,our Cycle CNNs can learn the mapping relations between these two feature space during the iterative process of these three terms.The learned mapping relationships can be used to cross-domain retrieving 3D shapes and 2D sketches.4.For the problem of 3D shape generation,we propose an algorithm to synthesize human performance via deep generative networks.This thesis proposes an deep generative architecture to synthesize human performance from sparse multi-view RGB capture.It takes as input sparse multi-view RGB footage,and outputs video-based performance of human model from arbitrary novel viewpoints.This architecture directly generate video-based performance from a target view without 3D reconstruction.It consists of two main steps: novel-view synthesis by using generative query network,detail enhancement by using generative adversarial network.The above method follows a coarse-to-fine strategy,which can balance the visual quality of synthesized results and the time complexity of training process.We show the effectiveness of this deep generative architecture on both synthetic and real world performance datasets by experiments.In summary,this thesis deeply studies the applications of deep learning method in 3D shape understanding,and designs a series of intelligent computer algorithms for 3D shape description ?analysis and generation.Firstly,for 3D shape description,this thesis proposes a method to learn shape descriptors by using triplet CNNs.Secondly,for 3D shape analysis,this thesis applies deep metric learning to different shape analysis tasks,including shape corresponding ? shape segmentation and within-domain shape retrieval.Thirdly,for cross-domain shape retrieval,this thesis proposes Cycle CNNs to cross-domain retrieve 2D sketch and 3D shape.Finally,for 3D shape generation,this thesis introduce a deep generative architecture to synthesize human performances.The results of a large number of experiments show the effectiveness of proposed algorithms.