| Polygonal meshes are the most popular representation of 3D objects by virtue of their high storage efficiency and powerful 3D complex shape characterization capabilities.However,these meshes are disordered and the surface is irregular,so it is difficult to directly apply the existing mature network framework to this data structure.Moreover,it is very difficult to learn high-level features on 3D models.The construction of each level depends on the preservation of spatial geometry,and the aggregation of local low-level features requires adaptive mesh density and variable mesh representation.Furthermore,the positional structure information in the 3D space is rich and complex,and the problem of the lack of moderate variability in traditional convolutional neural networks will be magnified.In order to solve the above problems,researchers have tried various methods based on multi-view,voxel,point cloud,graph or mesh itself,and achieved remarkable results.Nevertheless,there are still many difficulties that have not been properly resolved.This paper mainly explores the design and the implementation of deep learning framework on 3D irregular mesh data.The main contributions are summarized as follows:(1)A local convolution window representation in the form of continuous polynomial surfaces is proposed for the irregular 3D mesh structure.By local fitting to the surface of the original 3D mesh data,the polynomial representation can approximately describe the local shape information of the 3D data.Therefore,the irregular local mesh surface is represented as a uniform polynomial function with different coefficients.The convolution window form also solves the problem of different forms of mesh representation,whether for triangular mesh,quadrilateral mesh or mixed-represented mesh,surface fitting and representation can be performed.Based on this representation,we design a similarity measure between the original local surface polynomial and the convolution template surface polynomial,thus forming a local convolution operation.Similar to how 2D convolution kernels are learned,we adopt an unsupervised clustering learning method to learn the parameters of the convolution kernel polynomial function from the local surface set.After the above operations,local convolutional feature representations of 3D objects can be obtained.To complement the global features of the learning target,Markov chains are introduced to describe the spatial co-occurrence relationship between each local mesh surface.Finally,we verify the effectiveness of the proposed method on a classification task on a 3D non-rigid model.The implementation results show that the 3D mesh convolution model has excellent performance on the classification tasks of SHREC10 and SHREC15,and the convolution method can be used as a general feature extraction method to deal with 3D data signals on irregular domains.(2)A learning model based on the pyramid structure of 3D mesh models is proposed for the difficult high-level feature learning on 3D models.We take the local convolutional features as the underlying micro-features,and the model is gradually simplified as the level increases.The convolution result of the previous layer is stored in the corresponding position on the reduced mesh model of the next layer.A pyramid-like structure can be built step by step.We use the simplified algorithm of the mesh model to construct the hierarchy,which ensures that the geometric structure information of the entire model does not change while expanding the convolutional receptive field.In order to achieve semantic-level aggregation of local features,we adopt a mesh simplification algorithm based on vertex clustering,which simplifies local vertices with similar geometric properties and provides us with an index at the same time.The fusion of local information according to the index can avoid the interference of information with long geometric distance but short topological connection distance.Through this pyramid structure,we realize the learning from low-level features to high-level features on the 3D mesh model.Our ablation experiments on the number of network levels show that our proposed multilevel deep learning model has the ability to capture features at various levels from local microscopic to progressively more macroscopic.(3)To address the difficulty of long-distance feature association in convolutional methods,a deep learning network based on 3D mesh Transformers is proposed.We regard the local mesh surface as the basic token,which acts like a word in NLP and can encode the shape information of the local 3D mesh surface.The self-attention computation process is achieved by measuring the similarity between local surfaces,and based on this,a shape Transformer module can be formed.To build a deep feature learning structure,we propose a vector-based visual Transformer module.Among them,the visual token can summarize the underlying feature information in the shape Transformer.We construct a feature pyramid-like multi-level learning structure by stacking 3D visual Transformer modules and downsampling modules.To enable attention to support adaptive tuning of each feature channel and make it more expressive,we set the attention weights as a set of vectors.To verify the effectiveness of our proposed method,we conduct classification experiments on one 3D large dataset and two 3D deformable mesh model datasets.We also conduct part-based shape segmentation experiments on the human dataset.Experiments show that our proposed 3D mesh Transformer with explicit local shape context augmentation representation and multi-level feature learning structure achieves state-of-the-art performance in shape classification and part segmentation tasks.(4)A vector-type 3D mesh deep network is proposed to solve the problem that the convolution method can only learn the existence of features,but cannot learn the structural relationship between the features.The method is based on a 3D mesh shape Transformer to construct the associations between each local mesh surface.In order to learn the combination between the underlying local shapes,we introduce a multi-head shape attention layer to form a subspace between multiple local shape combinations to ensure the diversity of the combinations.In order to learn the mapping of part features to the whole,the above features are encapsulated into a primary capsule,and the large part shapes are continuously mapped to the whole model through a multi-layer dynamic routing mechanism.We verify the effectiveness of the vector-based network proposed in this paper through classification experiments on 3D deformation models,and our network construction method is interpretable. |
