3D Shape Representation And Recognition Via Multi-modal Networks

3D object representation and recognition are crucial in manufacturing and intelligent transportation systems,which have attracted much attention.However,complex representations and organization of 3D object(such as point cloud,multi-view,voxels,mesh,etc.)increase the task’s difficulty.In the multi-modal representation learning of 3D models,a well-designed multi-modal learning framework solves not only the single-modal learning problem,but also fuses the data representations of different modalities.In this work,we address the above-mentioned problem from three perspectives.First,we propose a Group-view Convolutional Neural Network(GVCNN),which can recognize 3D objects based on each view’s discriminative for multi-view representations.Second,Point-view Network(PVNet)is designed for objection recognition from the joint representations of both point cloud and multi-view,which combine the high-dimensional features of point cloud and multi-view from both local and global perspectives.Third,we proposed Point-view Relation Network(PVRNet),which can automatically match views and point cloud for multi-modality fusion in 3D object recognition.We also provide the visualization results of the three network models,respectively.Experiments on ModelNet40 demonstrate the effectiveness of the proposed GVCNN,PVNet and PVRNet in 3D object classification and retrieval task,respectively.