Research On 3D Model Retrieval Of Freehand Sketch Based On Semi-heterogeneous Deep Learning

With the development of industrial manufacturing,virtual reality(VR),augmented reality(AR)and other fields,the number of 3D models is increasing at an amazing speed.Facing the massive 3D models,how to quickly and accurately retrieve the target 3D model has become an urgent demand of the society.The existing 3D model retrieval methods include text-based retrieval,instance-based retrieval and sketch-based retrieval.Among them,the 3D model retrieval based on text description,language differences exist problems,such as 3D model retrieval based on instance is hard to get ahead of the problem,and the 3D model retrieval based on hand-painted sketch because of hand-drawn sketch drawing is not affected by age,regional restriction,and the advantages of the rich semantic information,become a new research direction.However,freehand sketch is a subjective expression with intra-class diversity.There is a huge difference between the hand-drawn sketch and 3D model.This makes 3D model retrieval based on freehand sketch very difficult.In this paper,the main work is as follows:(1)A 3D model retrieval network MRSB-SHNet based on semi-heterogeneous network is proposed.Considering the visual difference and semantic consistency between freehand sketch and 3D model,heterogeneous network is used to capture their different visual information,and homogeneous network is used to capture their consistent semantic information,forming an endto-end deep learning framework.In order to solve the problem of intra-class diversity existing in the same type of freehand sketch and 3D model,an adaptive multi-class center operation was adopted to better fit the distribution of data.In order to solve the problem that the existing methods only use category tags to complete cross-domain semantic feature embedding,which leads to the weakness of semantic information capture,a label word vector is introduced to further utilize the information brought by category labels.Experiments verify the rationality of the semi-heterogeneous feature extraction network,the adaptive multi-class center operation and the introduction of tag word vector,and obtain good retrieval accuracy on SHREC2013 and SHREC2014,which verifies the effectiveness of the algorithm.(2)Based on the algorithm presented in this paper,a multi-modal 3D model retrieval system for hand-sketching is implemented.The system can support users to complete text,sketch and 3D model of mutual retrieval.It provides a good visual interactive environment,including text retrieval of freehand sketch and 3D model,freehand sketch retrieval of freehand sketch,freehand sketch retrieval of 3D model,freehand sketch retrieval of 3D model and 3D model retrieval of 3D model.Through the comprehensive function test and performance test of the system,it is verified that the system has high retrieval accuracy and retrieval speed.