Research On Gesture Interaction Technology Based On Depth Map In Virtual Assembly System

With the rapid development of the industry technology,the requirements for assembly quality and assembly accuracy are also getting higher and higher.The virtual assembly system can assist engineers in discovering the problems existing in the assembly of the designed products in advance,thereby improving the efficiency and reliability of the assembly design process of the products.As one of the core technologies supporting the operation of virtual assembly systems,human-computer interaction technology has also made major breakthroughs with the vigorous development of computer technologies such as deep learning,big data,and artificial intelligence.As one of the main interaction methods of virtual assembly system,gesture interaction can not only improve the naturalness of virtual assembly,but also promote the digitization process of manufacturing assembly links,which has important research value.At present,in the gesture interaction of the virtual assembly system,there is still a problem that the accuracy of the input data preprocessing results and the hand joint point estimation results cannot satisfy that assemblers precisely control the virtual hand model to naturally interact with part.Aiming at the above problems the main research contents of the thesis are as follows:(1)Aiming at the blurred contour of the depth map itself,which will cause the cumulative error of subsequent hand joint estimation,the research proposes a hand region segmentation algorithm based on the alignment of the color map of the depth map.Based on the principle that the same position area of the aligned depth map and color map represents the same object,the thesis uses an adaptive elliptical skin color model to determine the hand region with clear contours form the color map,so as to obtain a depth map of the accurate hand region,and converts the deep map of segmented hand region into a 3D space hand point cloud to improve the utilization of data information.(2)Aiming at the problem that the hand is prone to self-occlusion and the estimation of hand joints is difficult due to the frequent movement of the assembler’s hand and the large change of the direction of the hand in the virtual assembly system,the research proposes a feature reconstruction method based on edge convolution(Edge Conv).Based on the theory that using Edge Conv to reconstruct features in Graph Convolution Neural Networks(GCNN)will consider the relationship between points,the thesis uses Edge Conv to reconstruct the features of joint points,and the reconstructed features consider the information between hand joints and improve the estimation accuracy of hand joints.(3)Finally,based on the depth map preprocessing technology and gesture estimation algorithm designed above,a virtual assembly system is developed and designed in the research.The virtual assembly system realizes the function of the control of virtual hand through binding the hand joint information output by gesture estimation and the virtual hand together,realizes the precise recognition function of the assembler’s grasping intention by combining the two-level distance threshold detection structure with the label prediction method based on gesture and shape.,in the assembly of parts,realizes the function of the virtual hand’s snugly grasping of the part model,the grasping movement of the parts,the collision detection between parts,and so on.The system designed and implemented in the research can complete the interaction between the virtual hand and the part model naturally and smoothly,and has strong versatility and promotion value.