Research And Implementation Of Key Technologies For Dynamic Gesture Interaction In Virtual Simulation Experiment

The experience triggered by the human-computer interaction design of virtual simulation experiment is the key factor to enhance learning interest.However,most of the existing human-computer interaction methods in virtual simulation experiments use mice,keyboards,and touch screen inputs,resulting in poor immersion and poor interaction.,new and convenient ways of interaction between users and computers are increasingly valued.Gesture interaction technology,due to its natural,friendly,and convenient characteristics,can provide users with a sense of control that approximates real experiments,gradually becoming a key research topic in virtual simulation experiments.Based on computational vision to detect and recognize gestures,combined with human-machine interaction control theory,can realize the interactive control of dynamic gestures on the virtual simulation experiment scene.However,the existing gesture interaction process still has the problems of slow speed and low accuracy of hand detection and tracking and joint point estimation,which makes it impossible to achieve natural and smooth gesture interaction.This thesis studies the key technologies in gesture interaction,such as hand detection and tracking,hand pose estimation,and gesture recognition,and realizes the natural strong interaction control of dynamic gestures on virtual simulation experiment scenes.The main research contents include the following three parts.(1)Aiming at the problems of complex network models,high computational complexity,and slow tracking speed of hand tracking methods based on detection,a hand detection and tracking method based on feature joint learning was proposed.Firstly,the detection network Yolox-s is used as an end-to-end detector,and added an attention module to improve the detection accuracy;Secondly,a feature joint learning module is constructed at the network head to share the training parameters of the backbone network for hand detection and appearance feature learning,reducing the complexity of the model;Finally,the learned hand detection and appearance features are fused to achieve hand tracking by data association matching.(2)Aiming at the problem that the top-down hand pose estimation method takes a long time and ignore the internal relationship between hand joint points,resulting in low accuracy of hand pose estimation.A hand pose estimation method based on GCN feature enhancement is proposed on the basis of hand detection and tracking.Firstly,Rex Net is used as the backbone network,and deep separable convolution is adopted to reduce the number of model parameters;Secondly,using deconvolution to replace the backend structure of backbone network,estimate the heat map from the feature map and obtain the heat map of 21 joint points;Finally,a GCN feature enhancement module is constructed to modify the heat map estimation to improve the accuracy of hand joint point estimation.(3)On the basis of hand detection tracking and hand pose estimation,a virtual simulation experimental system based on dynamic gesture interaction is developed to realize interactive control of dynamic gestures on virtual simulation experimental scenes.Firstly,a dynamic gesture template library is established to define gestures and their interactive operations;Secondly,dynamic gesture matching recognition is implemented through dynamic time warping algorithm;Finally,build a virtual simulation experiment scene in the Unity 3D engine,and C # scripts are written to call the algorithm output to achieve gesture control over the virtual scene,improving the interactive experience of the virtual simulation experiment process.