Design And Implementation Of 3D Spatial Handwriting System

Handwritten character input is an important field in real-time humancomputer interaction(HCI),and the current mainstream information input method relies on physical media devices such as mouse,keyboard,stylus,touchpad and wearable devices for information input.With the innovation of computer hardware and the development of computer vision technology,the user’s hand information can be collected by depth camera,which can realize the input of text writing and drawing trajectory in the air without relying on wearable devices,physical media restrictions,or changing the user’s writing habits,and achieve the goal of "human-centered" humancomputer interaction.At present,the aerial handwriting technology implemented by computer vision technology cannot realize the user’s stroke-splitting input,or needs to rely on designated body movements to realize the strokesplitting input,but the effect is poor.Therefore,this paper researches and designs a 3D spatial handwriting system based on computer vision technology and gesture recognition technology,which realizes the functions of recording,erasing,clearing and saving the user’s writing trajectory in 3D space in real time,while the system can accurately recognize the user’s continuous or split-stroke input.The main work accomplished in this paper is as follows:First,the recognition and acquisition of hand targets.The position of the hand bounding box is detected by the target detection algorithm,and the hand depth map is extracted from the background,and the complete hand depth map is extracted and projected into the world coordinate system.Second,a voxel-based gesture pose estimation method is implemented.The hand depth map is converted into 3D voxel data,and the spatial information of the user’s hand is fully utilized to estimate the location of key points of the hand in 3D space with the help of 3D convolutional neural network and residual network,and the key points are mapped into the writing plane.The experimental results demonstrate the superiority of this method by comparing it with the depth map-based gesture estimation method.Third,a voxel-based gesture recognition classification method is implemented.Using the spatial information of 3D voxels,a voxel-based neural network model for gesture classification was designed and implemented.This model achieved a classification accuracy of 96.9%in the gesture classification task,and compared with the commonly used classification models.Fourth,a 3D spatial handwriting system is designed and implemented based on the above.