Research On Gesture/Hand State Recognition And Handwriting Recognition Based On Inertial Measurement Unit

With the rapid development of computer technology,natural interaction in the field of human-computer interaction has become a research hotspot,and natural behavior recognition technologies represented by text recognition,gesture recognition,speech recognition,and context recognition are gradually gaining attention.Handwriting is a kind of natural human behavior,but there is still a shortage of handwriting recognition technology.At present,the optical text recognition technology based on camera is mature,but the problem of light blocking has become a major difficulty in the actual handwriting recognition application.On the other hand,handwriting recognition technology based on writing tablet has entered our life,but the writing tablet also limits the overall portability of the device and its practicality in paper-pencil writing environment or any writing surface.In order to solve the problems of current technology,this paper explores the feasibility of text recognition technology by starting from the problem of gesture recognition.This paper proposes a gesture and hand state recognition technology based on a single inertial measurement unit,and designs experiments and builds a real-time recognition prototype system around this technology: firstly,this paper visualizes and analyzes the hand state data and gesture data of the inertial measurement unit,clarifies the characteristics of the data in the time domain and frequency domain,and provides guidance experience for data set construction and neural network building.Second,a small dataset of hand states and gestures is constructed to provide training and testing data for subsequent model training.Again,the differentiated structure of the neural network is proposed for the variability of the data in the time and frequency domains,and the neural network recognition method is experimented and evaluated.Finally,a real-time program based on the neural network recognition method is constructed to verify the practical feasibility of the recognition technique,and the real-time computing capability of the real-time program is experimentally evaluated.The experimental results show that the gesture data and hand state data measured by the inertial measurement unit can be accurately recognized by the neural network,and the computation time consumed meets the real-time computing requirements.This part of the study also demonstrates the feasibility of stroke recognition and nib-touch recognition for writing recognition in subsequent experiments,providing a guiding experience for building a writing recognition model.The writing recognition process can be regarded as a series of simple gestures(strokes)recognition process.Therefore,based on the experimental results of the previous part of the study,this paper conducts a study on the recognition of capital letters based on stroke sequences.This paper firstly establishes the connection between gesture recognition and writing recognition,and proposes a technical route for writing recognition based on stroke sequences.Secondly,the data of 26 capital letters and their inertial measurement units containing strokes are collected and labeled.Thirdly,the minimum full set of strokes describing the uppercase letters was designed according to the types of strokes contained in the data set.Fourthly,a recognition network based on the fusion of frequency-domain and low-frequency time-domain features is constructed and trained,and both nib-touch recognition and stroke recognition are completed,and overall evaluation experiments and ablation experiments are designed to verify the effectiveness of the model.From then on,an additional classification method based on stroke duration is proposed for the stroke path repetition problem,and the feasibility of the additional classification method is analyzed using ANOVA method.Finally,the overall performance of the classification method is verified,and the performance is analyzed for both known users and unknown users,and a real-time system is built to verify the feasibility of real-time computation.The experimental results show that written uppercase letters can be accurately recognized using a single inertial measurement unit.Specifically,the recognition model frame recognition correct rate is98.2%;the overall stroke error rate reaches 0.6% and word error rate reaches 4.1% after combining additional classification methods;the simulated unknown user case reaches6.5% stroke error rate and 12.7% word error rate.