Research On In-air Handwritten Character Recognition Based On Inertial Sensor Signals And Path Signature

The rapid development of modern information technology is profoundly changing people's thinking and lifestyle.Nowadays,human-computer interaction(HCI)has become an indispensable role in people's lives.In recent years,the widespread use of mobile devices(e.g.,wearable devices and virtual reality)has caused in-air handwriting interaction to attract great attention in the field of human-computer interaction.In-air handwriting interaction refers to an interaction method that recognizes meaningful characters or text lines written in the air and converts them into machine instructions.Among various implementations,inertial sensor-based in-air handwriting has a wider range of application scenarios in the fields of smart home and smart education,due to its advantages of more relaxed handwriting environment requirements,lower equipment costs,and fewer violations of user privacy.However,because the inertial sensor signals are less intuitive in the form of time series,it is difficult to extract the hidden key information contained in them,which limits the related research and development.In recent year,the success of path signature(PS)in the fields of machine learning and financial data analysis has demonstrated its superior ability to model dependencies in sequence data.On the other hand,convolutional neural network(CNN)has achieved excellent performance in the research based on sequence data.In contrast to recurrent neural network(RNN)with memory limitations and weak parallelization within training,CNN has the advantages of high computing efficiency with highly optimized matrix multiplication code,and CNN is more suitable for deployment on wearable devices with limited computing resources.Based on the above viewpoints,this dissertation focuses on exploring how to apply path signature to inertial sensor signals and improve the recognition performance of in-air handwritten character recognition.The main research work and contributions are as follows:1.We explore and propose two PS-based methods to characterize non-intuitive inertial sensor signals.One is modulation-based path signature fusion representation(MPS),which uses modulation to fuse the inertial sensor data stream and its corresponding PS.Another is slidingwindow-based path signature feature representation(WPS).We calculate the PS feature of the inertial sensor data stream within a specific sliding window to extract the context information at different timestamp.2.We propose an MPS-based approach to recognize in-air handwritten characters.Experiments on the public 6DMG dataset prove that MPS can provides more effective information for different types of in-air handwritten characters to reduce the confusion between characters with similar in-air handwritten trajectories.This method achieve state-of-the-art recognition performance on in-air handwritten character recognition.3.We propose a WPS-based approach to recognize in-air handwritten characters.Experimental results demonstrate that,compared with MPS,WPS can ignore the original inertial sensor data,independently and effectively characterizing in-air handwritten characters based on inertial sensor.