Body Gesture Recognition And Interaction Based On Surface Electromyogram

As computing devices have become involved into all aspects of human living environment, the real world, digital world and the people who constitute the main body of the world are organized as an integrated whole. The demand of seamless communication and liberated interaction between people and the environment promotes the human body gesture recognition to become the hot spot of the research on future multi-channel and multi-modal human-computer interaction (HCI). Human body gesture recognition is known as the process that the computer automatically capture, analyze and understand the various types of gestures and human behaviors, such as fingers, wrists, arms, head, face and body posture or gesture patterns, to determine people's intentions and provide the corresponding services.Arbitrary body movements are completed by groups of muscles which are coordinated and work closely together under the control of the nervous system. The electromyographic (EMG) signal caused by muscular activities is regarded as one kind of important bioelectric signals. Attached on the surface of skin above relevant muscles, Surface EMG (SEMG) sensor can capture the information of human muscular activities, which not only reflect the state and strength of flexion and extension of the joints, but also reflect the information of limb postures and positions. The EMG processing technologies provide us with important opportunities to capture human body gestures.Aiming at the recognition of hand gestures, neck and shoulder gestures and leg motions, the dissertation investigates the detection and recognition of various kinds of human body gestures based on SEMG signals, designs real-time gesture-based interactive systems, conducts a certain amount of user testing experiments, and provides some practical solutions for the natural and harmonious HCI. The research will promote further the development, application and extension of multi-modal intelligent HCI techniques. Moreover, the research achievements are of sufficient importance in the fields of human behavior understanding, rehabilitation medicine, context awareness, pervasive computing, and navigation. The main work and achievements of the dissertation focuses on the following aspects:1. Hand gesture recognition based on the multi-channel SEMG. The purpose of this study is to realize effective algorithms for hand gesture-based HCI systems, and to provide theoretical foundations for the selection of input hand gesture commands and SEMG sensor placements. In one aspect, the SEMG processing and recognition methods including signal measurement, active segmentation, feature extraction and classification are studied to classify 8 kinds of commonly used hand gestures. Subsequently, an optimized algorithm is proposed for real-time interaction. And on this basis, a SEMG-based real-time hand gesture recognition system is established. In the other aspect, according to the anatomical knowledge, optimization study on the definition of hand gesture commands and SEMG sensor placement is conducted on the classification of 20 kinds of hand gestures including some subtle finger movements. Thereby, a practical interactive scheme for HCI applications is proposed. The user testing experiments are conducted on the real-time hand gesture recognition system in user-specific, multi-user, and user-independent classification. The experimental results demonstrate the robust performance of proposed interactive scheme. The achievements of this study can provide important references on the selection of input hand gestures and the placement of SEMG sensors in applications of SEMG-based HCI.2. Sign language recognition based on the information fusion of acceleration (ACC) and SEMG The aim of this study is to investigate the hand gesture recognition technique based on the information fusion of multiple sensors. ACC-based methods are capable of distinguishing larger scale gestures with different hand trajectories of forearm movement, whereas SEMG-based recognition systems are capable of distinguishing subtle gestures with different muscular activities, such as subtle finger or wrist movements. Considering the complementary characteristics of ACC- and SEMG-based measurements, a framework for hand gesture recognition based on the information fusion of a 3-axis ACC and multi-channel SEMG is presented. The framework utilizes multi-stream HMM and decision tree for information fusion of the two heterogeneous sensors. Based on the framework, the classification of 30 kinds of Chinese sign language (CSL) words and 16 CSL dialog sentences is implemented. Furthermore, a promising real-time interactive system is built for the control of virtual Rubik's cube game using 18 kinds of hand gestures.3. Neck and shoulder gesture recognition based on SEMG sensors. Neck and shoulder gestures can be regarded as a supplementary mean of natural and harmonious HCI. The feasibility and practicability of building muscle-computer interfaces starting from SEMG-based neck and shoulder gesture recognition is investigated. The multi-channel SEMG signals are measured from the relevant back, shoulder and neck muscles to classify 7 kinds of neck and shoulder gestures. Then, a real-time SEMG-based neck and shoulder gesture recognition and interaction system is established by the improvement of real-time hand gesture recognition system.4. Personal navigation fused with SEMG information. Considering the characteristics of the left and right leg alternately making each pace and contractions of muscles are cyclic when a pedestrian is walking, the EMG signal from the surface skin of calf (Gastrocnemius) showed a significant rhythm according with strength of every pace exerted by leg muscles. Taking advantage of human physiological characteristics during walking, a novel pedestrian dead reckoning (PDR) method is proposed with the fusion of SEMG-based technique for analyzing pedestrian's activities and digital compass based technique for measuring azimuth. In the PDR method, overlapped windowing schemes and sample entropy feature extraction are firstly utilized to process gastrocnemius SEMG signals, and HMM classifiers are used to classify pedestrian activities such as walking or standing still. Then the SEMG-based step detection and step length estimation are implemented during walking, and are combined with the heading of each step measured by digital compass to determine the trace and position of the pedestrian. The field tests demonstrate that a GPS receiver integrated with our proposed PDR method has great potential to provide feasible and effective solutions to seamless outdoor/indoor pedestrian navigation.The research is supported by the National High Technology Research and Development Program of China (The 863 Program) "Research on the Gesture Input Devices Based on the Accelerometers and Surface EMG sensors" (2009AA01Z322), National Natural Science Foundation of China "Chinese Sign Language Recognition based on Surface Electromyogram" (60703069), cooperation projects with NOKIA Research Center (Helsinki & Beijing) and Graduate Innovation Foundation of USTC.