Human Hand Motion Capturing And Recognition Based On Multimodal Sensing

By simulating the dexterous manipulation,perception and cognitive mechanisms of human hands,robots can perform complex human-like manipulation tasks and play an increasingly important role in aerospace,warehousing logistics,medical,military,and gesture-controlled safe driving etc.Multifingered robotic hand is the core component of robots that perform dexterous manipulation on different objects.Human hand and multifingered robotic hand have commonality to the dexterous manipulation of the object.The research results of human hand motion capturing and recognition are the basis of multifingered robotic hand manipulation.This thesis is partially supported by the Natural Science Foundation of China(Grant No.51575412,Grant No.51575338,Grant No.51575407)and the EU Seventh Framework Programme: DREAM Project(611391).It takes the human hand as the bionic object,and launchs a deeper research for human hand motion capturing and recognition.The main work and research results of this thesis are as follows:(1)The research on the human hand dexterous manipulation mainly shows in some simple motions,lacking in-depth research on the complex motions,such as twohand coordination,multiple manipulation,etc.By analysing the manipulation mechanism and natural structure of human hand,the human hand manipulation method is proposed.The force closure and the movement space of the fingers are analyzed,as well as the grasping points.According to the manipulation tasks,the object characteristics and the external environments,the human hand dexterous manipulation strategy is proposed.Considering the complexity of the human hand motions,a human hand motion classification method is proposed.It is divided into simple motions and complex motions,which lays the foundation for human hand motion capturing and recognition.(2)In terms of simple hand motion capturing,a multimodal data acquisition platform is designed,which mainly includes surface EMG signal acquisition system,hardware gloves and finger pressure sensing system,based on the limitation of unisensing technology.Firstly,the human hand motion capturing metgod and the motion set are designed by selecting appropriate subjects;secondly,the raw data is obtained,and simultaneously uploaded to the computer through the high-speed digital signal processor;thirdly,a threshold-based motion segmentation method is proposed to obtain the original signal;fourth,the correlation of the three signals is analyzed and studied based on the empirical connection function;then,six types of the original signal features are extracted to form the feature sample library.The research results provide abundant raw data for simple motions recognition.(3)An ADAG-SVM based simple motion recognition method is proposed to improve the recognition rate of human hand motion,due to the issues of inseparable regions,misclassification,rejection and accumulation of multiclass support vector machine for motion recognition.By selecting the reasonable kernel function,the best penalty parameter and nuclear radius,MATLAB software is used for hand motion recognition and comparative sresearch.The experimental results show that the average recognition rate of the proposed recognition method based on ADAG-SVM reaches 94.57%,which is higher than other recognition methods,and different subjects have obvious differences in the recognition results of different motions.The proposed recognition method based on ADAG-SVM is applied to vehicle driving assisted gesture recognition,and have a satisfactory recognition rate of 94%.The research results improve the simple motion recognition rate of human hands,and provide a practical theoretical basis for human motion recognition in the application of vehicle traffic.(4)To solve the joint calibration problem of two or more non-contacted sensors in the complex in-hand motion recognition,a hybrid multimodal sensing technology is used to capture the complex human hand motions.In order to more realistically simulate,the perception mechanism of humans,it is necessary to compensate for the drawbacks of uni-modal,sensors by using a combination of varied types of sensors.By using hybrid multimodal sensing,this thesis proposes a complex motions recognition method based on SEMG and Kinect.The contact sensor uses the SEMG to obtain the SEMG signal,and the non-contact sensor uses the Kinect to acquire the synchronized color and depth image.Then,the image based motion segmentation and feature extraction method including the distance feature and the curvature feature are proposed to obtain the multimodal feature information.The research results provide abundant raw data for complex motions recognition.(5)In the aspect of complex motion recognition,appropriate machine learning method is the key to improving the recognition rate of complex motions,so a Marquardt-Levenberg(ML)artificial neural network algorithm is used for complex motions recognition.Through MATLAB software,the hand dexterous manipulation experiments are expounded in two aspects including different sensing technique based complex motions recognition and different subjects based complex motions recognition.The experimental results show that the ML algorithm is easy to converge and the calculation speed is very fast.The average recognition rate is 95.10%,which is higher than other sensing technologies.Multimodal sensing based human hand motion acquiring and recognition provides a theoretical and technical basis for dexterous manipulation of bionic multifingered robotic hand.