Research On Human Action Recognition For Multi-modal Human And Robot Interaction

With the development of robot technologies, the applications of robot have beenextented to the service field and human world. To work in the human society, servicerobot need to have the natural human robot interaction ability. Currently, some problemsstill unsolved in human robot interaction. For natural and efficient communication, weneed to improve the service robot interaction ability on the basis of human intercationmodals. Considering the advantages of human action recognition for human robotinteraction of service robot, we systematically researched the human action recogntiontechnology, including hand posture recognition、upper body gesture recogniton、gaitrecogntion, and set up a multi-modal human-robot interaciton based on human actionrecognition.In the dissertation, we first proposed a vision based method to overcome theproblems of hand posture recogntion in human robot interaction. We casted hand posturerecognition as a sparse representation problem, and proposed a novel approach calledjoint feature sparse representation classifier for efficient and accurate sparserepresentation based on multiple features. By integrating different features for sparserepresentation, including gray-level, texture, and shape feature, the proposed method canfuse benefits of each feature and hence is robust to partial occlusion and varyingillumination. Additionally, a new database optimization method was introduced toimprove computational speed. Experimental results, based on public and self-builddatabases, showed that our method performed well compared to the state-of-the-artmethods.By fusing the information from color image and depth image, a new upper bodygesture recognition method was proposed. By means of the VGA camera and depthcamera calibration, the coordinate transformation between color image and depth imagewas estimated. Key points of upper body object were extracted based on human skeletonmodeling. Then the coordinates of key points in spherical coordinate system wererepresentated by3D histogram. According to the model-based sparse classificationmethod,10upper body gestures were recognized in the experiments. Compared withcommon used methods, our method achieved the better results, especially with the complex background and different users.A method for gait feature extraction is given based on laser range data. Thecontourship of the legs are quickly picked up from the laser data in the large area. Inorder to collect the gait features, such as walk speed, step length, step time and stepvelocity, the position of human were located by legs position extracted from laser data ofcontinous frames. Experimental results show that our method performs well for gaitfeatures extraction.Finally, multi-modal human and robot interaction system was formed based on ourservice robot. To achieve natural and universal human and robot interaction, a newhuman robot interaction architecture has been proposed on the basis of semanicunderstanding. By fusing human action information, facial expression, voice information,user was able to natural interact with service robot. According to the evaluation of themulti-modal human and robot interaction system, our system were test based on severalexperiments.In this thesis, to get natural and univeral human and robot interaction, a mobilerobot platform was developed with mulit-modal human and robot interation system byhuman action recogntion which improve the interaction performance of the robot.Experiments presented in this thesis verified that these techniques can improve theperformance of service robot and possess a practical reference value.