Research On Task Execution Of Interactive Service Robot Based-on Speech And Pointing Gesture

The natural language and gesture are the most commonly used communication ways between people.So the robot should not only execute tasks based on natural language instructions,but also search for the target object along the direction of pointing gesture when position of the target object is unknown.And the robot can communicate with the user automatically when the robot cannot identify the target object.These abilities are significant for service robots to execute daily tasks,especially assisting the disabled and the elderly.At present,there are few researches on Chinese natural language instruction parsing.What's more,the key information extraction model of these researches only considers the POS(Parts Of Speech)as feature.But the dependency syntax plays a very important role in natural language instruction parsing.The researches on gesture interaction mainly focus on gesture recognition,but the gestures can also provide direction information.The key information extraction model,pointing gesture extraction method and intelligent task planning method are discussed and studied in this thesis.The major works are listed as follow:(1)'Key information extraction model:natural language instructions are analyzed based on Chinese word segmentation,POS tagging and dependency parsing.The original corpus is annotated by combining the key information extraction rules and manual annotation.We use ten-fold cross validation method to train and test the CRF(Conditional Random Field)model.We then obtain the key information extraction model.To express the task concisely,the key information are expressed in the form of a vector.(2)Pointing gesture extraction method:the user's skeleton is tracked by ASUS Xtion Pro Live depth camera,and the head-hand and shoulder-hand vector are used to construct a second order cone to solve the problem of inaccurate pointing gesture.The internal area of the second order cone is a potential region where may contain the target object.When the target object is out of sight of the robot,ASUS Xtion Pro Live depth camera is used to localize the robot in the environment based on ORB-SLAM2 algorithm.It enables the robot search for the target object that lies in the second order cone.(3)The integration of speech information and pointing gesture:the Bayesian filter is used to ground the noun to the actual object in the environment by fusing the pointing and speech information.When the robot cannot accurately identify the target object and the target object is ambiguous in the natural language instruction,the robot will communicate with the user to acquire supplementary information based on the objects in the environment.(4)Intelligent task planning method:to enable service robots execute tasks autonomously,we use PDDL(Planning Domain Definition Language)language for task planning based on the initial state and target state of each object in the environment.The natural language instructions are then parsed into a sequence of actions,and the robot execute the task step by step according to the sequence of actions.