Research On The Master-slave Hand Cooperative Perception Based On Information Fusion Of Ontology Gesture And Interactive Tactile Perception

In recent years the research of robot operation perception technology has been widely concerned by scholars.Master-slave hand cooperative perception technology is an organic combination of human intelligence and robot's ability to adapt to the environment,which can make the robot complete complex operations in the unknown dangerous that is difficult to reach or harmful to people.Researches status on the master-slave hand cooperative perception technologies were introduced.A series of problems are existed,such as insufficient sensor flexibility,poor wearability,asynchronous master-slave hand mapping,difficult to accurately perceive and identify interactive state,difficult to realize the operation tasks and lack of feedback information,in the application of master-slave hand operation cooperative perception.Therefore,sensor design method,master-slave hand operation cooperative perception method design,analysis method of grasping operation perception,tactile feedback system design,and legacy analysis were explored by combining the theories of sensor technology,robot kinematics,artificial intelligence,neural computing,data fusion technology,to achieve the goal of the master-slave hand cooperative perception based on ontology gesture/interactive tactile perception by flexible tactile sensor.Main research contents and innovations are summarized as follows:(1)A novel flexible and stretchable strain sensor for master-slave hand ontology gesture perception,and a novel flexible tactile sensor for robot hand interactive tactile perception design method were elaborated.A fully flexible and highly stretchable strain sensor was proposed via hydrogen bond-assisted dual-mode cooperative conductive network.The stretchable sensors were arranged at the master-slave hand joints for perceiving the ontology gesture information.Meanwhile,a novel full flexible pressure sensor array was proposed which was placed in the palm of the dexterous hand to endow the dexterous hand the ability of interactive tactile perception.(2)Master-slave hand ontology gesture perception and coordination methods were presented.A fully flexible and wearable data glove and a strain sensor array based on the flexible stretchable strain sensor were constructed for capturing the joints bending of master-slave hand.The methods of master-slave hand ontology gesture capture,joints angle coordination and motion space mapping were studied.The master-slave hand joints motion coordination model was established by combining the nonlinear partition and regression analysis.Meanwhile,the fingertip motion space trajectory contour was established by combining the master-slave hand motion model and theory of robot kinematics so as to solve the inconsistent of the master-slave hand motion space.(3)A method of perception and recognition of object under blind grasping operation situation of master-slave hand was proposed.A multi-sensors perception array was proposed by utilizing ten flexible strain sensors,fourteen temperature sensors and seventy-eight flexible pressure sensors and to measure the finger joints bending angle of the perception hand,object temperature information and interactive pressure distribution information,respectively.The models of cross-modal information representation and fusion were established on time and space sequences.Furthermore,the multimodal information was fused with deep convolution neural network to realize the accurate perception and recognition of the interactive objects under blind grasping operation.(4)A perception method of dexterous hand grasping operation state was proposed.The interactive tactile perception information sequence is divided into a series of sub tactile sequences based on the tactile perception array.Combining dynamic time warping and linear dynamic system,temporal and spatial feature extraction and analysis of each sub tactile sequence,and neural computing technology was used to cluster and analyze the interactive state of grasping operation so as to realize the accurate perception and recognition of grasping operation state of dexterous hand.(5)The master-slave hand cooperative perception method based on ontology gesture/interactive tactile perception and tactile feedback was proposed.The multimodal information fusion strategy of master-slave hand ontology gesture/interactive tactile integration and feedback was constructed based on the data glove and tactile perception array to realize the accurate perception,real-time feedback and efficient integration of the grasping operation state and grasping force of the dexterous hand.The problems of the master-slave hand tactile perception information precise acquisition,the master-slave hand ontology gesture precise synchronization and collaboration,the master-slave hand cooperative operation grasping state precise perception,accurate recognition of the interactive objects and the master-slave hand tactile telepresence were solved and realized the precise cooperative perception of the master-slave hand based on information fusion of ontology gesture and interactive tactile perception.