Multi-modal Man-machine Interaction Technology For Robotic Teleoperation

As humankind's eagerness for exploration of a wider range of unknown nature grows stronger,and the concept of protecting its own security increases,systems that can be controlled remotely are needed in many conditions.The implementation of teleoperated robotic systems frees people from appearing directly in the robot's operating environment and can remotely control them locally,greatly changing the way robots operate and expanding the robot's ability to work.Robotic man-machine interaction technology has become a key and hot research direction in teleoperation field,and robotic man-machine interaction technology still has many difficulties in terms of naturalness,coordination,accuracy and security.Continual improvement of exsiting man-machine interaction methods of teleoperation or proposing a more complete interactive method of function features has always been one of the focuses of the development of teleoperated robot systems.Based on the research and analysis of existing technologies,this paper proposes a multimodal man-machine interaction technology for robotic teleoperation.It fuses multiple sensing technologies to detect operator's movement intentions,and combines virtual reality technology with Robot control technology to build a robot movement virtual environment,the specific content is as follows:A multi-modal man-machine interaction system for robotic teleoperation is designed.It is mainly composed of motion intent detection system,man-machine interaction software and robot movement virtual environment.The related equipment and technologies used in the system are briefly introduced,including two important interactive devices,the Myo armband and the Kinect somatosensory device,as well as the OpenGL graphics library and the Unity game engine.A multi-modal motion intent detection method based on multiple sensing devices was proposed.A surface electromyography signal acquisition device and a motion judgment method were designed.The C++ software for the gesture recognition information collection of the Myo armband under the Windows platform was developed.The arm end position tracking software based on the Kinect's skeleton frame is developed.The arm end posture tracking is achieved by using the Inertial Measurement Unit(IMU).Based on the OpenGL graphics library,interactive software was developed to provide functions such as controlling the acquisition process of interactive devices,dynamically presenting the operator's posture,related data,coordinate transformation,inverse kinematics of the robot,and sending control information.A 3D model of each part of the robot was produced,imported into Unity,a robot movement virtual environment was set up,and a control script was written to implement functions such as robot movement control.Finally,relevant experimental studies were carried out.The stability of some devices in the system was measured experimentally,and the human-machine interaction system was tested to follow the movement of the operator.Interactive experience experiments were conducted to compare the efficiency of various interaction methods.The interaction mode was analyzed and summarized.