Teleoperation System For Multi-joint Robot Arm And Predictive Control With Communication Delay And Packet Dropout

Teleoperation system is a real-time human-robot interaction method,which can expand the range of human activities and help humans complete tasks in complex and dangerous environments.On the one hand,the physical structure of the human arm and the multi-joint robot arm are different.The operator usually needs to wear a sensor device or manipulate a joystick,which affects the operator's natural extension in teleoperation.On the other hand,there are time delays and packet dropouts in the network transmission,which brings great obstacles to the teleoperation.The main research content of this paper is to make the human body teleoperate the multi-joint robot arm through visual teleoperation,and to ensure that the teleoperation system with delays and packet dropouts has good control performance.Firstly,human-robot posture-consistent teleoperation is proposed in this paper.The vision-based posture-consistent teleoperation can achieve the robot arm following a human arm and keep same postures.Based on the convolution neural network in deep learning,a multi-stage visual teleoperation network is designed to learn the nonlinear mapping between human body data and robot arm joint angle data.A novel human-robot posture-consistent mapping method is designed.A human-robot posture-consistent dataset is established by a data generator,which is able to calculate the corresponding robot arm joint angle data from the human body data.A neural network training framework is used to build and train the visual teleoperation network based on the human-robot posture-consistent dataset.Secondly,networked predictive control with a quantizer by an event-driven strategy is proposed for saving the communication badnwidth and compensating time delays and packet dropouts in the networked teleoperation system.A predictive controller is designed to predict the future input based on the past state and input,which is used to compensate the impact of time delays and packet dropouts in the networked control system.An event-driven strategy based on a state observer is designed to reduce the transmission frequency of data packets,and arbitrary area quantizers are used to quantify the input andstate in order to reduce the size of the data packet.Quantization method and event-driven strategy are proposed to save networked communication bandwidth while time delays and packet dropouts are compensated by predictive control.Based on a “zoom” strategy,sufficient conditions are given to ensure stabilization of the networked control system.Finally,a visual teleoperation experiment platform is built.The Azure Kinect sensor is used to sample human body information,the socket program is written to achieve networked communication,the ROS(robot operating system)is used to control the multi-joint robot arm,the simulation is given in Gazebo.A visual teleoperation network evaluation experiment and a robotic experiment are provided to demonstrate effectiveness of the proposed visual teleoperation framework.