The Application Of Deep Inverse Reinforcement Learning In Robotic Visual Servo Control

The application of reinforcement learning(RL)in robot visual servo control has always been a challenging task.In order to construct a visual servo control system which based on deep reinforcement learning algorithm(DRL),this paper starts from two aspects: 1)the modeling method of reinforcement learning in robot control system;2)is some engineering methods to enhance the generalization performance of reinforcement learning model.First of all,this paper discusses the basic algorithm of RL from the aspect of servo control modeling.In general,the model based methods of RL is the preferred method on robot control,this method uses the environment model which deduced from demonstration data to obtain the parameterized form of the actual environment,and then use it to optimize control policy.Therefore,this method relies on artificial modeling and is not expansibility.For this reason,this paper introduces a policy guided search algorithm(GPS)does not depend on the model and supports offpolicy learning.This algorithm adopts importance sampling method,which can not only introduce external teaching data in training process,but also improve data utilization efficiency through historical data resampling.Besides,it can introduce human knowledge into training process by inverse reinforcement learning(IRL)which is an important RL algorithm.For IRL,this paper discusses the maximum entropy IRL method which is characterized with non-deterministic policy and nonlinear reward structure.Contrast to other IRL,this algorithm increases the function representation ability of learning model,and can adopt into the complex visual servo control task.Though the combination of GPS and IRL can represent complex problems,it is difficult to training because of the nonlinear structures.For this reason,this paper introduces the engineering optimization method of RL.Firstly,in order to reduce the high dimension of the image in the visual servo application,the migrated convolution network is used for state compression and feature extraction.Secondly,in order to reduce the sample of reinforcement learning,this paper introduces a method of model pre-training in simulation environment,which extends the training samples in a random way,and improves the generalization performance of RL models.Finally,after completing the pre-training of visual model migration and decision model,we can use the real demonstration data to fine tune the model,and we can get better experimental results.Finally,under the guidance of the above theory,an enhanced learning system for robot visual servo control is designed.After that,this paper uses ROS to build a learning and control software for Jetson TX1 and UR5 robot.After completing the arm stretch experiment,we can find that the reinforcement learning method based on visual servoing can accomplish complex visual tasks successfully,with good adaptability.