On Reinforcement Learning Control For Bionic Underwater Robots

The bionic underwater robot is one of the hotspots in the underwater robotics research field in recent years. It has complicated dynamic characteristics and uncertain working environments which make the motion control of bionic underwater robots a challenging problem. This thesis takes the bionic underwater robot with two undulating fins as research object, and aims to figure out the motion control problem in the framework of reinforcement learning. The studies in this thesis concentrate on the motion control problems analysis, the reinforcement learning algorithm design, reinforcement learning based attitude stabilization, reinforcement learning based trajectory tracking, and corresponding experimental verifications. The main achievements and progress are as follows:(1) The motion control problems of the bionic underwater robot with two undulating fins are analyzed from the bionic inspirations, the dynamic characteristics of the bionic undulating fin and the bionic underwater robot. The morphology and swimming characteristics of Gymnarchus Niloticus and Bluespotted Ray are investigated to provide inspirations to the design of bionic underwater robots; and then, the bionic undulating fin thruster and the bionic underwater robot with two bionic undulating fins, two swing fins and a 2-DOF bionic bladder are designed. After that, the thrust testing and motion testing are carried out and the corresponding dynamic characteristics are analyzed, which provide directions for the motion controller of bionic underwater robots.(2) According to the requirements of robot control and the restrictions of the original Q-learning algorithm, a continuous state and action space neural Q-learning algorithm (CSANQL) is presented, which lays a foundation for the reinforcement learning control of bionic underwater robots. By utilizing the neural network, the database of learning samples, the fitting function of estimated Q values and the original Q-learning algorithm synthetically, the CSANQL algorithm realizes fast mapping between continuous states and continuous actions. The two structure of neural Q-learning, the mechanism of generating continuous actions by the fitting function of estimated Q values, and the effects of the database of learning samples on improving the learning efficiency are detailed. The approaches of incorporating the reinforcement learning algorithms into the motion control of bionic underwater robots are also discussed.(3) For the attitude stabilization of the bionic underwater robot, three reinforcement learning based attitude stabilization methods including reinforcement learning based adaptive PID controller, reinforcement learning controller and supervised reinforcement learning controller are put forward and implemented. The adaptive mechanism of parameters based on reinforcement learning is discussed, and the functions of the databae of learning samples and the supervisory controller are elaborated. Simulations are carried out to test the validity of the reinforcement learning control in attitude stabilization. Results indicate that reinforcement learning based adaptive PID controller can learn the optimal PID parameters actively, and has good attitude stabilization performance; the database of learning samples has great influence on the performance of reinforcement learning controller, and preferable performance can be achieved if proper capability of the database is given; the supervised reinforcement leanring controller have better performance than reinforcement learning based adaptive PID controller and reinforcement learning controller in learning efficiency and dynamic process.(4) For the trajectory tracking of the bionic underwater robot, a behavior control structure which based on reinforcement learning behaviors is devised and implemented. Thrusting behavior, yawing behavior and depth-keeping behavior are extracted from complex trajectory tracking tasks, and treated as three basic control behaviors which would realize almost all trajectories in 3D world. The implementation and performance of each basic control behavior are discussed; and a reinforcement learning based optimization method for behaviors combination is expounded. Simulations with straight line trajectory and curve trajectory are performed to validate the validity of reinforcement learning control for trajectory tracking. Results indicate that the reinforcement learning behavior control structure can respond to the target trajectory quickly and has favorable tracking performance.(5) Experiments are carried out based on the bionic underwater robot with two undulating fins to test the reinforcement learning control methods both in attitude stabilization and trajectory tracking. According to the experimental data, the CSANQL based supervised reinforcement learning controller behaves better than pure reinforcement learning controller and convertional PID controller in attitude stabilization; and the bionic underwater robot can perform some trajectory tracking tasks with good performance in the control of reinforcement learning behavior control structure.All the above achievements effectively facilitate the breakthrough at the motion control of the bionic underwater robots and the practical application of reinforcement learning control methods, and consequently, lay the foundation for realizing efficient autonomous motion control of bionic underwater robots within the reinforcement learnig framework.