Deep Reinforcement Learning Based Walking Control Of Biped Robot

With the continuous development of robotics,service robots are increasingly entering people's productive lives.In some specific application scenarios,legged robots are more flexible and effective than wheeled robots,which makes it gradually becoming a research hotspot in the field of robotics.Gait control is the basis for the normal operation of traditional biped robots,which refer to the robot completes balance control,robust walking,fall recovery and other actions on its own.This process involves multidimensional joint drive,motion control,attitude perception and other issues.In order to control the balance of legged robot,it's necessary to avoid leg and foot collisions during motion planning.The traditional control method begins with the establishment of an accurate mathematical model for the legs,followed by the design of balance criteria based on human gait in order to plan the trajectory of motion at the end of the legs and feet.Finally,the angles of the joints in the legs are determined by the forward and inverse kinematics equations.Traditional legged robot control methods have problems of high computational volume,low robustness and lack of generality.DRL(deep reinforcement learning,DRL)provide ideas to solve this problem DRL can learn the motion control policy from the interaction between the agent and the environment.The agent can explore the action space independently so that DRL can avoid the problems caused by inaccurate mathematical models,and continuously enhance the robustness in training.The aim of this thesis is to use DRL to control the walking movements of biped robots,and improve the generality and robustness of the walking control algorithm for biped robots.First,this thesis introduces the basic concepts and research progress of domestic and foreign biped robot balance control and reinforcement learning algorithms.A vision-based robot imitation system is designed.Experiments prove the deficiencies of traditional control methods and the importance of deep reinforcement learning.Secondly,several classic algorithms in reinforcement learning are discussed in Chapter 3.The development and characteristics of the algorithms are elaborated.The training algorithms and entity transformation in deep reinforcement learning of robots are discussed.Next,the PPO(Proximal Policy Optimization,PPO)algorithm and reality gap bridging training methods are introduced,and the reinforcement learning state action space and reward functions are designed.Then,Chapter 4 describes the progress of designing a MuJoCo(Multi-Joint dynamics with Contact,MuJoCo)simulation environment with reference to the real robot.A high-dimensional continuous space control strategy is designed based on the PPO algorithm.The robot learns to control the balance through interaction with the simulation environment.The virtual robots are tested in MuJoCo to verify the effectiveness of the DRL trained policy.Finally,the physical transformation from the MuJoCo simulation environment to the real Nao robot is completed based on the ROS(Robot Operating System,ROS).The Nao robot sensor and the server computer are integrated into a distributed communication architecture.The robust motion control task on Nao robot in the real environment is achieved via ROS.