Walking Stability Control Method Of Humanoid Robot Based On Reinforcement Learning

Excellent walking control strategies can help humanoid robots adapt to diverse workscenarios,and stability control during walking is a difficult and important issue in the study of humanoid robot gait planning.Aiming at the walking stability of humanoid robot,this paper designed an adaptive online stability control strategy based on reinforcement learning,realized real-time stability control during humanoid walking,and developed a robot co-simulation system for experimental verification.The main work of this paper is as follows:(1)Aiming at the problem that humanoid robots are easily affected by road surface so hard to maintain stability when walking in a non-horizontal environment,this paper proposes a biped walking control method that combines offline gait planning and online joint angle compensation.This method is based on the Actor Critic(AC)reinforcement learning algorithm to establish an adaptive control model for a humanoid robot.The control strategy is fitted with a BP neural network.By receiving continuous state information,the end-to-end control of the input signal to the output joint angle compensation is achieved,and the cubic spline interpolation method is used to achieve a smooth transition between actions.In addition,a continuous immediate reward function is designed to improve the convergence speed of the learning process.(2)In view of the bionic characteristics of the humanoid robot and the balance control function of the cerebellum in walking,this paper established a cerebellum-like control model based on the reinforcement learning control theory.The modeling of the cerebellar model is based on the anatomical structure of the cerebellum and the transmission process of nerve electrical signals in the cerebellum.In a modular design,the cerebellar model is divided into four parts: a state encoding module,a cerebellar function module,a motion mapping module,and an inferior olivary feedback module.Combined with the temporal-difference(TD)reinforcement learning algorithm to obtain evaluation information of environmental feedback to adjust the weights of behavior strategies and related synapses,the cerebellar model has the learning ability.Finally,an online stability controller was built with the TD reinforcement learning cerebellar model,which enables it to adaptively adjust the joints of the lower limbs of the humanoid robot on the basis of offline gait planning to improve the stability of walking.(3)In order to improve research efficiency and build more convenient development process,a cross-platform and cross-language robot co-simulation system was built.Throughrelated library functions and application program interfaces,the call of C++ programs to Python scripts and their bidirectional information transfer is realized.And MATLAB Engine was used to realize the real-time communication between MATLAB and Webots.The practice in research and development shows that the simulation system can simultaneously use the advantages of Python,MATLAB and C++ languages to develop robot control programs,which provides a platform for the development and verification of subsequent control algorithms.(4)In the robot co-simulation system,simulation experiments of the AC reinforcement learning humanoid robot control method and the TD reinforcement learning cerebellar model control method proposed in this paper were carried out The test was carried out on a slope with varying inclination angle,and the results showed that both control methods can adjust the posture of the humanoid robot in real time and enhance its stability during walking.In this paper,the humanoid robot gait planning,reinforcement learning and cerebellum model were studied in depth.The stable walking control strategy of humanoid robot was developed,and the walking control system was constructed based on this strategy.This paper promotes the application of reinforcement learning and cerebellar model,and it also improved the development of humanoid robot control strategy.