| The limb structure of a hexapod robot has redundant degrees of freedom.Its discrete ground support and rich gait give it better flexibility and stability when walking in unstructured and complex environments such as obstacles and ditches.With the emergence and vigorous development of reinforcement learning,its combination with the hexapod robot can enable it to obtain a motion policy through training in a static environment to complete global motion planning.However,due to the change of the state space under the dynamic environment,the trained motion policy is not suitable for the dynamic environment.Specifically,when the environment changes,the policy needs to be retrained.That is,there are many problems,such as low efficiency,poor generalization ability,and inability to meet the needs of task diversity.In a realistic environment,the unstructured environment is more likely to change than the structured environment.In this case,the hexapod robot must adjust its motion policy in time to cope with the dynamic environment.Therefore,this paper takes the hexapod robot as the carrier and carries out the following research on its adaptive reinforcement learning method of quickly adjusting its motion policy in dynamic environments:(1)The reinforcement learning process of the hexapod robot is modeled by combining the kinematics and dynamics of a hexapod robot,enabling the hexapod robot to perform adaptive reinforcement learning in a dynamic environment.At the same time,the changes in dynamic environments are classified,and the plum blossom pile environment is used to characterize the unstructured,complex environment.A series of dynamic environments are designed for experiments based on the plum blossom pile environment.(2)In response to changes in the dynamic environment,this paper designs a method for detecting the applicability of policies,which evaluates the applicability of policies in the changed environment through reward evaluation based on environmental feedback,and evaluates the degree of change in the dynamic environment.This method is used to select appropriate adaptive reinforcement learning methods to complete the movement policy adjustment of hexapod robots in the dynamic environment.(3)The adaptive reinforcement learning method is studied.According to the degree of change in the dynamic environment,an adaptive reinforcement learning method with incremental reinforcement learning method as the core and migration reinforcement learning method and meta reinforcement learning method as the supplement is proposed to achieve rapid adjustment of the motion policy of a hexapod robot in a dynamic environment.(4)Aiming at the problem that current incremental reinforcement learning methods do not consider the degree of environmental change when implementing incremental learning,resulting in inefficient performance of the adjusted policy,this paper proposes a more efficient incremental reinforcement learning method based on the adjustment requirements of hexapod robots for their motion policies in dynamic environments.According to the assessed degree of environmental change,this method uses a network parameter perturbation mechanism to pertinently perturb the learned approximately optimal motion policy,thereby providing a good initialization for the policy in the new environment.Finally,it achieves rapid adjustment of the policy through information weighting.Experiments have shown that the proposed adaptive reinforcement learning method can help hexapod robots adjust their motion policies more efficiently when the environment changes compared to retraining and can be used to complete motion planning in dynamic environments. |
