Research On Behavior Control Of Soccer Robot With Reinforcement Learning

This thesis, based on the problem of a single robot behavior control under the RoboCup MSL(Middle-Size league), focused on reinforcement learning applied to the single mission environment and multi mission environment. Simulation verification and the behavior control were realized preliminarily to improve the robot's autonomous behavior ability on the basis of reinforcement learning in real robot.The thesis carried out the research in the aspect of the single robot behavior control under the single mission environment firstly. This paper proposed a new CMAC model with triangulation for the characteristics of continuous state and strong real-time under the environment of the robot, which was on the basis of traditional CMAC neural network with uniform code. We have studied and did experiments on the interception problem in the robot soccer combined with Sarsa(λ) learning algorithm. The experimental results showed that the new CMAC model was able to take advantage of the generalization performance of CMAC network with the triangulation of the problem's state space; In comparison with traditional CMAC model with uniform code, the new CMAC model can obtain better performance of state estimation on the resolving the problem of large state space. On this basis, the thesis analyzed the problem to reinforcement learning in the application of real robot and made the corresponding improvement to apply the above reinforcement learning algorithm under the real environment of the robot's behavior control for the problem of interception. The results showed that the soccer robot was able to intercept the football successfully in most cases. At the same time, the application of heuristic information in reinforcement learning was discussed and the behavior selection policy withε-Heuristic was proposed. The experimental results showed that the learning efficiency was improved remarkably, that had great significance for the behavior control applied in the complex real environment.The thesis carried out the research in the aspect of the single robot behavior control under the multi mission environment secondly. A hierarchical reinforcement learning framework based on Sarsa(λ) learning algorithm was proposed for the problem of action-level control encountered in the robot soccer, that made use of the research results achieved by hierarchical reinforcement learning. A task was decomposed to high-level subtask and low-level subtask in this framework. High-level agent takes charge for upper planning. Low-level agent chooses corresponding behavior combined with self strategy according to the goal specified by high-level agent. The advantage of this framework is that the task can be decomposed to different levels subtasks, which can be applied to new task environment used as module. The interactions between high-level agent and low-level agent were more discussed for the influence of the task learning. The non-Markov property was proved by simulation experiment brought by the interactions. The influence on the total task by the learning of high-level agent and low-level agent was also analyzed through experiment. The results showed that high-level agent can make progress towards self policy before the low-level agent got the near optimal policy.This thesis carried out application research on reinforcement learning algorithm for the behavior control of single robot, according to actual platform situation in the environment of soccer robot competition. The research better solved relevant issues of theory and application, which promoted related work for reinforcement learning applying in robot intelligent control.