The Optimization Of Skills And Cooperation With Machine Learning In RoboCup3D

In this paper,we propose two key components,a machine learning based infrastructure for optimizing the behavior of humanoid robots using the high-throughput HTCondor computer cluster system and a cooperation scheme based on formation and role assignment,that are used by our Apollo3 D team at Nanjing University of Posts and Telecommunications in the RoboCup 3D soccer simulation competitions.Both of the two components are designed based on in-depth research on the individual skills and coordination mechanism of the robot team.On the behavior optimization problem,we use the Covariance Matrix Adaptation Evolution Strategy(CMA-ES)to optimize the multidimensional parameters of 5 different types of robots' motions in continuous space.Implementing this algorithm in a reinforcement learning infrastructure,we successfully optimized the kicking of the soccer robots.Facing the difficulties of over fitting in single training task when optimizing the walking skill,we designed a layered learning strategy using multiple subtasks.By using this strategy,enhanced 5 types of robots' walking,turning and dribbling behaviors of the biped robots are finally achieved.On the optimization problem of Multi-agent cooperation,we mainly focous on two points,a strategy positioning machenism based on Simulation Based Strategy Positioning(SBSP)for the optimization of the team formation which is on top of Delanuay triangulation subdicision of the football field and a role assignment machenism based on Markov Decision Process(MDP)and using the Sarsa(?)algorithm based linear function approximation to study the value of the action function in the MDP model,which effectively solves the role assignment problem caused by multiple factors such as different robot type,orientation,speed and distance.Large amounts of experiments have proved that the proposed methods in this paper greatly improve the individual and overall performances of Apollo3 D,especially in the team formation and role assignment.