Cooperation Models For The Multi-agent System And Application To The RoboCup Soccer Simulator

Over the last few years, a multi-agent system (MAS) has been the subject of controversy in the field of distributed artificial intelligence. The RoboCup soccer simulation league was established and has been used as a standard platform to test the proposed various MAS theories. Robot soccer contains an extremely complicated environment. In such surroundings, agents have to cooperate to achieve the objective of making goals as much as possible and winning the match.This dissertation contributes to the design of the RoboCup simulation team. Cooperation tactics and models of MAS have been built by integrating the technologies of planning, studying and prediction synthetically. The main achievements are as follows.To solve the cooperation problem of a RoboCup soccer simulation team, a dual MAS cooperative-model architecture, which is composed of a cooperation strategy and an action decision, is proposed. It reinforces the intelligence of the whole system as well as the dynamic real-time cooperation among the agents.To obtain a quick real-time response of the MAS, a planning and cooperation model is established based on the state of the system. The proposed model not only increases the reaction speed of the individual agents, but also improves the cooperation efficiency of the MAS. From attacking point of view in the soccer game, a ball-passing cooperation strategy is proposed based on a cooperation-desire matrix. It enables the agents in the MAS to have an obvious cooperation, which is independent of communication.Agents are linked into a group to achieve a common goal through formation. The allocation and standing-location cooperation for a pre-assigned task are carried out by introducing the concept of role. From the defend point of view in the soccer game, these strategies are performed to achieve dynamic defensive cooperation based on the formation switching. In order to break through the limitation of the human experience-based formation setup, the case-based learning is applied to the formation design and implemented the dynamic formation switching between active and passive defense. Therefore, the defenserequirement at different stages is satisfied, and the total defense performance of the team is notably improved.A dynamic defensive cooperation based on an affined-degree model is proposed. When the marking strategy is adopted, an agent calculates its corresponding affined degree and confirms whether or not to assist a teammate to accomplish a man-to-man marking defence task. When an area-based defensive strategy is adopted, the main leader in the area is selected by formation and the vice-leader is chosen based on the affined degree. They complete the defense task together. Adoption of the affined -degree model yields efficient cooperation. Thus, it solves the problems of omitting of marked-object and lack of player at the sides of the area caused by marking failure, and achieves a good combination between the division and cooperation.To improve the global cooperation capability of the simulation team, a behaviour-based predicting method is proposed. This method simplifies the design of cooperation model, and increases the response speed, flexibility and intelligence of agents. The implementation of the behaviour predicting based cooperation model in CSU_YunLu team shows that the cooperation decision, such as passing a ball and base line passing are carried out successfully.A statistic based Q-learning algorithm for multi agents is proposed by combining the statistic learning and Q-learning. An agent learns action policies of other agents through perceiving the joint actions. The employment of total probability of policies distribution matrix ensures that the learning agent chooses an optimal action, and guarantees the convergence of the algorithm theoretically. The algorithm reduces the multi agents leaning space from a conventional exponential space to a polynomial space, and improves the learning efficiency greatly. This algorithm has been successfully applied to the off-line training of the cooperation policy in RoboCup.The main differences between reinforced learning and other learning methods are delayed reward and trial-and-error. These two characteristics may cause a temporal credit assignment problem and infinite state visiting problem. In a MAS, these may result in slow convergent, or evenworse, not convergent in the learning. To solve such problems, a predicting based Q-learning model for multi agents is proposed. It has a structure of two levels, and integrates planning, predicting and Q-learning. The efficient on-line learning capability of the proposed model is demonstrated in RoboCup.The validity of the proposed MAS cooperation strategies and models has been demonstrated by the CSU_YunLu simulation team.