Multi-Agent Reinforcement Learning Through Weighted Experience Sharing

Reinforcement learning allows to program agents by reward and punishment without specifying how to achieve the task. Multi-agent reinforcement learning is an extension of reinforcement learning concept to multi-agent environments. From an individual agent’s perspective, multi-agent systems differ from single-agent systems most significantly in that the environment’s dynamics can be determined by other agents. When multiple agents learn and, hence, adapt their behavior in parallel, each individual agent faces the difficulty of learning in a non-stationary environment. In addition to the uncertainty that may be inherent in the domain, other agents intentionally affect the environment in unpredictable ways. Thus, all multi-agent systems can be viewed as having dynamic environments. Consequently, convergence guarantees, such as the convergence of the Bellman-style single-agent reinforcement techniques, no longer hold.Research in the area of multi-agent systems is concerned with the effective coordination of autonomous agents to perform tasks so as to achieve high quality overall system performance. Multi-agent coordination challenges include the lack of single point of control, local views of each agent that provide only incomplete information, private goals and solution procedures of the agents, communication asynchrony, dynamic environments and uncertainty. Coordination regimes include teamwork, where the agents cooperate and coordinate to achieve a global team goal, or regimes where the agents are self-interested and try to achieve individual goals. An extreme case is where agents are adversaries and try to fulfill their individual goals even while inflicting costs/harm on one another. This thesis proposes a new concept of multi-agent reinforcement learning based on weighted experience sharing. Under the new concept, each agent benefits from the experience of others, and adds his own experience to the knowledge base. By this means our method guarantees the achievement of the point of convergence in so called dynamic multi-agent environments. We also seek to proof that the use of multi-agents allows the rapid convergence of the learning process. Thus, the more number of agents, the quicker the point of convergence arrives. The extension from single-agent system to a multi-agent system is done by means of the Q-learning algorithm as a combination of two methods; Independent Learners and Cooperative Learning.