Research On An Approach Of Hierarchical Reinforcement Learning Based On Option Automatic Generation

Reinforcement learning is an approach that an agent can learn its behaviors through trial-and-error interaction with a dynamic environment. It has been more widely used in practice for its self-learning and online learning capabilities. But reinforcement learning is bedeviled by the curse of dimensionality. Hierarchical reinforcement learning(HRL) was presented to combat the curse of dimensionality, and has made great progresses. The essence of HRL is to decompose the overall task into different levels of subtasks by adding "abstract" mechanism on the basis of reinforcement learning. Each subtask can be solved in smaller scale of the problem space. The acquired strategy of subtask can be reused, thus the speed of solving the problem will be increased. There are several typical works such as Option, HAM, and MAXQ.In this dissertation, a novel approach of hierarchical reinforcement learning, named AOM, is proposed. The theoretical and computational issues in AOM are addressed as well as the rising problem in practice.The main contributions of this dissertation are:(1) The AOM approach for Hierarchical reinforcement learning is presented and its theoretical framework and learning algorithm are discussed. The AOM can generate Option hierarchies automatically based on Ant Colony Clustering algorithm(ACCA), and it utilizes the good ability for online learning of MAXQ. The experimental results verify the superiority of the algorithm.(2) An algorithm for automatic AOM hierarchy based on ACCA is presented. The state space clusters according to ACCA. Then the subtasks are constructed based on the clusterings. The experimental results show that the algorithm solves the problem that the learning performance strongly depends on the divisibility of state space.