Subgoal discovery for hierarchical reinforcement learning using learned policies

Reinforcement learning has proven to be an effective method for creating intelligent agents in a wide range of applications. However, it suffers from the need for a large number of training episodes, a problem that is especially noticeable in large domains. Although the utility of hierarchy is commonly accepted, there has been relatively little research on autonomously discovering or creating useful hierarchies. A system is desirable that can scale reinforcement learning to complex real-world tasks and autonomously discover hierarchical structures within their learning and control systems.; This thesis introduces a method that allows a reinforcement learning agent to autonomously discover and create hierarchy from a learned policy model. A hierarchy of actions helps to create an abstraction which is an encapsulation of a set of actions into a single higher level action that allows an agent to learn while ignoring details that appear at finer levels. The main idea is to find subgoals in a learned policy model by searching for states that exhibit certain structural properties. These subgoals are used to create hierarchies of actions. The hierarchies of actions help the agent to explore more effectively and accelerate learning in other tasks in the same or similar environments where the same subgoals are useful. It is demonstrated that the hierarchical action sequences created with autonomously discovered subgoals can facilitate learning and enable effective knowledge transfer to related tasks.