Research On Hierarchical Reinforcement Learning Based On Action Space Partitioning

Reinforcement learning(RL)is an important branch of machine learning.It finds a optimal policy for solving problem by interaction of agent and environment.In a practical application,the curse of dimensionality will arise with the growth of the number of state dimensions and seriously affects the efficiency.Hierarchical reinforcement learning(HRL)is an extended form of RL and it can alleviate the curse of dimensionality to some extent.But a hierarchy of MDP is required to be constructed manually in some typical hierarchical reinforment learning methods,such as MAXQ,Option and HAM.If there are not enough prior knowledge,these methods will always produce unsatisfactoryresults.Some automatic hierarchical approaches find subgoals by abstracting and decomposing environmental state,but these methods also produce unsatisfactory results in some environments that have not obvious subgoals.There are two aspect works in this paper:(1)A new automatic hierarchical approach was proposed based on partition of set of basic actions.This approach decomposes set of actions into some subsets,then infers relationship of subsets by analyzing the execution order of actions during the whole Markov decision process.Finally,hierarchy will be constructed by using relationship of subsets.(2)In order to use MAXQ method to find optimal policy in this hierarchy,this paper presents a new subtask termination condition and illustrates how to adjust hierarchy dynamically.The experimental results show that the proposed approach can automactically construct hierarchy and find optimal policy more efficiently.