Research On Basis Function Construction Methods In Reinforcement Learning

Reinforcement learning is a kind of learning methods, mapping from environmental states to actions. Many practical problems can be described as reinforcement learning problems, thus, reinforcement learning has a broad application prospect. However, the state space of practical problems is always very large or even continuous, which makes reinforcement learning methods inevitably encounter the ”curse of dimensionality” problem. Aiming at solving this problem in large scale or continuous problems, this paper puts forward several new basis function construction methods. The main research is outlined as follows:(i) In order to address the problem of the existence of unnecessary split in adaptive Tile-Coding method which leads the storage space growth and learning rate slowing down, we propose a mergeable adaptive Tile-Coding method. This method can merge two adjacent regions according to a certain threshold condition. This method eliminates the adverse effects resulting from those unnecessary splits. It can not only solve the ”curse of dimensionality” problem by further reducing the storage space, but also improve the learning efficiency.(ii) With respect to the problem of too large sample size and sampling when using batch reinforcement learning in problems with large scale state space, this paper proposes an exploring tree based fitted Q iteration method which is suitable for problems with large scale state space. This method extends the traditional fitted Q iteration method with an exploring tree which drives the agent to do local sampling before each iteration according to the convergence and density of samples of each region. This method makes the convergence result more stable. Moreover, it improves the efficiency of samples and avoids the problem of over sampling which reduces the amount of calculation.(iii) In view of that the traditional function approximation methods are difficult to apply to the problems with high dimensions, this paper puts forward a new kernel based basis function construction method which do not establish kernels in the whole dimensional feature space but in a relatively low dimensional feature space. This method solves the problem that traditional function approximation methods represent poor generalization ability and approximation result. It improves the generalization ability and the basis function approximation accuracy.