Reinforcement Learning And Its Applications In Navigation And Control Of Mobile Robots

In recent years,reinforcement learning has become one of the key research areas in artificial intelligence and machine learning and it has attracted many researchers in other fields including operations research,control theory and robotics. Reinforcement learning is different from supervised learning in that no teacher signals are needed and a reinforcement learning system learns by interacting with the environment to maximize the evaluative feedback from the environment. Thus,reinforcement learning methods have wide application areas in solving complex optimization and decision problems,where teacher signals are hard to be obtained.Since mobile robots will be widely applied in industry,transportation,construction and aerospace in the 21st century,much higher requirements have been placed on the intelligent navigation technologies for mobile robots. To improve the ability for autonomous navigation and the adaptiveness to the environments is the key problem for the applications of mobile robots in complex,unknown environments. Making use of machine learning methods,especially reinforcement learning methods,to realize the optimization and the adaptiveness of the autonomous navigation controller for mobile robots becomes an important research topic not only in robotics but also in artificial intelligence.Supported by the National Natural Science Foundation of China (NSFC)2,the research topic of this paper has been focused on reinforcement learning and its applications in mobile robot navigation. One part of the main contents in this paper is the generalization methods for reinforcement learning in solving Markov decision problems with continuous states and actions. Another part of the main contents is the applications of reinforcement learning methods in the optimization of the path tracking controllers and the autonomous navigation controllers for mobile robots.In the first chapter of this paper,a comprehensive survey on the research of reinforcement learning algorithms,theory and applications is provided. The recent developments and future directions for mobile robot navigation are also discussed. Based on the above analysis,the research topic of this paper has been focused on 5 parts as follows:1) the algorithms and theory of temporal difference learning;2) gradient learning algorithms for solving Markov decision problems with continuous state or action space;3) hybrid learning methods for solving Markov decision problems;4) the applications of reinforcement learning in the path tracking problems of mobile robots;5) reactive navigation methods based on reinforcement learning for mobile robots in unknown environments.The main contributions of this paper include:1. In the research of the algorithms and theory of temporal difference learning,a new class of multi-step learning prediction algorithms based on linear function approximators andrecursive least squares methods is proposed,which are called the RLS-TD(t) learning algorithm. The convergence with probability one of the RLS-TD(t) algorithm is proved for ergodic Markov chains,and the conditions for convergence are analyzed. In the RLS-TD(t) learning algorithm,the eligibility traces mechanism and the recursive least squares methods are combined together so that better convergence properties can be obtained in learning prediction problems.2. Reinforcement learning algorithms that use Cerebellar Model Articulation Controller (CMAC) are studied to estimate the optimal value function of Markov decision processes (MDPs) with continuous states and discrete actions. The state discretization for MDPs using Sarsa-learning algorithms based on CMAC networks and direct gradient rules is analyzed. Two new coding methods for CMAC neural networks are proposed so that the learning efficiency of CMAC-based direct gradient learning algorithms can be improved. The two new coding methods are called non-disjoint over-lapping coding and multi-resolution coding,respectively,and their effectiveness is illustrated in the learning control simulation of the c...