| Robot motor skill learning is able to obtain desired skills autonomously based on interaction data.Therefore,it can better understand and adapt to uncertain and changing environments compared to traditional approaches,and provide an effective way to enhance the autonomous cognition and decision-making capabilities of robots in household chores,disaster assistance,and military reconnaissance.Reinforcement learning obtains experience data from the interactions with the environment and learns an optimal policy by maximizing cumulative environment rewards,which enables robots to learn motor skills in complex environments.In order to improve sample efficiency,robustness and performance of motor skill learning,it is essential to research state representation learning,exploration and mutual information-regularized policy learning in reinforcement learning.However,there are some unsolved problems in the above-mentioned research domains.Namely,(1)since the learned state representations by existing representation learning methods aren’t Markovian and rely on observation reconstruction,learning a good skill policy requires a large amount of interaction data and training time.(2)existing exploration approaches are prone to capture novel but task-irrelevant state novelty and thus cannot effectively guide the agent to explore the environment.(3)the existing policy learning methods with information regularization just usually ignore compressing and preserving predictive information in a sequence of observations,which causes poor performance.This thesis aims to solve the above problems and improve the sample efficiency,robustness and performance of robot motor skill learning.This thesis carries out research in three corresponding areas: self-supervised state representation learning,highly robust exploration in reinforcement learning,and continuous skill policy learning with information constraints.The main contributions and innovations of this thesis can be summarized as follows:(1)A self-supervised state representation learning based on mutual information maximization and dynamic models.To address the problem that learned state representations aren’t Markovian and rely on reconstruction,a neural network model is built that allows to extract compact representations from high-dimensional observations and estimate mutual information between latent representations.A self-supervised objective for optimizing the constructed model is proposed based on mutual information maximization and dynamical models.Experiment results show that the proposed method significantly improves the sample efficiency of robot motor skill learning.(2)Robust exploration based on self-supervised sequential information bottleneck for reinforcement learning.To solve the problem that constructed intrinsic reward cannot effectively qualify task-relevant state novelty,firstly,an optimization objective based on self-supervised sequential information bottleneck is introduced that learns latent bottleneck variables and a neural network model is built to parametrize stochastic distributions.Then a variational upper bound of the introduced self-supervised sequential information bottleneck objective is derived based on variational inference.Finally,an intrinsic reward function is proposed by capturing the mutual information between learned bottleneck variables for guiding the exploration.Experiment results show that the proposed method improves the robustness to Gaussian white noise and natural backgrounds.(3)A skill policy learning method with the constraint of sequential information bottleneck.To compress and preserve the predictive information in a sequence of states to decrease the information cost,an objective with the constraint of sequential information bottleneck is firstly proposed used for optimization.Then an objective with Lagrangian multiplier and an information-gain reward function are derived for practical optimization.Finally,an optimization method is proposed to jointly optimize the continuous skill policy,the state encoder and the dynamical model.Simulated experimental results show that the proposed method improves the performance of robot motor skill learning.(4)Position control experiment of an end-effector on a mobile manipulation robot.In order to verify the effectiveness of the proposed skill policy learning algorithm,the position control experiment of an end-effector is conducted on the mobile manipulation robot.The experimental results show that the position error of the robot end-effector is small and the motion of each joint axis is smooth after adopting the proposed algorithm,which demonstrates the proposed skill learning algorithm can be applied to handle the practical end-effector position control tasks. |
