| Deep reinforcement learning has always been a long-term challenge in the field of artificial intelligence.Due to the sparse rewards obtained by traditional deep reinforcement learning in some high-dimensional spaces and complex task scenarios,the design of reward functions is complex,and the learning effect is often not good.Therefore,learning from the demonstration data of human experts has become a new method in deep reinforcement learning,and the mainstream method is deep reinforcement learning based on generative adversarial networks,also called generative adversarial imitation learning.Learning from expert demonstrations assumes that the expert’s demonstration data is generated by a perfect optimal policy,but this assumption is limited in real problems,and collecting expert data is difficult and expensive,which greatly limits the data scale and learning efficiency.This paper focuses on the problem of learning optimal policy from imperfect demonstrations.Through the research and application of deep reinforcement learning based on generative adversarial networks,we can make full use of a large number of imperfect data obtained in the real world to solve the problem of poor learning effect caused by sparse rewards and difficult design of reward functions in high-dimensional space and complex task scenarios of deep reinforcement learning.The main content of this article is as follows:(1)Aiming at the problem that imperfect demonstrations need to be labeled in previous work,this paper proposes a method using automatic reweighted confidence,Reweight Confidence Adversarial Imitation Learning(RWCIL),to learn the optimality information of imperfect demonstrations.By combining CAIL’s bi-level optimization model,our method can calculate the confidence in an automatic reweighted manner,and by redesigning the outer loop of the bi-level optimization model,it can automatically learn and optimize the confidence score of demonstrations during the learning process.No manual labeling of data is required.We conduct experiments in the MuJoCo environment and show that our algorithm performs on par with previous work without requiring manual data annotations.(2)Aiming at the problem of learning from imperfect demonstrations under different dynamics,this paper proposes a cross-domain imitation learning method called Fused Gromov-Wasserstein Imitation Learning(FGWIL)based on fused Gromov-Wasserstein distance.FGWIL can convert and learn from the demonstration’s distribution in similar scenes by calculating the optimal transport distance based on the fused GromovWasserstein distance of different state-action metric space distributions.We conduct experiments on the MuJoCo environment and show that our algorithm is effective in cross-domain learning from imperfect demonstrations under different dynamics.(3)Aiming at the problem that the decision-making system in dynamic scenarios cannot adapt,we developed a dynamic decision-making system based on deep reinforcement learning.Through the reinforcement learning dynamic migration model based on generative adversarial networks,the dynamic decision-making system is developed based on the research points(1)and(2)to ensure that the model has good decisionmaking ability under different dynamics. |
