Research On Model Interpretability And Knowledge Injection In Reinforcement Learning

Reinforcement learning,one of the machine learning paradigms that learns from feedback by interaction,has shown great potential for applications in many fields,including gaming AI,personalized recommendation,and autonomous driving.However,achieving high returns requires a large amount of data that consumes lots of computational resources and time,which hinders the large-scale application of reinforcement learning.Utilizing knowledge can significantly reduce data requirements and build robust and explainable intelligent systems,but there are still challenges in applications.On the one hand,it is hard to distill knowledge from the black-box models so that the information can be understood and delivered.On the other hand,integrating existing prior knowledge into the training of the models is difficult but crucial,which can enhance the capabilities of the agent and apply it to a wide range of application scenarios.In this paper,we research reinforcement learning from these two directions,and our work is as follows.1.In this paper,we propose a policy explanation method by model distillation.It explains and verifies the policy by distilling the black-boxed deep neural network model into a high-structured decision tree model.The traditional model distillation is usually a supervised learning task under a stationary data distribution assumption,which is violated in reinforcement learning.Therefore,in this paper,we propose a novel distillation objective.The objective maximizes an approximated cumulative reward and focuses more on disastrous behaviors in critical states,which controls the data shift effect.We evaluate our method on both classical Gym environments and complex scenarios.The empirical results show that the proposed method can preserve a higher cumulative reward than behavior cloning and learn a more consistent policy.Moreover,by examining the extracted rules from the distilled decision trees,we demonstrate that the proposed method delivers reasonable explanations and even reveals the model's intention.2.In this paper,we propose a hybrid programming framework.The framework utilizes functions as a bridge to allow programmers to perform hybrid writing of code and neural network models to solve complex tasks.Compared to instructions or natural languages,the programs have many advantages,such as structured,expressive,and unambiguous,which help inject human knowledge into the models.Specifically,we use the input and output of the function as the input and output of the neural network.The reinforcement programming framework allows setting individual interaction environments for each function and sharing the environment in multiple ones.To simply the complex interactive influnces among networks,we introduce a shared reward network to enable reward redistribution.We conduct experiments on the programming problems and serval sequential decision tasks.Our proposed reinforcement programming can tackle them by coding gracefully,while classical reinforcement learning algorithms cannot.