Research On Knowledge Graph Multi-hop Reasoning Methods Based On Reinforcement Learning

Knowledge graphs describe real-world knowledge in the form of structured directed graphs.In recent years,knowledge graphs have been widely adopted in plenty of areas,such as question answering,information retrieval,recommender systems,machine reading comprehension,and conversation generation.However,there are a large number of missing facts in commonly used large-scale knowledge graphs.The incompleteness problem faced by knowledge graphs seriously hinders their application ability in related downstream tasks.To complete the knowledge graphs,the reinforcement learning-based reasoning method performs multi-hop reasoning over the knowledge graphs to predict missing facts.Due to its desirable effectiveness and interpretability,it has become a research topic with unique advantages.However,current reinforcement learning-based multi-hop reasoning methods treat all the arrival paths equally,ignoring the distinction of semantic validity for different paths.Furthermore,the agent obtains rewards to update its policy only when it successfully reaches the target entity after a multi-step exploration.The severely sparse reward signals are usually insufficient to encourage a sophisticated reinforcement learning model to work well.Based on the above issues,the main work of this paper is as follows:(1)A counterfactual soft reward-based knowledge graph multi-hop reasoning method is proposed to address the issue of not distinguishing the semantic validity of the reached paths in current approaches.The method first learns a semantic-aware relation reasoner,which predicts relation links between entity pairs based on the set of the paths between them.The reasoning is based on the measure of their semantic relevance.Then in the reinforcement learning framework,the relation reasoner is used to construct a counterfactual relation reasoning task.The semantic contribution of a certain path to the reasoning relation is measured according to the impact of removing the path on the relation reasoning task,and further quantified as the counterfactual soft reward of the path.The reward is used to guide the agent to find higher-quality paths.The experimental results show that the method proposed in this paper can achieve superior prediction performance and further enhance the interpretability of the knowledge graph multi-hop reasoning method.(2)A intrinsic curiosity reward-based knowledge graph multi-hop reasoning method is proposed,which introduces the curiosity mechanism to alleviate the reward sparsity problem.In the reinforcement learning framework,the intrinsic reward signal is designed based on the prediction error of the agent's knowledge of its environment.The agent predicts the next state based on the current state and the action it has taken.The intrinsic curiosity reward is formulated as the error between the predicted state and the true state,which is used to drive the agent to explore the environment more thoroughly.To verify the effectiveness of the model,experiments are conducted on three benchmark datasets.The experimental results show that the intrinsic curiosity reward can motivate the agent to find richer paths and improve the reasoning performance.