Research On Causal Discovery Method Based On Reinforcement Learning

The research and discovery of causality between things is the core problem of data science research,which contains a lot of opportunities for scientific research and exploration and huge commercial value.Therefore,discovering causality from data has become an important research topic in many disciplines.At the same time,causal discovery faces two problems:First,because the size of the search space is a super-exponential order of the number of nodes,it becomes a challenging combinatorial problem to learn reliable directed acyclic graphs from jointly distributed samples;Secondly,the enhanced causal discovery algorithm based on policy gradient has some problems,such as low evaluation index,slow convergence speed and poor convergence effect.To solve the above problems,this paper focuses on the causal discovery method based on reinforcement learning.The main work is as follows:(1)In order to solve the challenging combinational problem of learning reliable directed acyclic graphs from samples of joint distribution,this paper describes the structural learning problem as a real matrix continuous optimization problem,improves the structural constraints,and uses the strategy gradient reinforcement learning algorithm to search for the DAG with the highest reward.Firstly,the encoder model and the single-layer decoder model of the self-attention mechanism are used to explore the causal relationship between the data and output the adjacency matrix of the graph.Secondly,combined with the condition of causal structure constraint,the strategy gradient fusion stochastic optimization method is used to optimize the parameters of the neural network model.Finally,the directed acyclic graph DAG obtained from the output is the reward maximum causal graph to be searched.Through the experimental study on the synthetic data set and the real data set Sachs,it is shown that the enhanced causal discovery method with structural constraint improvement can ensure that the learned causal structure graph is more reliable while avoiding matrix index calculation.(2)Aiming at the problems such as poor evaluation index,slow convergence rate and poor convergence effect,a new method of enhanced causal discovery based on asynchronous strategy is proposed.By using the asynchronous advantage actor-critic algorithm,the gradient of n worker networks is calculated to update the parameters of the global neural network model respectively,so as to ensure its continuous development in a good direction and infinite approximation to the real causal relationship diagram.Through the corresponding experimental research,it is proved that the asynchronous policy reinforcement learning algorithm can effectively solve the problems in the strategy gradient reinforcement causal discovery algorithm,and improve the performance of the reinforcement causal discovery algorithm.(3)Based on the above research methods,continuous observation data of the expression levels of phosphorylated proteins and phospholipids in human immune system were selected as application scenarios,and a causal discovery application software system based on reinforcement learning was designed and developed.Through the operation of the software in the selected practical application scenario,the effectiveness of the proposed method is verified,and the technical basis is provided for its popularization and application.Through the research of structural constraint improvement,asynchronous dominance algorithm and application system development of causal discovery method based on reinforcement learning,new solutions and technical support are provided for causal discovery method based on reinforcement learning.