A Deep Reinforcement Learning Based Dynamic Recommender System

Recommender system is the main tool to deal with information overload problems in industry and academia.It helps users to find what they are interested in from the massive data by analyzing the relationship between users and recommended items,or analyzing the existing historical behavior records of users.Recent decades,recommender system technology has made great progress.Collaborative filtering based,machine learning based and deep learning based models have been widely used in practical application.However,these models are limited by their expression ability,and each recommender process is based on a fixed strategy and fail to adapt to the changing dynamic interest.Secondly,these static models regard each recommendation as an independent process,without considering the continuity of user behavior,and cannot make full use of information.The recommender process can be essentially regarded as the process of interaction with users,so its distinctive feature is interactivity.Reinforcement learning is good at modeling this process.Deep reinforcement learning technology is widely used in the fields of robot control and natural language processing because of its good expression ability and decision-making ability.In recent years,researchers combined deep reinforcement learning with recommender system to build some dynamic recommendation models that can continuously optimize the recommender strategy in the process of continuous interaction with users.These models overcome the shortcomings of the static recommendation algorithm which fail to deal with the dynamic interest changes of users.But they still face the problems of unstable training and low sample efficiency.This paper focuses on the dynamic recommender system based on deep reinforcement learning,the main works is given as follows:1.A stable and dynamic recommender method based on Soft Actor-Critic architecture is proposed.Firstly,based on the Markov decision process of user's long-term and short-term interest,the reinforcement learning algorithm can accumulate user's long-term and stable interest preference according to user's high reward feedback behavior.On this basis,combined with recurrent neural network,two encoders are designed to obtain the feature expression of short-term and long-term interest respectively.Finally,the whole framework is designed based on the Actor-Critic architecture of reinforcement learning,and the robust deep reinforcement learning algorithm Soft Actor-Critic is introduced to train the model.Compared with other models on offline and online experiments,the proposed method has better stability and higher accuracy in recommender scenarios.2.A dynamic recommender algorithm based on Dyna framework and graph convolution network is proposed.Deep reinforcement learning algorithm often faces the problems of low sample efficiency and sparse user feedback when applied in recommender system.As auxiliary information,knowledge graph can provide rich semantic information,which can effectively alleviate the problem of sparse user feedback.This method uses knowledge graph to model the Markov decision-making process of dynamic recommender system,and combines graph neural network to construct the recommender agent of reinforcement learning,so as to mine the correlation between items in knowledge graph and alleviate the problem of low sample efficiency.In order to reduce the frequency of real interaction with users,the Dyna framework of reinforcement learning is introduced to further improve the sample efficiency.Through online simulation experiments,it is proved that the proposed method can obtain better performance in the face of user feedback is sparse.3.A multi-agent reinforcement learning based dynamic recommendation algorithm is proposed.In this method,the recommendation is divided into user based and group based Markov decision processes,and DQN algorithm in deep reinforcement learning are used to model them.By two DQN agents,the global DQN grasps the changes of real-time hot spots,which controls the current trend;the local DQN records the changes of users' personal interests,which obtains the current users' personalized preferences.The experimental results in the user cold start environment verify that the proposed method can effectively improve the recommender accuracy.