Dynamic Real-time Recommender Research For News In Distributed Environment

With the high speed evolution of Internet technology,various Internet applications continue to emerge constantly.Recommender technologies for solving information overload problems have also been applied in every field.For example,recommendation research in news application fields has recently attracted the attention of a wide range of academic researchers.At the same time with the big data era has come,a sharp increase in the amount of application data,the result of which gives more and more prominence to a personalized recommendation technique.Among them,personalized news recommendation plays an important role in helping readers find interesting news from a gigantic amount of news items.Diverse methods related to news recommendation have been proposed to provide readers with personalized suggestions.However,little research focuses on an effective user model that can captures the dynamics of user interests and a real-time recommendation engine in news domain for recommendation.In this paper,we propose a news recommendation framework based on user profiling tree(UP-Tree)technique.Unlike user profile(UP),the essential difference between them is that one is static and the other is dynamic.Meantime,by introducing clustering based multidimensional similarity computation method(Multi-SM),the nearest neighbor selection of UP-Tree is effectively solved and a complete hybrid recommendation strategy combined the advantages of both contentand collaborative-based methods is formed.Finally,based on Spark framework,our system added a real-time processing stream module on the original basis to solve the real-time problem in the case of large amount of recommendation dataset.The system monitors the users' reading behaviors,from which it infers their interest in particular articles and updates the profile accordingly.Results show that the proposed approach is capable of making recommendation accurately and efficiently and are sufficiently flexible to capture the unique properties of news articles.The main works of this paper are as follows:(1)Propose the user co-occurrence matrix multiplier recommendation strategy,and design a user-based distributed collaborative filtering algorithm to solve the bottleneck problem of traditional recommended algorithms in distributed environment.(2)Through natural language processing technology,the content of news articles can be transformed into word vector which can be easily processed.In the light of the particularity of news items,topic is used to associate users and news items so that the user-item is further abstracted into user-topic-item.By combining the training of decision tree and the user profile,an effective storage model UP-Tree that can accurately describe the user information is generated.(3)Based on the user profile tree,a multi-dimensional similarity calculation model based on clustering is designed to solve the cold start problem of user profile,so that the user profile tree itself has dynamic characteristics.(4)Based on distributed platform Spark,Shuffle Streaming model is introduced to implement the above-mentioned proposed model design as a concrete real-time news recommendation system.