Research On Real-time Movie Recommender System Based On Spark

With the advent of the Internet era,Web2.0 and the Internet of things technology are developing rapidly,and more and more data are generated every year in the world.How to help people find information that they are interested in from these massive data,and bring more profits for the merchants and achieve the win-win situation of consumers and businesses,is of great practical significance.In order to meet this widespread demand,there are two technologies to solve this problem in the Internet industry,one is the well-known search engine technology,and the other is our daily contact recommendation system technology.Although on the Internet,the application of recommendation system is very wide,there are many algorithms and mature theories,but there are still many problems and challenges,such as: data sparsity,recommendation quality and recommendation real-time,cold start.In order to solve these problems effectively,this paper discusses the two aspects of recommendation quality and recommendation real-time.First,it introduces the relevant background of the theory,and then discusses the related large data processing techniques,including the major data technologies,such as distributed file system(HDFS),memory computing engine(Spark)and message queue(Kafka),and also introduces the related algorithms in the recommendation system,and explores the application of deep learning technology.In the recommendation system,the strong fitting ability of the neural network is used to improve the quality of the recommendation.Finally,a simple real-time movie recommendation system is realized by using the popular data processing technology.The main achievements of this article are as follows:(1)In the recommendation quality problem,the implicit feedback data Lai Jianmo of the user is fully utilized,and the depth learning related technology is used in the recommendation algorithm,and a neural network structure is used to replace the inner product of the eigenvector of the hidden space.The collaborative filtering algorithm is formalized by the neural network model and indirectly from the user's implicit formula.The user's preferences are deduced from the feedback,thus improving the quality of recommendation.Experiments were carried out on the MovieLens dataset to verify the validity of the model,and the influence of hidden layer number and activation function on recommendation quality was further explored.(2)In the aspect of recommending real time,the Lamda physical architecture of the real-time recommendation system is proposed,and then the system is designed.It mainly includes three important modules: one is the data preprocessing module,including the analysis and statistics of the data,the cleaning of the original data,the format required for the calculation method and the storage to the HDFS;The two is model training module,through parameter cross tuning to train the optimal model,and save the model to the distributed file system for subsequent module calls;three is the recommendation module,including the common top-K personalized recommendation,and real-time recommendation.The experimental results show that the proposed neural network model can effectively improve the quality of recommendation.The design of real-time recommendation system architecture also has some practical value,and can effectively solve the two major problems of recommendation quality and real-time.