Research And Design Of Real-time Recommendation System Based On Spark

Under the background of big data era,people are facing the problem of information overload.The indispensable tool to solve this problem is recommendation system.Recommender system helps users filter out information that does not meet their preferences by analyzing user behavior data,and provides the user with information that may be needed.Therefore,the recommendation system plays an important role in various fields such as e-commerce,social networking and music websites.Traditional recommendation systems often update the recommendation results by regular calculations,resulting in a loss of recommendation accuracy.Therefore,it is very important to improve the real-time performance of the recommendation system.Spark introduces the Resilient Distributed Datasets data model and memory-based computing model,and Spark Streaming has the ability of real-time flow calculation.Therefore,the application of Spark to the recommendation system will greatly improve the efficiency and real-time performance of the system.This paper first introduces the background of the topic,then introduces the related technologies of big data,and deeply discusses the related theories of recommendation systems and recommendation algorithms.Finally,combined with the above related research,this paper aims to design and implement a recommendation system that can real-time perceive the changes of user interest and adjust recommended content in real time according to the changes of user interest.In response to this goal,this paper has done the following work:1.An optimization model of matrix decomposition algorithm based on Spark is proposed.Based on the original matrix decomposition recommendation algorithm,this paper proposes a recommendation algorithm of matrix decomposition and K nearest neighbor fusion based on Spark.The algorithm has a great improvement in computing efficiency and recommendation accuracy.2.Design and implement an efficient data acquisition and data warehouse module.This module effectively provides a solid and reliable data supply,which lays a foundation for the stability of the recommendation engine module.3.Design and implementation of multiple groups of recommendation engines.In order to adapt to a variety of application scenarios,the recommendation engine module contains multiple recommendation engines and is divided into two parts:Online recommendation and offline recommendation.While guaranteeing the real-time nature of the recommendation,the user is presented with a variety of recommended content.4.Design and implementation of a hybrid recommendation model.The model automatically adjusts the weight of each recommendation engine according to the user's choice,and then fuses the results of multiple recommendation engines according to the weight,and finally obtains recommendation content which is more in line with the user's requirements.