Experiment And Research Of Recommendation System Based On Spark Parallel Framework

Recently the rapid development of Internet technology and popularity,people have a choice of ways to access information online more and more,such as getting news and information through online portals or mobile apps,shopping online,and more.Along with the increase of various information websites and mobile applications,the information on the Internet is being exploded.The information on the one hand meets people's demand for personalization and intelligence,on the other hand,people face severe information overload problem.The recommendation system,as an effective way to overcome the overload of information,can proactively push information that people may be interested in and save time spent on information screening.The recommendation algorithm,which is the core of the recommendation system,determines the accuracy of the recommendation system.In order to provide the accuracy of the recommended system,this paper presents a recommendation system based on the Spark parallel framework,which is a parallel recommendation system based on the GBDT hybrid algorithm.The research contents of this thesis include: 1)Comparison verification of the proposed algorithm based on collaborative filtering: This paper verifies the traditional collaborative filtering algorithm and analyzes different collaborative filtering Algorithm principle,contrast the advantages and disadvantages of different collaborative filtering algorithm.2)Hybrid recommendation algorithm based on GBDT: this paper proposes a hybrid recommendation algorithm based on GBDT to overcome the shortcomings of traditional collaborative filtering algorithm and improve the recommendation accuracy of recommendation algorithm.Compared with other model integration strategies,such as linear weighted,fusion,stochastic forest and other integrated algorithms,the advantages of hybrid recommendation algorithm based on GBDT are described.3)Spark-based parallel architecture recommended algorithm: Utilize the high performance and ease of use of Spark to achieve the parallelization of GBDT-based hybrid recommendation algorithm to improve the execution efficiency and recommendation of the proposed hybrid recommendation algorithm based on GBDT.GBDT-based hybrid recommendation algorithm faces performance bottlenecks and other issues.The experimental results show that the proposed system based on Spark parallel framework proposed in this paper can efficiently and accurately recommend users to the information.On the one hand,the hybrid recommendation algorithm based on GBDT can effectively improve the accuracy of the proposed algorithm.On the other hand,using Spark to implement the parallel recommendation system can improve the running speed and the concurrent performance of the algorithm under massive data.