Research And Implementation Of Collaborative Filtering Recommendation Algorithm Based On Spark Platform

With the rapid development and wide application of Internet technology, plenty of information comes forth to us. Facing the increasing data, how to obtain required information has become a serious problem. Recommendation system is one of the effective ways to solve this problem, because it can build a model to make personalized recommendation through the analysis of users’ behavior and preference. Among many personalized recommendation systems, collaborative filtering recommendation is one of the most successful and widely used recommendation technology. Nevertheless, many problems are exposed during recommendation in existing collaborative filtering recommendation algorithms, especially when the number of users and items is increasing, it would be a sharp drop in recommendation performance.In this paper, collaborative filtering recommendation algorithms are analyzed and the advantages and disvantages of these algorithms are summarized. Aiming at the shortcomings of these recommendation algorithms, we put forward a new recommendation algorithm based on similarity between items and Slope One. The core of this algorithm is to combine similarity between items and Slope One to predict. First, Slope One is used to predict ratings and fill the ratings matrix to solve data sparseness and to be part of the final results. Second, we use the similarity between items as the weight for predicting. Finally, two part of results are combined through a parameter that can be trained, In order to improve the ability to deal with large amount of data, we have parallelized Slope One, similarity and the improved collaborative filtering recommendation algorithms based on Spark. At the same time, the improved algorithm is also realized on Hadoop to be convenient to compare the running efficiency between Hadoop and Spark.We have run the algorithms on MovieLens dataset many times, and the results show that the new algorithm can improve the performance of the recommendation system. Besides, compared to Hadoop, Spark is more suitable for big data processing.