Design And Implementation Of Recommender System Based On Spark

In the era of big data, people have to face information filtration problem. For those demands that user do not or can not express clearly, recommender system can analysis user information more proactive and intelligent to filter out something they want, this property makes recommender system play a very important role in the field of e-commerce, social network and so on.The character of filtering out vast amounts of information makes recommender system always face a large scale of data. In order to respond to user quickly, recommender system requires big data handling capacity. For now there are many frameworks in the big data processing area, among them Spark is the latest distributed computing framework with a strong big data processing ability. Recommender system will have a significantly performance improve when combining with Spark.The paper first describes relevant background of the arguments, then further explores big data processing technologies, including distributed computing framework Spark, distributed file system HDFS, column-oriented file type Parquet, and also introduces a number of recommender algorithms. With the big data processing technology, we design and implement a recommender system, and elaborate the implementation of its main parts.The main contribution of this paper are:1) Design and implement an efficient data warehouse, which stores origin data and intermediate results of offline computing, this data warehouse makes the performance of offline and online computing greatly improved;2) Implement the parallelization of three recommender algorithms based on Spark programming model, then design and implement three recommender engines. These engines can be well integrated with the underlying data warehouse, while its Spark-based design greatly reduces the online and offline computation;3) Design a hybirdization recommender system that combine the result of each recommender engine and adjust the weight according to the user selection, in order to achieve a more personalized recommendation.