Research And Application Of Recommender System Based On Spark Platform

In the age of Big Data,the data volume is growing exponentially.How to pick up the valuable information from numerous and complicated data is a big problem.The recommendation system is a kind of more initiative and intelligent information filtering method to push the data which the users may be interested in by analyzing users' history behavior and the similarity of users or objects.At present,the recommendation system has been widely used in many fields,such as social networking systems,e-commerce and life-support services.The capacity of users interests analysis and meeting users demand accurately is obviously essential for the recommendation system,in addition,finding the valuable information from a bulk of data quickly is equally important.Combining the distributed computing platforms with recommendation system,and making it keep the ability of big data processing and analysis,is of the great importance.Spark is a new distributed computing platform after Hadoop based on internal storage computing,which is more superior on performance and response speed than Map Reduce in view of its iterative parallelization.The main study contents of this paper involve:(1)Established a data warehouse based on Spark,to provide distributed calling interfaces for recommendation engines,and to store the information of users,objects,ratings and off-line calculating results.(2)Developed 3 recommendation engines based on Spark,and divided the system into on-line and off-line according to the calculating complexity and data updating frequency,and conducted a series experiments on recommendation system's performance.This paper puts forward a recommendation system with 3 recommendation engines based on Spark,with a deep theoretical study of Spark,recommendation algorithms and their application scenarios respectively.In the current hardware and software conditions,all of the 3 recommendation engines achieved the expected target,and lay the foundation of other recommendation engines based on Spark.Generally speaking,this paper integrates the theory and practice together,and has certain practical significance.