Design And Implement Of The Item Recommendation System Based On Spark

With the development of information technology and the amount of data has exploded society come to enter the era of Big Data.At the same time,it is becoming more and more difficult to quickly and effectively find the information users need in the massive data.Recommendation system is a type of information retrieval system,which can effectively solve this problem.For users who cannot express their information needs well,the recommendation system filters information by analyzing user dynamic and static data,so as to show users what they potentially need more proactively and intelligently.This function makes recommendation systems play an important role in e-commerce,social networking and other fields.In recent years,the research of technology promoted the emergence and landing of big data processing frameworks.At present,the mature frameworks used in the industry include Hadoop,Spark and Flink.Among them,Spark's computing based on memory makes it stand out in iterative computing and has become a research hotspot in big data computing.It has good distributed characteristics and fault tolerance,so it can significantly improve the performance of the recommendation system.This paper mainly studies the design and implementation of item recommendation system based on Spark memory computing model.The main research contents of this article are as follows:(1)Analyze and implement several recommendation algorithms mainly used in the industry,including their principles and characteristics in detail.The algorithm includes collaborative filtering algorithm based on items,recommendation algorithm based on matrix factorization and recommendation algorithm based on ALS.(2)Realize a data warehouse,then clean and classify original log data,so to realize more concise and efficient access to the system.The data warehouse makes the architecture of the system clearer,and effectively improves the computational efficiency of the algorithm.(3)Implemente three sets of recommendation algorithms based on the Spark memory computing model.Analysis user behavior logs on online by the Flink stream processing framework,which has the real-time advantages.Finally improve the real-time performance of the system.(4)According to the above,provide recommendation service by offline recommendation and online recommendation,which improves the integrity and practicability of the system.Then display the movie system through angularjs Vue.Finally implement the ABTest test system for algorithm version choosing.All in all,the system builds an integrated movie recommendation system from user interaction,back-end construction,and model scheduling.