The Design And Implementation Of User Personalized Recommendation System Based On Big Data

With the rapid development of the Internet,the continuous generation of high-tech tools has improved people's living standards.Recommendation systems play a vital role in this process,having been widely applied in many online services including social networks and e-commerce websites.Recommendation system is a kind of software tool,which can recommend some items or services that users may be interested in.Its existence allows people to obtain more diversified effective information and services.In addition to social networking and e-commerce,movies,music,books and articles can also be seen everywhere.The continued popularity of recommendation systems benefits from the continuous development and optimization of recommendation technologies.Nevertheless,the existing recommendation systems still face the problems such as cold start,sparsity,and low prediction accuracy.Especially in the era of big data,the traditional recommendation system architecture cannot meet the ever-changing needs of businesses and users.At the same time,there are still problems in scalability.With the continuous development of big data technology,these traditional problems have new solutions,including the Hadoop distributed computing platform and the Spark distributed computing engine.Spark is better at in-memory iterative computing,and it is also the current mainstream big data technology.This paper mainly researches several classic recommendation algorithms.For the problems of cold start,sparseness,and low prediction accuracy,the algorithm is improved.A new recommendation method is proposed and designed and implemented on the Spark cluster.This paper proposes a cache management strategy based on RDD dependencies to improve the efficiency of memory computing resources.Specific research contents are as follows:(1)Aiming at the cold start and data sparseness of latent factor model-based recommendation algorithms,a Latent Factor-Based Matrix Factorization Completion Based Hybrid Weighted Recommendation Method,referred to as the LF-WMC recommended method.The preliminary prediction is made from two aspects of matrix decomposition and matrix completion.At the same time,the neighbor information set of the user item is considered.The above two prediction results are mixed according to the local and global impact of the user item score to obtain a new prediction result.The weighted average of the RMSE of the prediction results effectively alleviates the problems of cold start and data sparsity,and improves the accuracy of the recommended prediction results.(2)Aiming at the computational efficiency of the Spark distributed computing engine,a cache management strategy based on RDD dependencies is proposed.Before the Spark job is executed,the RDD reference counting and execution are introduced based on the dependencies between the RDD Stage internal and the Stage Time,for the cross-stage cache RDD,calculate two kinds of energy consumption to achieve dynamic switch of cache in disk memory,which improves the reuse rate of memory resources.(3)Design and implement a user personalized recommendation prototype system based on big data,and test each functional module,which basically meets the actual needs.