Research On Distributed E-commerce Recommendation System Based On Flink

Faced with the ever-increasing amount of data on the Internet,it is impossible for users to traverse the entire Internet in order to find certain information.The recommendation system is one of the ways to solve this problem.It has outstanding performance in news,music,movies,short videos and other fields,and has brought huge commercial value to related enterprises.However,since there are many factors that affect user decision-making in e-commerce websites,related recommendation algorithms still have great research value and practical significance.The main work of this paper is as follows:(1)This paper designs a recommendation framework that combines offline batch training and real-time incremental update with the help of Flink's batch-flow unified feature,and can provide real-time recommendation services.(2)In the traditional offline recommendation algorithm,after the user scores in real time,because a piece of data has little effect on the entire data set,even if the recommendation model is updated,the recommended results are similar to the results before the user scoring,and they cannot focus on the user Short-term interest.This paper proposes a real-time recommendation algorithm.The user's recent ratings are considered,so that the recommended results can cater to the temporal locality of user interests.(3)Since traditional matrix decomposition methods are usually based on static data sets,for the arrival of new data,they must be integrated into historical data sets and retrained to update the model,which is difficult to adapt to the data stream environment and cannot meet real-time requirements.In this paper,the traditional matrix factorization model is improved by data partition parallelization and online incremental update.Construct InNode and OutNode information to partition the data,reduce the dependence of the data between nodes,and greatly reduce the communication cost in the operation of the algorithm;in accordance with the characteristics of the e-commerce website,the traditional ALS iteration method is improved to adapt it to the streaming data environment Incremental updates in.(4)This article improves the current mainstream distributed big data platform architecture,designs and builds a distributed cluster,and then,in order to meet the various recommendation needs of e-commerce websites,implement a distributed e-commerce hybrid recommendation system based on Flink.Experiments show that the various algorithms in this paper perform well in many aspects such as novelty,communication complexity,and running time while ensuring a certain prediction accuracy.