A Real-time Recommendation System Based On Stream Computing

Collaborative filtering recommendation algorithm based on items is widely used in e-commerce,advertising,news,music,video and other occasions,but still accompanied by problems such as unpredictable accuracy and cold start.At the same time,as technology advances and the interaction between people and devices increases,user-generated data is more dynamic,which results short-term validation of a large amount of information,which needs to be processed in time.The traditional recommendation system analyzes data and updates the model at regular intervals,and cannot meet the user's requirements in real-time.To solve these problems,this thesis proposes the following solutions:(1)Type-similarity is introduced in this thesis for the problem of cold start of items based on item-based collaborative filtering recommendation algorithm and low prediction accuracy under the scenario of scoring data sparsity.When calculating the similarity of items,the similarity of item types and the similarity degree of collaborative filtering are combined to reduce the negative effect of data sparse on similarity calculation.(2)To solve the problem of poor real-time performance of traditional item-based collaborative filtering algorithms,an item-based real-time recommendation algorithm is proposed.The algorithm consists of two processes: similarity calculation and recommendation priority calculation,and similarity calculation using off-line calculation method.The priority uses the online calculation method,and at the same time,in order to reflect the dynamic change of the user's interest,a time factor is added to the recommendation priority calculation.(3)To solve the problem that the traditional recommendation system is slow in calculation and cannot be recommended according to the user's real-time behavior.This thesis designs and implements a real-time recommendation system based on flow calculation.The system uses Flume to collect logs,Kafka to buffer messages,Spark Streaming to process streaming data in real-time,and Redis and Mongo DB to store data.Combining off-line processing with on-line processing,the offline processing part completes the computation with high complexity and large amount of computation,and the online processing part completes some lightweight calculations,so that the recommendation system can respond to the user's behavior in real-time.In this thesis,the accuracy and performance of the designed real-time recommendation system are tested through experiments.Experiments show that the real-time recommendation system based on Spark Streaming has good performance.