Research And Design Of Recommendation System Based On Multi-source Heterogeneous Data

The rapid development of the Internet and the continuous enrichment of online content have prompted people to enter big data era.The rapid growth of data has brought serious "data overload" problems.Users are difficult to get the information they want from the massive data,and it is difficult for the information publishers to find the customers matching the content they publish.Recommendation system emerges as the times require.In addition,with the increasing complexity of software systems and the increasing richness of network content,the data sources and data structures of recommender systems are also more diversified.Online data and offline data coexist,and unstructured data accounts for an increasing proportion of the total data.Recommendation system will face many challenges.In this thesis,a recommendation system which can integrate real-time streaming data and local offline data,structured data and unstructured data is established based on the call-sort recommendation system architecture proposed by You Tube.In the recall model,according to the model updating method of collaborative filtering method based on matrix decomposition technology,this thesis adopts the incremental updating method to improve the efficiency and accuracy of the recommendation model;The sorting model takes into account the input coding mode and data distribution of heterogeneous data,uses convolutional neural network,Attention mechanism and Inception mechanism to fuse all kinds of data.The system integrates the current mainstream real-time data processing components(Fluem,Kafka)and the streaming data computing engine Flink,and utilizes the efficient collaborative filtering recommendation model and the accurate deep learning recommendation model.Various components and algorithms complement each other to improve the performance of the recommender system.The main works of this thesis are as follows:1.Build an incremental and updated recall model based on Flink and matrix decomposition model.This model integrates online incremental data and offline data,and incorporates attention sharing matrix to improve the accuracy and calculation speed of the recall model.2.Build a sorting algorithm based on deep learning.This model uses convolutional neural network,attention mechanism and Inception mechanism to integrate various types of multi-source heterogeneous data,such as text data,discrete data and continuous data,to improve the algorithm accuracy in the sorting stage.3.Based on the micro-service framework,our work uses the log collection component Flume,streaming data computing engine Flink and message queue Kafka to build a real-time,highly fault-tolerant and extensible recommendation system.