Design And Implementation Of Key Algorithms For Large Scale Online Content Recommendation System

With the rapid development of information technology,especially Mobile Internet technology,the information on the Internet has been growing exponentially.Therefore,users are facing the challenge of looking for interesting contents from massive data.As a solution to this challenge,recommendation system is used frequently to provide personalized services to users.This paper targets to design an online recommendation system for Yiche Application and optimize the key recommendation algorithms to be used in recommendation system.?To enable this online recommendation system appropriate for millions of users and hundreds of thousands of content resources of Yiche Application,there are three challenges.First,the recommendation system needs to meet online real-time requirement and accuracy of recommendation result in order to improve user experience.Second,the recommendation system needs to be flexible enough to facilitate the expansion of recommended content and different scenarios.Third,the recommendation system needs a sufficient extensibility to support efficient updating of the recommendation algorithm.To meet these challenges,this paper designs and realizes an online recommendation system which is composed of a data processing layer,a candidate generation layer and a sorting layer.Furthermore,the recommendation algorithm is optimized by improving candidate set trigger strategy and ranking model.The main contributions of this paper list as follows.(1)In terms of system design,we use a hierarchical design method to decouple the online recommendation system into three parts:data processing layer,candidate generation layer and sorting layer to improve its extensibility.In the recall layer,we implement a content-based recommendation algorithm,a tag-based recommendation algorithm,a recommendation algorithm based on association rules and a recommendation algorithm based on hotspots to generate a candidate set of recommended results to improve recommendation effect.In the sorting layer,the candidate contents are sorted using a machine learning model to obtain more accurate recommendations.(2)In terms of candidate set generation,this paper optimizes content-based and hot-based recommendation algorithms.1)A recommendation algorithm(CB_time_delay)using the timeliness of content for content similarity calculation is proposed.Online A/B testing shows that the click rate of articles and videos is 30%higher than the original content-based recommendation algorithm.2)This paper proposes a measurement index to measure the content popularity,and implements a hotspot-based recommendation algorithm using the measurement index.Online A/B testing shows that the click rate of articles and videos is 9.5%higher than the original hotspot-based algorithm.(3)In terms of content sorting,we compare AUC and computational complexity of Logistic Regression,GBDT and Wide&Deep offline,the result shows that the Logical Regression model with L1 regularization is more accurate and sparse than the other two algorithms.Finally,the L1 regularized Logical Regression model is chosen as the online ranking model.The system we developed has been deployed practically.The new recommendation system has raised per capita amount of reading by 1.7 times and doubled the click rate of articles and videos.