Design And Implementation Of News Recommendation System Based On Spark Framework

With the increasing popularity of smart phones and tablet computers,everyone will come into contact with some Internet products.In the process of daily use,a large number of behavior logs will be generated directly.Internet practitioners and technicians can get some users' interests and preferences by analyzing these behavior logs,which greatly improves the effect of log recommendation,However,every user's operation on the Internet system will directly generate a line of behavior log,which leads to the rapid growth of the data scale of the behavior log,and the data can not be collected and processed in time.In the early personalized recommendation system,the historical behavior information of each user should be collected offline,and then the data should be processed and analyzed regularly,According to the data after cleaning,select the model to predict or update the model.This way of recommendation has delay,so it can't track the user's interest in time,which leads to the decrease of the user's retention rate in the product,or even abandonment of the product.After analyzing and comparing the advantages and disadvantages of various big data processing frameworks,combining the use scenarios,this paper finally chooses the Spark framework as the basis to design and implement a real-time recommendation system.This system collects and analyzes the log data generated by users in real time,and then combines the offline part.The training results can be used for real-time recommendation feedback.According to the characteristics of recommended items,this paper designs a system architecture combining offline computing and real-time computing.The offline computing part adopts the combination of ALS algorithm and content-based recommendation algorithm supported by spark framework to ensure the accuracy of recommendation results.At the same time,the ALS algorithm is improved on spark framework,The gradient descent part of ALS algorithm is optimized.The real-time computing part uses the spark streaming stream processing technology on the spark platform to process the log information collected by the log collection framework flume,and incrementally update the user profile and popular content.In addition to the recommendation part,it also realizes the normal functional requirements of the system,and designs the interface.Ordinary users can browse the news content,change the basic information,add the role of system administrator,and manage the users and content manually,which makes the system experience better.