Design And Implementation Of Combinatorial Recommendation System Based On Distributed Platform

Nowadays,with the rapid development of the Internet technology,there has been an explosive growth trend of the Internet data,and the information-overloading problems have become increasingly prominent.In order to allow users to obtain the information they need quickly and efficiently,and also enable enterprises to discover users' interest preference effectively from the massive information,recommendation algorithms come into being.The traditional single recommendation algorithms have achieved personalized recommendation or social recommendation to a certain extent,but they still face the problems of cold start,data sparsity,system scalability and so on.The distributed processing platforms represented by Hadoop and Spark can not only optimize the resource utilization of storage and computing of each computer node,but also achieve parallel computing while maintaining high reliability,high availability and data consistency.Benefited from these advantages,these distributed processing platforms provide a new solution for large-scale data processing.Based on the current status of the recommendation algorithms,this research combined a variety of signal recommendation algorithms at different levels,and designed a combinatorial recommendation algorithm.Guided by the strategy of mid-combination,the content based recommendation algorithms and the collaborative filtering recommendation algorithms were combined to design a user based mid-combination recommendation algorithm and an item based mid-combination recommendation algorithm,which solved the the problem of user cold start and item cold start respectively.In accordance with the strategy of post-combination,both mid-combination recommendation algorithms were used to form a user-item based post-combination recommendation algorithm.While solving the problem of user cold start and item cold start,the algorithm also maintained the personalized and social features of collaborative filtering recommendation.To solve the problem of data sparsity and further improve the accuracy of recommendation,the user-item based post-combination recommendation algorithm and the recommendation algorithm based on alternating least squares were combined to design a cascade-combination recommendation algorithm based on alternating least squares.Finally,to adapted to the processing of large-scale data sets,the combinatorial recommendation algorithm designed in this research was deployed to the distributed processing platform.The innovation of this research is as follows:1)A dynamic linear weighting method for matrices was proposed,which calculated the weighting coefficients according to the prediction error of single recommendation algorithms relevant,thus solved the cold start problem optimally.2)A solution for data sparsity was proposed,filling the missing values in the original data with the preliminary prediction results.The test results show that the cascade-combination recommendation algorithm based on alternating least squares designed in this research can solve the problems of user cold start,item cold start and data sparsity,and also improve the accuracy of recommendation.Besides,when deployed to the distributed processing platform,it also makes the system scalable.