The Research And Implementation On Ranking-oriented Distributed Collaborative Filtering Algorithm

In big data applications,the problems of personalized recommendation have been widely concerned by the industry and academia.In the current field of the personalized recommendation,the problems of parallelization and hybrid recommendation have always been a hot research topic.With the advent of the information age,the data volume of all walks of life is exploding,the traditional stand-alone recommendation algorithm can not solve the problem of the big data,the design and implementation of the scalable parallel recommendation algorithm is imminent.At the same time,the recommendation system based on single index can not meet the diversity requirements of users,and therefore can not guarantee the quality of recommendation.Hybrid recommendation systms based on multi-criteria are becoming the focus of research.In order to study the parallelization problem of the recommendation algorithm,this paper propose a sorting-based distributed collaborative filtering algorithm named DistCofiRank.Firstly,the parallelization scheme of DistCofiRank algorithm is designed to minimize the traffic between the computing nodes which guarantees the parallel running performance of the algorithm.Then,this paper analyzes theoretically the optimization of the objective function of the DistCofiRank algorithm,that is,BMRM algorithm.When the BMRM algorithm is used to solve the objective function of the DistCofiRank algorithm,the Hungarian distribution algorithm is used to solve the subgradient of the objective function,and the projection gradient method is used to update the learning parameters of the algorithm.At the same time,according to the parallelization design of the DistCofiRank algorithm,this paper designs and implements the DistCofiRank algorithm based on the Spark platform,and finishs four modules of the DistCofiRank algorithm: training module,prediction module,model saving and loading module and evaluation module.Experiments show that the algorithm has better parallelism and expansibility as the speedup of the parallelization algorithm DistCofiRank is approaching to the linear acceleration ratio with the increase of the computation of the algorithm.Meanwhile,the experiments show that the DistCofiRank algorithm has a good evaluation results for the test set.Because of the recommendation list provided by the recommendation algorithm of single evaluation index can not meet the diversity needs of users,this paper proposes the hybrid recommendation model of ALS and DistCofiRank based on the PredictionIO framework.The ALS algorithm is based on the score of the collaborative filtering algorithm,this algorithm can provide users with a good list of recommendation,but not the highest correlation with the user items at the top of the list,but the DistCofiRank algorithm is based on the sort of collaborative filtering algorithm,which can solve the shortcomings of the ALS algorithm.In this paper,ALS algorithm and DistCofiRank algorithm are combined with hierarchical blending strategy to guarantee users a good recommendation list,and the items with the highest relevance are ranked at the top of the list.Based on this hybrid recommendation model,this paper also designs and implements a movie recommendation system to display the recommendation effect of the algorithm in a visual way.This algorithm engine of the recommended part of the movie recommendation system is free to switch,which used to research and contrast the recommendation algorithms.