Research On Collaborative Filtering Recommendation Algorithm Based On Big Data

With the continuous updating of the Internet technology,the volume and complexity of network data is increasing.It is becoming more and more difficult for users to dig out valuable information resources from large amounts of data,and the "Information Overload" problem is becoming more and more serious.Recommendation algorithm is one of the main technical tools to alleviate the problem of "Information Overload",which has received extensive attention and research from academia and industry.The large amount of data brings great challenges to the traditional data processing platform and technology.The open source distributed computing platform Hadoop provides effective platform support for big data's processing and is widely used.In this paper,the relevant theory of the recommendation algorithm is deeply studied,and the recommendation algorithm based on collaborative filtering,Slope One,is optimized and the parallel implementation under the large data frame is implemented.In view of the shortcomings of the Slope One algorithm,the similarity measurement method is added to the item dimension,the clustering of the user dimension and the neighbor set of the target user are obtained.Then,in order to improve the accuracy of recommendation,we use the idea of ensemble learning to use ensemble clustering instead of a single user clustering method.Finally,parallelization is implemented on Hadoop computing platform.The main work of this paper is as follows:1.In view of the fact that the Slope One algorithm does not fully consider the influence of weight on the user dimension and item dimension,a user clustering Slope One algorithm based on Bhattacharyya coefficient(BC-Slope One algorithm)is proposed.The Bhattacharyya coefficient is used as the weight to describe the difference between the items,and the user clustering is used to find the neighbor user set of the target user.In real-world movie data,the proposed Slope One algorithm can improve the accuracy of prediction in the case of low computational complexity.2.For the complexity of recommendation data,this paper proposes an improved Slope One algorithm based on ensemble clustering instead of a single clustering method,which is called Mix-BC-Slope One algorithm.Through the research and analysis of data and algorithms,three kinds of user-based clustering methods are proposed,which can generate excellent and different clustering results,and then integrate to generate the best clustering results.The experimental results show that the proposed algorithm can further improve the recommendation performance.3.By analyzing the current situation of big data's research,the Hadoop computing platform and its related sub-project HDFS,MapReduce are deeply studied.The research progress of recommendation system based on big data is introduced in detail.The feasibility of parallelization is analyzed based on the proposed algorithm and implemented on the platform of big data.Finally,a summary of the full paper and the further research content to be studied are proposed.