The Research On Collaborative Filtering Recommendation Algorithm In The Big Data Era

With the rapid development of network and social platform, the network information has shifted from an era of shortage to that of overload. Network information overload is a difficult problem for users and administrators. In order to improve the present plight, the search engine is born. It is through the users' input keywords to search the information needed; but if the keywords input by users are not accurate, the search results will be inaccurate as well. Hence, personalized recommendation system comes into being, which is able to independently store users' preference and then set up a model to provide users with accurate recommendations when users are surfing online. Personalized recommendation system can not only offer information resources to meet the needs of users, but also will help users find out whatever they will be interested in,thus, it has greatly developed.Collaborative filtering recommendation algorithm is the maturest algorithm in the personalized recommendation system. However, faced with an era of big data, it has some problems such as data sparse and scalability. This paper aims to solve these problems and improve the quality of collaborative filtering recommendation algorithm. The main contents fall into as follows:(1) An algorithm based on the time and the number of common rating was put forward to handle the data sparseness problem in collaborative filtering recommendation algorithm. It was a new similarity method, considering the user's interest would change with time passing and the number of common rating would influence similarity calculation between users. Therefore, the time factor and common rating function was introduced on the Pearson similarity formula basis to improve the accuracy of the similarity and ease the data sparseness problem.(2) An algorithm based on BiasSVD and clustering user nearest neighbor was proposed to deal with the scalability problem in collaborative filtering recommendation algorithm. It was through BiasSVD model to simplify matrix dimensions, and obtain predict ratings. Given there would be some certain errors, the average difference between the actual rating and predict rating of the target user nearest neighbor was used to adjust prediction rating of target user. At the last, scalability and quality of recommender systems were greatly improved.(3) In view of the two kinds of improved algorithms in this paper, I designed and deployed experiments to verify the validity of algorithms. According to the experimental results, it shows that the two kinds of improved algorithms are greatly improved on the recommendation accuracy and the online recommended time is greatly reduced. Therefore, the efficiency of recommendation is improved, compared with the traditional collaborative filtering recommendation algorithm.