Research On Collaborative Filtering Recommendation Algorithm Based On Big Data

With the rapid development of computer technology and network technology, many products of information age, such as EC, Social networking, portals, and intelligent applications of medical, education and public platforms, have entered our daily lives. They have effects on our behavior and thoughts. Also, the consequent massive data makes us live in big data era of information explosion. To alleviate the information blindness and reduce the difficulty of finding useful information, personalized recommendation becomes an excellent helper.Collaborative filtering is one of the most successful technology applied in recommendation services. It is unrelated to properties of users or items, but exclusively relies on the ratings. So it can work well over interdisciplinary fields with good generality. However, with big data era coming, the problem of data sparsity is more serious, which brings negative effects to recommendation quality.In this background, this paper introduces the concept of average similarity, integrates user as well as item, and then proposes a kind of improved model called ASUCF, which is proved that effectively improve the accuracy of prediction by some experiments.Meanwhile, faced with big data, how to deal with the data quickly and efficiently by parallel processing has also become a hot academic research. At present, there are many kinds of processing framework for parallel computing. But the concept of cloud computing and MapReduce parallel framework published by Google has been widely used in data processing because of its high scalability and ease of use. Hadoop, the open source cloud computing system, which implements the function of Google cloud computing, has been widely used by researchers. This paper proposes and designs a kind of improved collaborative filtering algorithm called ASUCF and its MapReduce parallel processing on the basis of the combination of recommendation technology and cloud computing technology. Recommendation quality is improved from accuracy and computational efficiency. The main research works are as flows:(1) Common recommendation techniques and their principles, characteristics and applications are introduced. How collaborative filtering algorithm works is analyzed, including the principle, steps, evaluation standards and its classifications— Memory-based CF and Model-based CF, as well as the level of recommender system in big data.(2) For the problem of data sparsity, a kind of improved model called ASUCF is proposed. It uses the average similarity to punish the fluctuations of user’s ratings or item’s score. With some experiments, the algorithm is proved to work better in accuracy.(3) With MapReduce programming framework of Hadoop, an open cloud computing platform, how to make ASUCF algorithm in parallel is analyzed.(4) For the problem of computational efficiency, this paper uses Hadoop, studies Taste recommendation engine in Mahout, and designs the algorithm process of ASUCF adapted to MapReduce programming model. With some experiments, the algorithm is proved to work better in computational efficiency.