Research On The Network-based Inference In The Context Of Big Data

Thanks to the rapid growth of Internet technology, the amount of information grows at a very fast pace and brings the problem of information overload. The recommender system is considered to be one of the most effective methods to solve the so-called information overload problem. Recommender system has become a hot issue in the field of information technology.The thesis first discusses a variety of important recommender algorithms basing on the basic theory of recommender system, and then analyzes the network-based inference in detail. Based on this, four algorithms, named user-based network inference, item-based network inference, user nearest neighbor based network inference and user similarity prediction based inference are proposed.User similarity and item similarity are introduced into user-based network inference and item-based network inference respectively, so the resources in the items and users bipartite network can flow in a more personalized way. The simulation results on MovieLens dataset show that the improved algorithms have better recommendation results than the original network-based inference.User nearest neighbor based network inference and user similarity prediction based inference not only reduce the size of the data set and data sparsity, but also filter out the interference information by screening data sets and link prediction. The simulation results on MovieLens dataset show that the improved algorithms can achieve or exceed the effect of the original network-based inference with much smaller amount of data.Finally, the thesis applies MapReduce design patterns to redesign the execution flow of the recommender algorithms and introduces the functions and implementations of each module of the recommender algorithms in detail. The recommender algorithms implementationed by MapReduce design patterns can adapt to the high concurrency runtime environment in the context of big data.