Research And Implementation Of Recommendation System Based On Mapreduce

With the developping of Web2.0, video sharing, social networking services, and microblog become popular applications. While surfing the Internet, users leave a large amount of data. Faced with such a large data set, information overload has almost become to a problem which many users will meet, therefore, finding out useful information from massive data, not only can help users save time, but also gives users a better Internet experience.Web data mining has a wide range of usage, in the e-commerce, we can use the shopping data of users to mining the users’buying preferences and buying trends, as for social networking services, we can dig out the potential value through analysis users’information, microblog comments. Relationship in social network can be abstracted to a graph composed with persons and relations, through analyzing the graph we can unearth potential law. In this context, we proposed a recommendation system using large-scale data processing algorithm in cloud computing environment, and the acquisition and processing of data is designed and implemented on Hadoop platform.The recommendation system discussed in this article is composed of three parts, data acquisition, data processing, and the strategy recommendation. The function of data acquisition module is to capture users’data that system required, such as social network users’id, the users’friends’id and followers’id, the users’ information will be handled and be stored in HDFS; data processing module uses large-scale data processing algorithms to processing data under the cloud computing environment, the distance between presentee and other users is calculated by Dijkstra’s algorithm, PageRank algorithm is used to calculate the influence of users in the social network; strategy recommendation module, use the result of data processing module to recommend, the user’s influence is choosen as the factors to sort friends of friends of the presentee.Based on this system, the social networking service can recommend strangers whom users may want to add as friend to users, which can keep users active and spending more time on social network sites; users can meet new friends by taking advantage of this system, and alse increase their influence and expand their contacts. The system takes Twitter’s data as an example while doing experiment, actually the the system can be used for some other large-scale data processing, as the data meet the requirements of the format of the data processing, and this system based on Hadoop platform, which means it has good scalability and can be able to handle big data.