Research And Implementation Of Key Technologies In CDN-P2P File Distribution Network Based On Hadoop

Along with the the rapid development of Internet technology and the explosive growth of Internet information’s, the network congestion become the bottlenecks of the Internet service quality. In order to overcome the bottlenecks and improve improve the quality of network services, CDN (Content Delivery Network) technology and P2P (Peer to Peer) technology both played important roles. CDN is composed of servers which located in the edge of the network, users can get the contents from the nearest server through the application layer intelligent routing. The traditional CDN based on C/S (client/server) mode, if the number of clients increase rapidly, it is easy to cause the bottleneck of performance. To improve the qualiy of service, the ISPs need to deploy more severs, which lead to high cost. In P2P network, each node can be either as a server or as a client at the same time that broke the bound of C/S mode. Introduce P2P network into CDN network can effectively alleviate the access pressure of edge server and enhance the quality of network services when the number of clients increased. The integration of CDN and P2P is the focus of current network services research.In the CDN-P2P system, as the network scale become more and more big, it can produce a large number of user data and system data, we need to process and analysis the data in order to find the users’ demand. Then, according to the similarity of users" social attributes and demand to change the way of of network organization, improve the network quality of service, it demand that the system has the ability of mass data processing. In recent years, the most popular mass data processing technology framework is the Apache open source project:Hadoop. Hadoop can use its own HDFS file system to store mass data, can also use HBase database store mass data, and in order to deal with very large scale data (PB level) processing, Hadoop use Map/Reduce calculation model.This paper mainly discusses how to combine the Hadoop framework and the CDN-P2P system based on social attributes to improve the network organizational efficiency and service quality of CDN-P2P system. Mainly has the following several key points:First, it introduces the concept of user’s social attributes, the similarity of each user’s social attribues and content demand can be as the important factors affect network topology organization, fully excavate the similarity of users, according to the change of similarity between users, dynamic adjust the network topology to orginaze the network which has higher query efficiency.Second, based on the Hadoop distributed framework, use HBase database to store mass network data and user data of CDN-P2P system, improve the CDN-P2P system for mass data storage capacity and guarantee the data security.Third, based on the Hadoop distributed framework, use Map/Reduce calculation model to improve the ability of large-scale data processing and analysis, extract and mine user’s demand, calculate the similarity of users and improve the efficiency of network organization.