Design And Implementation Of Video Cloud Storage Oriented HDFS Load Balancer

Online video service is an important part of internet service, and storage is the basis of it. As a cloud storage system aiming at storing common files, HDFS has been applied by many video service websites, but its load balancer doesn’t take videos’ characteristics of bandwidth consuming when played online into consideration to make full use of bandwidth resource of cluster.In order to solve the problem that original load evaluation model doesn’t take bandwidth resource into account, this paper proposes an load evaluation model based on bandwidth consumption estimation. By analyzing characteristics of videos like bitrate, size of block and popularity, this paper estimates real bandwidth consumption of each datanode when supporting an online video system and uses it to evaluate bandwidth consumption characteristics of datanodes in load evaluation model. Simulation experiments show that the load evaluation model described in this paper can more accurately evaluate bandwidth consumption characteristics of datanodes.In order to solve the problem that datanodes pair badly in generation of load balancing plan when evaluation metrics rise from one-dimension to two-dimension, this paper proposes a generation method of load balancing plan based on symmetric match. This paper analyzes and proposes the load balancing goal under two-dimension circumstance, proposes symmetric match principal under two-dimension circumstance by analyzing the composition of blocks of overloaded, normal and underloaded datanodes, and designs pair method of datanodes and generation method of data migration tasks. Simulation experiments show that this generation method can more effectively accomplish the part related to bandwidth in load balancing goal.In order to avoid the clustering of high bandwidth comsumption blocks in load scheduling process, this paper proposes a load scheduling method based on conditional selection. By analyzing the overall characteristics of the blocks needed to be moved in data migration tasks, this paper proposes a load scheduling method which alternatively uses random choosing method and definite choosing method according to specified conditions. Simulation experiments show that this method can effectively reduce bandwidth balancing goal.Design and implement video cloud storage oriented HDFS load balancer on the basis of original load balancer in HDFS. Tests show that the method described in this paper can make bandwidth resource be utilized more effectively, and it outperforms original balancing method in 90% of cases and manages to decrease bandwidth peak of bottleneck datanode of cluster by 20% in the best case when high bandwidth consuming videos act as hot spots.