Research On Optimizing Energy Efficiency For Hadoop Storage System

With the recent emergence of cloud computing based services on the Internet,Mapreduce and HDFS have emerged as the paradigm of choice for developing large scaledata intensive applications. Given the scale at which these applications are deployed,minimizing power consumption of these clusters can significantly cut down operationalcosts and reduce their carbon footprint-thereby increasing the utility from a provider’spoint of view.This paper addresses energy conservation for clusters of nodes that run MapReducejobs. The algorithm dynamically reconfigures the cluster and the data on it based on thewhole and part cluster utilization. And it can also benefit from the controlling of datalayout. So much good features can be displayed in the algorithm, such as saving powersignificantly, strong real-time, and less load balancing cost, that it can be applied fordata-intensive clusters or enterprise data-centers.In the realization of the algorithm, the entity, which implements the algorithm, iscalled power controller. It is comprised of three modules, data locator, failure controller,and power tuner. First, data locator modified the data distributing and duplicating processof HDFS to customize the data layout. Second, failure controller allowed NameNode inHDFS to tolerate the failure of DataNode. Finally, power tuner is the core part of powercontroller, which contains two threads implement dilution and enrichment methodsrespectively. One of the threads implement the dilution mean, which adds nodes into theHDFS in case that utilization of one of racks rises above thresholds; Another one carry outthe enrichment mean, which retires nodes in the HDFS when utilization of one of racksfalls below thresholds.This paper use GridSim toolkit to simulate HDFS architecture and performexperiments. To verify whether the algorithm is able to reconfigurate the cluster inaccordance to the workload imposed on it, functional test was conducted; To check theenergy saving effect of the algorithm, back-to-back test with the traditional CS was carriedout. The paper evaluated the algorithm and the results show that the proposed algorithm achieves an evident energy reduction of30.32%under high workloads and up to69.77%under low workloads.