Research On Energy-conserving Strategies Of File Storage Based On Cluster Scale Adjustment

With the development of human society, the social informatization level is much higher, which brings the rapid increasing of data-the carrier of information and creates big data technology and makes our life more convenient. Big data technology's application is inseparable from big data storage, and for cluster storing big data, the energy consumption is an important issue.In this paper, we find a common problem exists in cluster that the usage rate of nodes is low through the understanding of the cluster file access and cluster server energy consumption rule. So the main research content in this paper is to improve the usage rate of nodes and reduce the energy consumption of the cluster by dynamically adjusting the cluster scale.First of all, this paper divides the cluster into three regions by combing advantages of the traditional static and dynamic partition. And puts forward a cluster scale adjustment for IO optimization which aims at the problem that a large number of data blocks needed to be migrated for the dynamic adjustment of cluster scale which can achieve the purpose of fast adjusting the cluster scale by migrating the data blocks as little as possible. Secondly, takes different storage strategies to cold and hot data based on characteristics of data strategy, in which concentrated load strategy to cold data that can save a lot of cold data storage service resources. In addition, proposes two factors elimination cache algorithm and its improvement which aims at solving the problem that extensive using LRU algorithm will lead to a sharp decline in the hit rate when it encounter sporadic, periodic batch access and can improve the hit rate by considering the number of file access and time.In order to validate the feasibility of the proposed cluster energy-saving strategies, Hadoop cluster simulation platform is developed in this paper. Hierarchical design model is adopted in our simulation platform, for the new energy-saving strategy, it just need to program prepared corresponding functions in the strategy layer then compiled it. This simulation platform can be used to customize the cluster caching strategy, data migration strategy, cluster energy saving and so on.The simulation experiment results show that using the energy-saving strategies of HDFS energy-saving varies between 37%~42% than traditional HDFS. And due to implementation of the strategy, performance of system is slight effected, which 0.3% access need to wake up the servers. Because adopting of the cache strategy, the hit rate of the cache is about 13.5%, while the hit rate of widely used LRU is 8.4%, so the performance of the cluster is improved by 5.1% than traditional HDFS.