Research And Develop Of The Energy System Based On Hadoop Platform

In recent years, with the rapid development of communication industry and Internet industry, more and more people enjoy the convenience of technology development. At the same time, the scale of the data centers is growing rapidly. The energy consumption of data centers can’t be ignored. In2012, the power consumption of China data center are664.5billions degrees, which has accounted for1.8%of the total industrial power consumption, and power consumption is still in the high-speed growth. In order to reduce the energy consumption of data centers, real-time monitoring of data center’s energy consumption is required. Based on energy consumption data collected, the system consists of energy query, energy analysis, alarm management and other functions. With real-time energy consumption information of the data centers, the system can provide comprehensive information support for data center managers.Based on the above requirements, this paper designs and implements energy system based on Hadoop platform. The related word of this paper mainly includes three parts:1. This paper designs and implements a mixed mass data storage architecture based on Hadoop and relational database. Hadoop has good performance in parallel processing of big data; relational database has advantages in terms of fast query, combining the advantages of both, build the framework layer of the energy system. Use Sqoop to achieve importing and exporting data between Hadoop and relational databases.2. Relying on the mixed mass data storage system, design and implement energy system functional module layer. Function module layer includes energy query module, users can customize the query conditions (time, energy options, energy type, etc.) to query data needed; energy-analysis module, analyze data for multi-dimensional; parallel data mining module, energy consumption data collected for parallel data mining, deeper digging out useful information; multi-tenant management module, improve the resource utilization of the energy system, at the same time, through data isolation and application isolation to ensure the reliability of data services.3. Combine energy consumption data collected with computer CPU utilization, memory utilization and other hardware data, this paper presents an energy model; this model can use the hardware data to predict energy consumption. Computer hardware data can be easily obtained, obtaining energy consumption data through sensors exists sensor deployment costs and its own energy loss. Through energy model to predict the energy consumption greatly reduce the cost of data acquisition. This paper gives a derivation of the energy model, get the parameters of the model by experiment, and finally test the accuracy of the model.