Data Mining Of Huge Amount Of Energy Consumption Data Based On Hadoop

Telecom operators support communication operation of the whole society, a large number of IT data center and communications base stations are carrying vast amounts of user requests all the time. In recent years, with the rapid development of communication industry, volume of business and services increase year by year while bringing huge energy consumption to telecom operators. How to monitor various types of telecommunication equipment and do further analysis with the energy consumption,in order to help companies to cut down energy consumption, reduce operating costs? It is undoubtedly a.far-reaching problem. At the same time, large numbers of engine room and base station are generating a large huge amount of energy consumption data, how to organize these data, make proposed mining method of these data is also a challenging field.In this paper,we build a huge energy data analysis system prototype using data warehouse and Hadoop clusters.With this system,we do multi-dimensional analysis of energy data and mine huge amout of data using Hadoop clusters.The main work is as follows.Design and implementation of a mixed energy analysis system based on Oracle data warehouse and Hadoop.Using Oracle to store business data and do OLAP ayalysis,using. Spoop for data exchange between Oracle and HDFS,using Hadoop to do data mining of large-scale data.The system is realized with Struts2framework and the front display technology is ExtJS. Implementation of a batch BP neural network algorithm with MapReduce.The algorithm can train a large number of.BP neural network,taking full advantage of the big data processing capabilities of Hadoop.The train of neural network will be carried out while the system is in leisure time,then the result will be stored in Oracle,when users do want to make a prediction,the system can use the result rather than train neural network right now.The realtime performance will be improved.Implemention of x-means algorithm base on Hadoop.The algorithm is a improvement of k-means.There is no need for user to give a k before clusteringjthey only need to give a scope of k,the algorithm will search a most appropriate value of k within this range,which can largely avoid the uncertainty by a blind specified k.Making use of che open source data mining algorithms package Apache Mahout to do association rules analysis of related properties affecting energy consumption,so that we can find the potential relationship between energy consumption and various factors.