Research And Implementation Of Electric Energy Calculation And Analysis Automation System Based On Hadoop And Spark

Electric energy is the basic component of modern society, and its supply directly affects the development of society and economy. Electric energy has found an increasingly wide utilization in the daily life with the rapid development of the electric power industry in recent years. Electric energy calculation automation system becomes more and more important in power management. A variety of metering terminal equipment produces a large number of electric energy data whose real-time data collection, data transmission, processing and analysis are overwhelmingly important to the electric power industry. The application of electric energy calculation automation system saves human resources and improves running efficiency. Electric energy calculation automation system solves problems caused by traditional manual meter reading, for instances, data error, data missing, and large time span, which makes energy data accurate. And the system also provides a good storage and management and accurate analysis of data, which greatly reduces electricity using costs and makes automatic power management possible.With the increasing of the business of electric power departments and the amount of electric data, people put forward high requirements in the ability of massive data processing and the corresponding architecture design of the electric energy calculation automation system. The traditional electric energy calculation automation system is less efficient because it uses traditional relational database to store, query and retrieve data.At the same time, the traditional electric energy calculation model and algorithm have been difficult to meet a variety of new requirements in the data calculation, statistics and analysis. Therefore, in this thesis, the use of distributed file system Hadoop and large data computing framework Spark for the design of electric energy calculation and analysis automation system is useful in many applications. The main contributions of this thesis are as follows.1. Design electric energy calculation and analysis automation system according to the composition and design goals of the system. Then the system is divided into five parts, including tenninal equipment, network communication module, data acquisition module, electric energy calculation and analysis module and electric energy calculation business application. And the electric energy calculation and analysis module is designed in detail, which includes the analysis of the electric energy, data storage and electric energy calculation and analysis. HDFS and Spark play an important role in the system.2. Achieve the function of electric energy calculation in the electric energy calculation and analysis module and study on existing electric energy calculation algorithm, including calculation algorithms of electricity usage, load rate, power factor and three-phase unbalance rate. And the parallel computing algorithm of electricity usage is designed and implemented and a detailed Spark RDD transformation diagram is given out.3. Analyze users' electricity usage by the clustering analysis algorithm k-means.Parallel algorithm for k-means is studied and its running process on Spark is described.4. Analyze the performance of the electric energy calculation algorithm in parallel,and study the influence of running time on one node, four nodes, eight nodes and sixteen nodes in the cluster. In the experiment of users' electricity usage, users' daily and monthly electricity usage are analyzed for clustering research by k-means algorithm in order to find the law of user electricity usage.