Research Of Massive Power Quality Data Processing Based On Hadoop

As the scale of power quality monitoring network is expanding and data acquisition frequency of power quality monitoring terminal is improved, power quality monitoring system accumulates massive power quality data. However, the current power quality monitoring system mainly depends on a relational database to store power quality data, and uses the traditional centralized approach to calculating power quality data. For the growing data sets, a relational database has no ability to meet the storage requirements. And in the face of large-scale data sets, the computational efficiency of traditional centralized approach is low, making the analysis results lag seriously. So it is an urgent need for the power quality monitoring system to solve the storage problem of massive power quality data and enhance computational efficiency.This paper describes the theory of HBase, MapReduce, distributed file system HDFS and other key technologies of Hadoop. Taking advantage of large storage capacity of HBase and parallel programming model of MapReduce, this paper puts forward a massive power quality data processing scheme based on Hadoop to solve the bottleneck of the current power quality monitoring system. Taking into consideration of characteristics of the basic data, it uses HBase to store basic data. But to analysis results, its data size is not too large and to be adapted for the existing Web front-end, the scheme continually uses SQL Server 2008 to store analysis results. For calculating power quality data, the scheme implements the parallel programming model of MapReduce based on Hadoop platform. At the same time, through setting up configuration parameters of a Hadoop cluster, the cluster performance is imporved and MapReduce jobs on the cluster are carried out efficiently.In the experimental part, it will compare massive power quality data processing scheme based on Hadoop with traditional power quality data analysis methods to verify the feasibility and superiority of the massive power quality analysis platform based on Hadoop.