Research On Real Time Storage And Management Of Mass Power Quality Data

With the complexity of power grid structure and the increasing diversity of electricity load,the power quality problems,represented by harmonic pollution,have become increasingly prominent,resulting in huge economic losses.Using power quality data monitoring equipment to obtain power quality data,analyzing and forecasting based on real-time data and managing power quality problems are considered to be effective solutions.However,the power quality data has a wide range of sources,multiple standards,and massive data.The collection and storage of power quality data can not be managed by traditional relation database methods anymore.The management of power quality data is facing a challenge of real time storage performance;in addition,the "dirty" data problem,caused by a wide range of data sources,will also have a serious impact on results of data analysis.How to detect and correct "dirty" data is a key issue for power quality data management.Aimed at the above problems,combined with the characteristics of power quality data,the problems of real-time storage as well as data management of power quality data are studied by using large data technology.The main work carried out included:1.Combined with the characteristics of power quality data,such as strong timing,large data volume and multiple data sources,a key value pair storage structure suitable for power quality data is designed,and a distributed power quality data storage system is constructed to achieve the collection and real-time storage of multi-source power quality flow and batch data.2.A task scheduling model based on ant colony algorithm is proposed,which optimizes the task scheduling layer of the distributed power quality data storage system,improves data collection and storage efficiency and realizes load balancing of receiving nodes.3.Taking data quality optimization in data governance as research objective,we implement threshold based,abnormal data processing method based on slope change and cloud model to screen abnormal data.4.To achieve the power quality data storage system,experiments on task scheduling and abnormal data processing are carried out.The experimental results show that the task scheduling model of ant colony algorithm is 2.65 times higher than the average storage efficiency of the original scheduling system based on the structure,and can achieve load balancing;for highly volatile harmonic data,compared to the method based on normal distribution and cloud model,the anomaly data processing method of adaptive cloud model is more accurate in screening abnormal data.