Research On Online Detection Of Power Abnormality Based On Spark Streaming

The phenomenon of stealing electricity in China is very serious.The annual power loss reaches more than tens of billions of yuan.Except for natural line losses,most of them are non-technical line losses,that is,losses caused by stealing electricity.In the process of using electricity,illegal acts of stealing electricity and stealing electricity,and privately changing power equipment have seriously affected the healthy development of power companies,national economic construction,and social stability.Although power companies have spared no effort in combating theft of electricity,the abnormal power consumption has a long detection time and users are widely distributed,and the results cannot be obtained quickly or even in real time.With the development of the power grid,various smart power metering devices are widely used,generating a large amount of power data,which brings a new opportunity for the research of power anomaly detection based on power big data.However,in the past,the abnormality detection method of power consumption is usually based on historical data analysis.The advantage of real-time transmission of power data flow is not obvious,and the test results are often lagging behind,which can not meet the timeliness requirements in the existing environment,and help power companies achieve timely stop loss..In view of the above problems,this paper proposes to combine the stream data mining algorithm with Spark to realize the online detection of user power anomaly.Firstly,for the historical data,the Kmeans algorithm is used to cluster the historical data of the user horizontally and vertically,and the user's clustering label and behavior pattern are obtained.Then the current user data stream is selected by setting the size of the sliding window,and the flow clustering algorithm is adopted.-Streaming Kmeans quickly obtains the user's current power behavior pattern,and finally compares the user's current power behavior pattern with the behavior pattern of the same user and the user's historical power behavior pattern to discover potential abnormal users.To narrow the scope of inspection for power companies and achieve rapid detection.The main work of this paper is as follows:1.The convection data processing technology and the stream data clustering technology have been studied in detail.In this paper,the landmark model,sliding window model and snapshot model are compared.The common flow clustering algorithms,such as streaming Kmeans.hierarchical clustering,streaming DBSCAN,etc.,are studied and compared.2.Introduce the Kafka message subscription publishing system and Spark Streaming stream processing technology for data transmission.Then,a stream processing platform with Kafka is built,and the streaming Kmeans algorithm is implemented on the platform to verify the data throughput and fast processing performance of the stream processing platform.3.Offline clustering analysis and online anomaly detection on the built platform using UCI's public data of Portuguese resident users.The experimental data includes data collected by 370 households in Portugal from 2011 to 2014,and the data volume is 4*365*96*370 points.Experiments prove the effectiveness of the proposed algorithm and scheme,and provide a basis for the identification of suspected users.