The Scalable Intelligent Analysis Platform Of Electric Power Big Data Based On Spark And Its Application

With the development of big data and artificial intelligence,traditional data computing technology and intelligent analysis technology are undergoing profound changes,and emerging big data intelligent analysis platforms are emerging.Along with the further development of electric power information technology and the rise of concept of smart grid,the power industry data increases exponentially,and the business needs of intelligent analysis for large power data are growing.Therefore,the construction of power data intelligent analysis platform is of great significance.Firstly,the thesis studies the big data technology of Spark ecosystem and the general data analysis platform architecture in-depth deeply,analyzes the shortcomings of the current big data intelligent analysis platform in electric power industry and core requirements.In view of the problems and needs,the thesis proposes a scalable and high available intelligent analysis platform of electric power big data based on Spark,or SSHA for short.The thesis studies and improves those problems in the following aspects.In the aspect of SSHA platform architecture design,the thesis firstly analyses the advantages and disadvantages of the Lambda architecture and designs the architecture of SSHA platform.The design is a platform structure of the fourth layer and the narrow sixth layer.Secondly,aiming at the various levels of the architecture,the thesis explores the corresponding big data component technology,analyzes the advantages and disadvantages of each group at the same level,and determines the technology selection.Finally,adhering to the principle of software design "single responsibility",the thesis optimizes the architectural details for "loose coupling,high cohesion" and the communication between the platform and the heterogeneous system.In terms of the implementation of SSHA platform,the thesis makes key technical implementation in high availability,scalability and computing performance.In the implementation of high availability,the thesis proposes and implements a double-offset middleware for "data zero loss" and a mechanism based on WAL and ECS for the highavailability and final consistency of data.In the implemention of scalability,the thesis designs the business algorithm layer for electric power business.Through the unified interface communication,theoretically the layer can extend the power intelligent analysis business indefinitely.For the excellent computing performance,the thesis proposes the problem of repeated calculation and optimizes the speed of real-time processing by decoupling batch layer and flow calculation layer.In addition,the thought fusion of "flow and dynamic table" is applied to the update of full volume data to ensure the accuracy of data and optimize the accuracy of batch calculation.Based on the platform construction,the usability,correctness and efficiency of the platform are tested and verified.Finally,the thesis combines the ultrasonic test data from the practice unit to study and implement the ultrasonic partial discharge signal recognition system based on SSHA platform.Through the experiment of ultrasonic test data,the validity and accuracy of the recognition system are verified,and the scalability and efficiency of the platform are further verified.