The Design And Implementation On Architecture Of Power Big Data System Based On SOA And Private Cloud

With the development of Power Automation and Smart grid,the five major links of power grid is constantly generating tons of data,giving great pressure to data processing system that the grid currently uses.The bond of power grid and big data processing technology is becoming a new hotspot of research.To improve power grid's capability of processing bigdata,and reduce the technical requirement for grid staff,this article introduces the designment and deployment of power big data system including OLAP interface,complexed computing system and ETL system.The big data system uses Micro Services Architecture,a special and light form of Service-Oriented Architecture.Through blending of Hadoop Distributed File System,GreenPlum Datastash and Spark Cluster Computing System,this article introduces a design of Big Data System,containing an Online Analytical Processing interface,a Spark-based Cluster Computing System,and an Extract-Transform-Load system based on spark.Cluster Computing System maintains a group of SparkContext/SparkSession and provides users with the interface of RESTful APIs,which allows users to upload binaries and execute computing task through HTTP Requests.ETL system is based on Cluster Computing System though usage of Spark Streaming,giving users interfaces to extract data from source and store them to target database.OLAP System is a web application of manipulating sql queries,the application of OLAP System is shown through data visualization technology.Application is deployed on private cloud based on kubernetes,which is a system of auto deployment,extension and management of containers,giving people a convinent way of deploying micro-service based applications.The article provides a spark application running on Cluster Computing System,which compares event sequences to event templetes to analyze the behavior of grid on abnormal events.Sample ETL application shows a program of extracting information from original data stored in HDFS.OLAP application gives user current status of grid from multi-aspects through visualization.