| With the advancement of information technology,great progress has been made in the production,processing and storage of data.In particular,data processing technologies have made great strides.Traditional unstructured data such as natural language processing,pattern recognition and machine learning and other technical processing also can provide more knowledge and information.As a national important energy enterprise,power grid enterprises have been upgrading their level of automation and information technology in recent years.The data generated by their operation and management and smart grids are huge in scale and complicated in form.The development of smart grid puts forward higher requirements for real-time data collection and access of massive data,on-line calculation and processing and correlation analysis and mining capabilities.The innovation and development of business management and business model put forward new requirements on the value of data assets.Big data is one of the key technologies.This article briefly introduces the related technologies and principles of mature big data,summarizes the batch processing,stream processing and hybrid processing systems and puts forward the applicable scope.Conduct research and analysis on the existing data center of power grid enterprises and put forward the idea of building a big data platform in line with the characteristics of power grid enterprises.To study how to use the existing big data technology to develop the big data platform model of power grid enterprises based on the existing huge data resources so as to realize the construction of a big data platform suitable for power grid enterprises and provide strong support to the management and efficiency development of enterprises.The platform consists of three levels of data integration,data storage,data calculation,data analysis and platform service,and three security functions of data security,data management and management configuration.It provides data storage,calculation,analysis and presentation capabilities and supports the construction of business applications.The platform realizes the management and control of data mining to the whole process of mining analysis.The platform is based on Hadoop as the core architecture.It combines data mining and statistical analysis algorithms,and uses HBase database to access data in a distributed manner to meet the needs of unstructured business applications,Semi-structured and structured low-latency storage requirements,HDFS to file distributed management,MapReduce to complete the job of high-speed computing,in order to achieve real-time data storage,efficient processing,complex calculation and analysis,data integration,data computing,Self-help analysis and other related functions support real-time collection of application-based building.This article through the "data migration to the big data platform,the original data management and service functions unified into the big data platform"approach,to achieve big data platform and structured,unstructured,massive historical/real-time,grid geography information four big data Center to provide a unified pool of data resources for the data resource pool.Based on this,a unified data storage and management service function is provided through the big data platform to provide a multi-shaped public application service for the public application service resource pool. |
PDF Full Text Request