Design And Implementation Of Industrial Big Data Energy Efficiency Analysis Platform Based On Spark

The rapid development of information technology and the wide application of the Internet have triggered the strategy implementation of "re-industrialization" in the world successively.The strong protection of Internet in data transmission,management software and information application opens the door for various technical applications including Internet of Things.The globalized industrial revolution of leading by intelligent manufacturing and enhancing the level of industrial intelligence and the competitiveness of enterprises through emerging technologies has been put on the agenda.Industrial intelligence will be the key to form the industrial system in the future.The industrial big data analysis technology and its application based on industrial internet will become the key elements of promoting intelligent manufacturing,enhancing manufacturing production efficiency,competitiveness and the important basis for implementing intelligent production process,intelligent process management and intelligent manufacturing mode.With the expansion of business production,the most intuitive is the exponential growth of data.More importantly,the data sources are becoming more and more abundant.The original historical data obtained from the database is expanded to production management logs and data of equipment operations,as well as multi-dimensional data such as sound,images,and video.traditional data analysis and management methods cannot cope with the urgent need of companies to increase their production efficiency.Therefore,it is significative to design and implement an industrial big data processing platform to achieve unified management of enterprise data,and tap the huge potential of these data to guide production,provide decision support,and promote the transformation of production from manufacturing to service manufacturing.This paper designs and implements a big data platform that can store,manage and analyze industrial big data based on Spark.The main research contents are as follows:(1)Researched industrial big data processing technology in this paper,starting from the important processes of data processing such as data integration,data storage,data processing,and visual analysis.The paper also studied the data processing framework based on Spark,and designed a NOSQL database based on HDFS,HBase,and Hive and Mysql,Oracle and other RDBMS database multi-level data storage system,using different storage models for different data types,to achieve unified access and management of multi-source heterogeneous industrial big data;(2)The energy efficiency analysis module is integrated in the industrial big data platform.And taking ethylene production as an example,the energy efficiency in the ethylene production process is effectively analysed combining with the ethylene production data by machine learning algorithm in the platform algorithm library;(3)Use the open source Apache HUE tools to achieve the visualization of big data,to achieve support for Hive's sql query,spark interactive query,sqoop heterogeneous data conversion and other functions,to provide users with a good human-computer interface that allows user to conveniently analyze and manage industrial big data.