Real-time Data Collecting And Computing System Optimizing Under The Background Of Industry Of Internet

Recently,the internet of Industry is developing rapidly in China.This June,The department of Industry and Information brings the target towards the end of 2020:"The overall target of our country developing the Internet of Industry is,till the end of 2020,build up the basic infrastructure and industrial system." This plan shows the confidence and determination of our country developing the internet of industry.Under the background of the Industry of Internet 4.0,the great development of Internet of things and the Industry of Internet promotes the informatization of those Industrial manufacturing companies.However,at the same time,those companies are facing the serious problems like how to collect data from industrial devices,how to do edge calculation,how to do real-time computing of data.Thus,this paper presents a system that can do real-time collecting and computing of industrial devices based on NiFi,MiNiFi,SparkStreaming,HBase and Hive.In the system,NiFi is used for data collecting in industrial devices,so as to send the collected industrial data in binary format using MQTT protocol,MiNiFi is used for caching the sent data of those factories,besides,Spark Streaming receives the binary formatted industrial data in message queue and does the parsing work,meanwhile,Spark Streaming does the real-time computing of the key elements such as the real state,the current amount of production of the industrial device.For the parsed and computed results,they could be stored in HBase(Real-time Data Storage)and Hive(Historical Data Storage).The real-time data stored in HBase could be searched for and presented,while the historical data in Hive could be analyzed and mined.This system presented in the paper uses the distributed data collection terminal built for the Internet of Things,and the centralized NiFi cluster to collecting the distributed collection terminal data in concentration and also the data distribution using the MOTT protocol it also uses the distributed real-time streaming data computing framework Spark Streaming,;To build the system presented in the paper,I built up a cluster of 8 nodes which is the mix of physical and virtual machines.,and also build up the real-time data collecting and computing system of industrial devices that could be expanded in large scale.Thus improve the low efficiency of data collecting,data interpreting and data storing exists in the traditional framework,and also the efficiency of data transmission and data storage.To validate the result,I built up a 20-node cluster testing environment and did a performance test for the sub system of the system built for the paper,so as to get the actual performance after I did the improvement of the system architecture.