Design And Implementation Of Data Processing System For Industry

With the advent of the industry 4.0 era,coupled with the rapid development of the Internet and computer technology,a large amount of data has been brought for industrial production and industrial management.The analysis and mining of industrial big data is the key to the future development of industrial informatization.However,due to the extensive field,the complex structure,and the diverse influence factors of the industry,the industrial big data has the characteristics of large volume,high real-time performance,complex structure,and various storage media.As the data volume accumulates,traditional way of data processing are insufficient to solve these problems.So a new industrial big data platform is urgently needed to manage these massive data and explore the potential value of them as much as possible in order to provide decisions for intelligent production.Based on the characteristics and needs of industrial big data,this thesis designs an industrial-oriented data processing system.The system is designed with a five-layer architecture,including a data acquisition layer,a data buffer layer,a data storage layer,a data analysis layer,and a data interaction layer.The main function of the data acquisition layer is to communicate with the data forwarding devices,which is responsible for real-time data collection in the industrial production process.Above the acquisition layer is the data buffer layer,which utilizes Kafka distributed message middleware to buffer and storage the data collected in real time temporarily,and the purpose of which is to achieve the decoupling of data acquisition and data processing.Aimed at the multi-source heterogeneous characteristic of industrial data,the data storage layer is located above the buffer layer,and it selects MySQL and distributed file system HDFS as storage medium,responsible for multi-level storage function of industrial data,which solves the storage problem of massive industrial data.The data analysis layer uses Spark as the computing framework of big data mining to realize rapid analysis of massive data.And the main function of the interaction layer is to manage and monitor the whole system and to provide good management screen.According to the design scheme,this thesis implements an industrial-oriented data processing system,builds a big data infrastructure platform Hadoop and a parallel computing framework Spark,and carries out modular development.Finally,the designed system is tested,the experimental results of which reveals that the processing system designed for industrial big data has good performance and good practical significance for the processing of industrial data.