Industrial Quality Data Analysis Platform For Streaming Data

Big data processing technology is one of the important development directions in the field of contemporary information technology.With the continuous development and application of big data related technologies,real-time computing is getting more and more attention.The big data processing technology based on MapReduce computing framework,represented by Hadoop,often uses batch processing to analyze historical data,while the real-time computing field emphasizes the online processing of instantaneous data.On the other hand,in the production process of traditional industrial enterprise products,massive amounts of data are generated all the time,such as sensor data of industrial equipment.The characteristics of these data are continuous,infinite growth,disorder,and need realtime response,which is called streaming data.In the face of these massive streaming data,industrial enterprises have the following three main problems: 1)Unable to solve Large-scale storage problem of massive streaming data;2)Processing streaming data as ordinary data cannot realize the real-time characteristics of streaming data;3)Traditional technology architecture cannot analyze streaming data in a higher dimension,and cannot apply new technology such as machine learning model to analyze and process.Obviously,the traditional data analysis and processing method cannot effectively analyze and store the streaming data.How to mine the deeper value of industrial quality flow data,analyze the dimensions of these quality data,and display them to ordinary users dynamically in the form of visualization is the main purpose of this thesis.Based on the actual research background and requirements of this topic(quality big data analysis cloud service platform for industrial enterprises),this thesis starts from the design of a relatively universal and highly available real-time streaming data processing framework,the entire quality data analysis platform is divided into the following four key modules: 1)data acquisition module;2)streaming data real-time processing module;3)data storage module;4)big data visualization module.In general,the main research contributions of this thesis are as follows:1)Based on the generation scene and characteristics of industrial quality streaming data,the design and implementation of real-time collection of industrial quality data are completed based on Apache Flume.2)Based on the Spark cluster,a relatively common and highly available real-time streaming data processing framework is designed and implemented,Spark Sql and Spark mlib can be used to complete the structured query of streaming data and the application of machine learning model based on the framework,so that the framework has good scalability.3)Applying the front-end and back-end separation architecture to the big data visualization module,so that the front-end project can focus on the responsive presentation of streaming data to achieve the visualization requirements of real-time streaming data processing.The back-end project can realize the micro-service architecture based on RESTFUL style,and realize the high availability,high concurrency and low coupling data visualization module.Starting from the overall design of the quality data analysis platform,this thesis introduces the specific design and implementation of each module.On the basis of the real data generated by the equipment of 1580 hot rolling production line in a steel plant,the construction of the whole real-time flow data processing flow is completed.