Research And Implementation Of Data Mining Platform Based On Industrial Big Data

At present,industrial enterprises do not have very clear ideas on how to use existing data to promote digital transformation and improve business efficiency.In response to this phenomenon,exploring the collection,analysis and application of industrial big data has become a new direction of research and development in the field of big data,and it is necessary to establish an exclusive data mining platform to provide an ecological fulcrum for industrial digital transformation.This thesis is based on industrial big data and industrial application scenarios as the research object,mining generalized modules in industrial application scenarios,and based on the Hadoop distributed framework and Spring Cloud microservice framework to implement a basic foundation that can provide a variety of industrial application scenarios Service supported data mining platform.In order to meet the above requirements,a new definition of the layered design of the platform architecture is made,and Drools is used to extract data regularly based on the characteristics of industrial data sources to flexibly respond to the data requirements of different upper-layer applications.Through the analysis of the specific needs of different industrial applications,basic microservices such as algorithm libraries and translators are extracted.Among them,the algorithm library can effectively complete the algorithm support for the upper recommendation system,prediction system,and bargaining system through the implementation of K-Means,CART Decision Tree,Support Vector Machine,TF-IDF and other algorithms,and realize flexible call and intermediate through interface configuration Data storage.Secondly,the translator can realize high-level language conversion of custom grammars through the original Sugar conversion method,unique lexical division and Abstract Syntax Tree structure,with reliability and flexibility.The bottom layer of the platform uses Flink to implement streaming data processing,Map Reduce completes batch data calculation,and cooperates with high fault tolerance,suitable for batch processing,and high scalability to complete the storage of large quantities of data,and is designed with Mongo DB,Redis,and Mysql to complete the storage of business data,whether it is distributed storage or the storage and return of upper-layer application data,it has extremely high stability.And this type of component can well support cluster expansion,which is convenient for enterprise-level applications and expansion.This thesis made a microservice design based on the specific application requirements of the project.At the same time,in the project practice process,it also designed many basic microservices to support a wider range of platform applications,and finally realized a support for high concurrency,scalability,high stability,Highly flexible and reliable distributed data mining platform.The application innovations of the thesis are as follows:(1)Made an adaptive design for the distributed integrated environment.Including the use of lightweight architecture Flink for streaming data processing;splitting the log system into distributed system logs and upper-level business operation logs;splitting platform storage into distributed storage and business database storage;(2)designing the data service middle layer with microservices as the core.Including a series of microservices with Spring Cloud architecture as the core,focusing on ETL,algorithm library and original translator design.Through achieving the above innovations and experiments,the feasibility of the technical scheme in this thesis is verified.Finally,in the complete function and performance test,the reliability and advancement of the platform are confirmed.