Development And Research Of Data Collecting And Application System Based On Cluster Optimization

With the extensive application and in-depth development of technologies such as"Internet +",cloud computing and AI.data processing technology has become the basis of the above research directions.Mining valuable information from multivariate and unstructured data is the main research direction of most scholars at present.Based on this,aiming at solving the problems of opaque purchasing information and difficult decision-making in mechanical parts industry,this paper studies the data of mechanical parts on commercial platform.It mainly includes the design of an intelligent crawler system based on modular design to obtain the target parts information data efficiently and accurately,and the unsupervised clustering analysis of structured and unstructured text data using K-means method and K-means++algorithm based on hierarchical partitioning to provide the basis for the final procurement decision.The chapters of this paper are arranged as follows.In the first chapter,this paper introduces the background and importance of data analysis and mining,points out the main problerms in the process of data application,and elaborates the research status of data acquisition and data mining at home and abroad.In this way,the main research content and main frame of this paper are given.In the second chapter,the design of intelligent crawler system based on module design is completed.The basic module design is completed from engineering creation,data structure definition,Xpath parsing and data storage.The intelligent craw;ler design based on module is completed by crawler speed-up module,anti-crawler processing module,invalid text filtering module and data preprocessing module,meanwhile the web page data is obtained.In the third chapter,a clustering data analysis algorithm based on K-means optimization is proposed for the characteristics of mechanical parts data.For structured data,this paper proposes a K-means clustering algorithm based on hierarchical partition,which optimizes the K-means clustering algorithm through outlier processing,optimal K-value selection and hierarchical method to select initial center points,and visualizes the classification results.Unstructured data is transformed into structured numbers through text segmentation,vectorization and feature extraction.According to this,K-means++ algorithm is used to complete the text feature classification,and finally obtain the information of mechanical parts after analysis,which provides guidance for decision-making.In the fourth chapter,we complete the development of part purchase data service system,and carry out the interface test and function test.We take the bearing as an example and have verified the operability of the system.In the fifth chapter,we summarize the work of this paper,and point out the direction of future work in view of the shortcomings of this paper.