| In recent years,with the rapid development of high-performance communication technology and industrial Internet,all walks of life will face the processing of PB-level data.Cluster analysis,as a common unsupervised analysis method in data mining,has become very hot in the industry area due to its high ease of use and high efficiency.DBSCAN is an algorithm for clustering based on the density of sample point distribution.Clustering can be accomplished by setting density threshold min Pts and neighborhood radius distance Eps.However,when the cluster density distribution of datasets fluctuates greatly or the amount of clustered data is large,DBSCAN algorithm has some drawbacks such as too sensitive to noise point identification,difficult to set initial parameters,and too long clustering time.This paper presents an improved algorithm for the initial parameters Eps based on the density distribution of datasets,through the study of DBSCAN algorithm and the improvement of the algorithm by existing scholars.At the same time,a parallel Spark-based clustering algorithm is proposed to solve the problem of inefficient clustering algorithm execution in a large amount of data environment.Finally,based on this algorithm and text mining technology,the automatic extraction of news summary is implemented.The main work of this paper is summarized as follows:First,an improved algorithm DR(Density Recognition)-DBSCAN,which adaptively selects the parameters Eps based on the density of data distribution,is presented to solve the problem of difficult initial parameter selection in complex environment.By calculating and counting the k nearest cluster convergence distance of all data points,the algorithm finds the Eps value that satisfies the density of the area.Implement an adaptive selection of this parameter.For the high complexity of neighborhood data point query in DR-DBSCAN algorithm,R-tree is used to improve the query efficiency.Secondly,in order to better apply DR-DBSCAN algorithm to cluster analysis of large amounts of data,a Spark-based PDR(Parallel Density Recognition)-DBSCAN algorithm is proposed,which divides the dataset into equal subsets by a secondary partitioning strategy,and then performs independent adaptive Eps calculations on each subset and achieves local clustering.Finally,cluster fusion is implemented based on the coincidence labels of local data points.The experimental results show that the computing efficiency of PDR-DBSCAN algorithm is much better than that of single machine clustering and other parallel clustering algorithms.Solves the problem of low efficiency of clustering algorithm in large data environment.Finally,we applied PDR-DBSCAN algorithm to text mining.Through data collection,data preprocessing,feature representation and other steps to complete the construction of the spatial model of text data,and based on the algorithm in this paper,word vectors and sentence vectors are clustered,and finally the automatic extraction of keywords and summaries in text data is achieved. |
