| In the current era of big data,the rapid development of network technology and hardware equipment has led to the exponential growth of data.Although in small data sets,the traditional data mining technology has been fully used and developed,which has proved its great value and significance.However,under the challenge of massive data,there are still some problems in the field of data mining,such as low efficiency of algorithm execution,insufficient parallel optimization of algorithm and poor usability of data mining platform.The current mainstream distributed computing frameworks mainly include Hadoop,Storm and Spark.Storm is a streaming processing engine,which is used to solve the realtime computing problem of data stream.Hadoop is a batch computing framework,which is mainly used for the storage and calculation of massive data.The HDFS component is oriented to big data storage and has the advantages of high reliability,high fault tolerance and high extension,while the MapReduce model greatly simplifies the parallel programming.Spark is a new generation of memory-based parallel computing framework.The proposed RDD model is simpler and far more efficient than Hadoop.Based on the above background,this paper investigates investigated the current development of parallel data mining,mainly focusing on parallel data mining algorithms and parallel data mining tools.Based on Spark as the programming model and processing engine,this paper has mainly completed the following work:(1)Optimization and parallelization of KMeans++ algorithm: The principle and implementation of typical algorithms KMeans and KMeans++ in clustering problems are studied in this paper,and analyzed the advantages and disadvantages of the algorithm,aiming at the problem that the initial clustering center of KMeans++ algorithm may not be ideal,the KMeans++ algorithm is improved based on the idea of dynamically adjusting the clustering center,and the parallelization of the improved algorithm is also realized.Compared with KMeans++ algorithm in Spark MLlib,the error sum of squares in clusters of the improved algorithm is reduced by over 4% in experiment when k value is large.(2)Support Vector Machine(SVM)parallel algorithm optimiztion: the principle of classification problem and linear SVM algorithm,as well as the process of solving SVM problem with small-batch stochastic gradient descent method and adaptive gradient descent method were investigated.It studied The implementation of the conventional parallel algorithm of adaptive gradient descent method for solving SVM.and iteratively optimizes the conventional parallel SVM algorithm based on the idea of delayed update.Compared with SVM algorithm in Spark,the optimized algorithm shortened the execution time of algorithm without losing high accuracy.(3)In order to verify and apply the above optimization algorithms,this paper designed and implemented a distributed parallel data mining scheduling framework based on Hadoop and Spark,which can meet users' requirements for mining and analyzing large data sets.The scheduling system realized the common data mining algorithms such as classification,prediction,clustering and data preprocessing,and can complete data mining modeling by visual drag algorithm program. |
