| With innovative information technology, data is already showing explosive growth, the Internet industry is facing a huge shift from IT to DT. How to improve the ability of mining knowledge about massive data has become a problem at this stage. Distributed computing architecture appears to provide a new massive data mining solutions. Migrating traditional data mining algorithms to the cloud platform can make a greatly improvement for the efficiency of data processing.Due to the defect that traditional data mining algorithms can not cope with massive data mining, the thesis research on the open source distributed parallel computing frameworks such as Hadoop and Spark for making the parallelization of traditional data mining algorithms to strengthen their scalability.The main work is as follows:(1)The improvement of the parallel DBSCAN algorithm. When the existing parallel DBSCAN algorithms making data partitions, the original database is usually divided into several disjoint partition, with the increase in data dimension,the splitting and consolidation of high-dimensional space will consume a lot of time.In order to solve this problem,this thesis proposes the improved parallel density-based clustering algorithm(S_DBSCAN),and its implementation on Spark.The experiment shows that,the improved S_DBSCAN algorithm guarantees correct clustering results, and provide better operating efficiency and scalability as well.(2)The improvement of the parallel locally weighted regression LWLR algorithm.The existed parallel LWLR algorithms use traditional way to get neighborhood,which resulted in low efficiency.In order to solve this problem,this thesis proposes the improved parallel locally weighted regression algorithm based on new neighborhood searching strategy named T_LWLR.The experiment shows that the improved T_LWLR algorithm has good scalability.Compared with the existing parallel algorithm KNN-LWLR,T_LWLR algorithm has a shorter running time,so that the operating efficiency has been improved.(3)Designing and implementing data mining analysis system based on improved S_DBSCAN and T_LWLR algorithms in view of Spark. Users can get access to this platform by setting a simple parameter configuration according to the specific application scenario.Meanwhile,applying the improved algorithmns to specific scenario to get data analysis in this platform. |
