| With mobile Internet, networking, social networking trend sweeping the globe, the amount of data generated rapid growth worldwide, it is clear that we have entered the era of big data, while the data type covers text, images, video, voice and other vast part of unstructured data. How fast access to knowledge has become the main problem we need to deal with current from big data. Data mining is a blend of statistics, databases, machine learning, artificial intelligence and other areas of comprehensive discipline. Traditional data mining algorithms are run on a single serial, as a stand-alone memory, computing power and other reasons, will waste a lot of time to perform the job, inefficient and clustering effect will also be reduced, which can not accommodate large-scale data mining the work. To improve the data mining of large-scale data processing capabilities, a distributed computing technology, divides data sets into node processing on multiple machines, and then the results summary. So you can take advantage of the computing power of multiple nodes efficient handling of large data. Hadoop is a distributed computing framework that provides HDFS, Yarn, MapReduce core component and a number of supporting components that have been widely used in the Internet company’s cloud computing platform.In this paper, by studying data mining clustering and classification, the defect of Kmeans and Random Forest algorithm is proposed to improve the Canopy_Kmeans and Select-attr Random Forest (SRF) algorithm, the main contents are as follows:(a) For clustering and classification of based on data mining algorithm the basic understand and analysis, particularly of Kmeans algorithm and Random Forest algorithm research and analysis, and introduces a number of improvements to this program now two algorithms.(b) Studied the relevant technology of the Hadoop platform, detailed analyzed the Hadoop HDFS, Yam, graphs, the technical architecture.(c) On Hadoop platform realizes the Canopy_Kmeans parallel algorithm, to solve the two problems of Kmeans algorithm:the initial cluster center choice question and cluster number uncertainty problems. And through the experiment proves the improved algorithm has good clustering on cluster effect and operation efficiency, high speedup and scalability.(d) On Hadoop platform realizes the Select-attr the Random Forest parallel algorithms, influence the Random Forest algorithm is classification effect of the three main factors:the number of trees in the forest, random property features a choice, the correlation between constructing the tree. To solve these problems is proposed SRF algorithm, so that construction of the forest have a higher classification performance, the improved algorithm experimentally confirmed to have better classification performance, a higher speedup and scalability on clusters. |
