| Haze has been significantly negative effects in aspects such as climate, environment,health, and economy, et al. The formation and evolution of the haze has its own mechanism and law of different areas, the reasons for the formation of urban smog, form of expression is different. Now, the scientific understanding of fog is still not systematical, complete. The main reason is that human without a thorough perception, cognition, and fully analyzed for haze. For a large number of meteorological data has been accumulating by the meteorological department with the enhancement of informatization degree, the related theory of data mining methods can be used to discover knowledge, seek the law of haze formation and evolution, so as to improve the forecast accuracy, and guide the production and living of people.Firstly, the data mining and its application in meteorological research status at home and abroad are summarized, the application of data mining in meteorological analysis are studied thoroughly. Secondly, the related theory, implementation process, advantages, and disadvantages of Apriori, one of the classic algorithms for association rules mining are analyzed; the related theories of cloud computing and Spark, one of the popular large data processing platform are surveyed; the knowledge of haze is introduced, and the effects of meteorological factors on smog is analyzed. Thirdly, on this basis, a novel algorithm for mining association rules called AMRDD(Apriori Matrix Resilient Distributed Dataset)which is based on Spark is proposed. This algorithm expresses frequent itemsets with matrix, stores data based on HDFS. The numbers of frequent itemsets candidates are reduced by using the principle of global and local pruning. The time efficiency is improved by using Spark based on the calculation of memory. Finally, the algorithm is applied to the correlation research of meteorological elements and haze.In order to validate the performance of the proposed algorithm and its application on haze research, a Spark platform is built by using three entities to compose a local area network. It is a platform of the correlation analysis on the data flow between the haze and meteorological elements. In addition, three cities of Beijing, Changchun, Shijiazhuang meteorological data from October 2013 to April 2014 are used as tested data set,respectively. The experimental results show that the proposed algorithm(AMRDD) is suitable for the haze and the meteorological elements correlation research for the conclusion is consistent with the traditional methods. Moreover, some other testing experiments with randomly generated large data sets are also completed. The results illustrate that compared with the traditional Apriori algorithm and the Apriori algorithm based on Hadoop, AMRDD can improve the time efficiency significantly. |
