Research On Mining Algorithm Of Weighted Association Rules In Big Data

With the popularization of traditional industry informatization and the rapid development of Internet accelerated speed technology,a large amount of management data has been produced.How to efficiently mine its data has become the current research focus.Association rule mining is mainly used to discover hidden knowledge in data.However,the traditional association rule mining algorithm ignores the importance of items in the data set and the efficiency of mining large data sets is not high.To solve this problem,based on the distributed computing framework,this paper proposes a weighted association rule mining algorithm in big data environment,which realizes the efficient and accurate mining of weighted association rules in large data sets.Firstly,the problems related to weighted association rules are analyzed,and the problems of common weighted model are found,and the independent probability complete weighted model is established.This paper analyzes the association rule mining algorithm,big data computing framework and prefix division strategy,and finds the method that can optimize the operation efficiency of big data mining algorithm,so as to improve the mining efficiency of the algorithm on big data.Secondly,based on the analysis of weighted association rules in big data environment,the MR-CEWR algorithm was proposed,which used the independent probability fully weighted model and the idea of dividing data blocks for data distribution,and mined weighted association rules in parallel on the Map Reduce computing framework.This paper proposes W-Sp Eclat algorithm,which uses bitmap data structure,prefix partition strategy and memory-based Spark computing framework to improve the time efficiency of mini ng weighted association rules from big data.Through the comparison experiments of the two algorithms,the efficiency and mining performance of the algorithm are verified.Then,based on the analysis of negative association rules,the Sp-CEWPNR algorithm was proposed.The algorithm used the correlation-weighted support model to screen the weighted candidate itemsets and identify the positive and negative correlation attributes of the itemsets based on the Spark big data computing framework,and combined with the independent probability fully weighted model to mine the positive and negative weighted association rules that meet users’ expectations.The memory,CPU occupation and running time efficiency of the algorithm are verified by experiments.Finally,in order to verify the actual effect of the algorithm,the research results of this paper were applied to the learning effectiveness analysis application system,and the system developed functional modules such as data preprocessing,data mining and result analysis.The application results show that the algorithm in this paper can accurately and efficiently mine the strong correlation between students ’failing subjects.