Research And Application On Association Rules Mining Algorithm Base On Hadoop

The traditional data mining technology can effectively mine the decision-making information from data.During the development of data technology,it is a hot issue of how to mining information from big data.Association rules mining is important branch of data mining technology but the traditional association rules mining algorithm needs a lot of compute resource.Based on the current big data technology,this paper studies the association rules mining algorithm based on Hadoop framework.Redesigned the association rules mining algorithm in Hadoop framework and it has better performance.In this paper,we use the vertical data set to represent the transaction database.It changes the traditional association rules mining to compute the intersection between the transaction.The extend the lemma of Apriori,using the prefix to distribute the data to different data node and using bitmap to store and compute,in generated the longer association rules we cache the candidates for pruning that reduce the cost of storage and computed.In order to extend the application scope of association rules,this paper introduces the concept of positive and negative association rules.Due to the particularity of negative association rules,a large number of unintentional candidate sets are often generated,which leads to the failure of algorithm execution.This paper introduces admit based on tradition support-confidence model.In order to improve the execution speed,we introduce invert index based on bitmap.It makes the algorithm can compute the negative association rules at the same time and lessen the number of Map Reduce steps so that improve the computed speed.In order to reflect the importance of association rules mining technology in modern data system,this paper designs the academic warning system as an application case.The platform has a complete interface system and background data system,integrates the relationship between various data sources,and has a relatively perfect data visualization interface.Clustering method is used to cluster students and association rules mining are used to mine the relationship between and within the class.We use positive and negative association rules to validate the association rules so it can improve the predict accuracy.