Research And Implementation Of Incremental Association Rules Based On Spark For Smart Phone Viruses Mininng

With the outbreak of smart phone viruses and the scope of the constant change, smart phone security has aroused widespread concern in the information security. Therefore, how to quickly identify the continuous variation of the smart phone virus to avoid the leakage of privacy and economic loss is very important. For the single version of the Smart Phone Virus Mining System training time is too long, this paper realized the distributed association rule mining algorithm based on Apriori and Spark, and implements the distributed incremental updating Association Rule Mining (ARM) algorithm based on FUP and IUA,provides an efficient solution for smart phone virus detection.The main work of this paper is: (1) In order to set up the experimental testing environment, the distributed Hadoop cluster and the Spark cluster based on Yarn are deployed on three virtual machines of Linux systems. In order to facilitate the development and testing of Spark application program, we configured the Scala language environment and IDEA. In this paper, the efficiency of distributed parallel algorithms is verified and tested based on the Spark cluster. (2) We introduce the concept of association rules and Mining steps in detail.Then analyzes the single version of Apriori and its improved algorithm, and presents the realization of parallel Apriori algorithm and parallel improved prearranged-attributes-support-statistic-based ARM algorithm based on Spark distributed computing framework, and the parallel performance of the two algorithms is analyzed and compared through experiments. (3)Based on the analysis of the FUP algorithm and IUA algorithm, we have adopted a new method of database operation method and incremental updating technology according to the shortcomings of these algorithms,and propose an improved scheme for these two algorithms. Then, we describe the scheme of parallel improved FUP algorithm and improved IUA algorithm based on Spark, and the effect of two parallel algorithms is verified by experiment. (4) In the application of Smart Phone Virus detection System, we have completed the design and implementation of association rules module and detected virus using incremental association rules algorithm based on Spark, and analyze the experimental results.The contribution of this paper as follows: (1) According to the shortcomings of FUP algorithm and IUA algorithm, this paper proposes an improved incremental updating association rule algorithm FUP and IUA algorithm, which improves the efficiency of the algorithm. (2) In this paper, the association rule mining module combines the Apriori and improved FUP and IUA association rules incremental updating algorithm.Firstly, we established the association rules model by Apriori algorithm,then train the incremental data by FUP algorithm, at the same time, based on frequent itemsets qualifying rate of moderate degree, we change the minimum value of support in support of the standard deviation of the permit conditions using IUA algorithm, and generate the stronger association rules, in order to find the unknown virus. (3) In this paper, we implement the distributed parallel of improved Apriori algorithm,Incremental Updating Association Rules FUP improved algorithm and improved IUA algorithm.This not only improves the efficiency of the algorithm, but also makes up for the lack of correlation analysis algorithm in MLlib, and can be applied to other scenarios for large-scale data association analysis.