Research On Optimization Of Knowledge Discovery Algorithm For Massive Data Based On Coarse And Fine Grain Size

Nowadays,with the widespread use of the Internet,Internet applications represented by Internet e-commerce,mobile short video,etc.have rapidly spread,and the global data capacity is growing at an unprecedented rate,and eventually reached a massive amount.The phenomenon of“excess data”but“lack of information”has become increasingly prominent.The need to discover new and valuable knowledge from massive data is becoming more prominent.In the face of massive data,traditional knowledge discovery algorithms have a problem of a surge in hardware demand and inefficiency.In view of the above problems,this paper studies the existing knowledge discovery algorithm and proposes a knowledge discovery algorithm based on the“Coarse and Fine tuning”engineering thinking(FAMCF),in order to improve the efficiency of massive data knowledge discovery.The main research contents include:(1)In view of the low performance of the traditional Apriori algorithm for processing massive data,an optimization algorithm named ICRP-Apriori for narrowing the knowledge mining scope is proposed in the “Coarse tuning”stage.The algorithm uses pruning technology to perform pruning in 2 stage.Firstly,project constraints are introduced to prune frequent itemsets.Secondly,the candidate rule sets are pruned by chi-square test.Tick non-target rules is used to improve the efficiency of association rules mining.Experimental results show that the optimization algorithm is superior to the traditional Apriori algorithm in efficiency.(2)In order to further explore more fine-grained knowledge,a transfer matrix computational adaptive method for Markov chain prediction model is proposed.This method maps the estimation error of the transfer matrix to the error of the predicted value,and realizes the adaptive value and calculation of the transfer matrix by evaluating the index ?,and then obtains the fine-grained quantized value of the association degree of the association rule.(3)The FAMCF knowledge discovery algorithm for comprehensive coarse tuning and fine tuning methods is proposed.The algorithm combines the ICRP – Apriori algorithm and the Markov chain-based prediction model to achieve knowledge mining from coarse-grained to fine-grained.Verify the FAMCF with specific medical e-commerce instance data.The experimental results show that the FAMCF algorithm proposed in this paper is superior to the traditional algorithm in performance.