| Data mining,which uses the exploration of potential models to provide enterprises with decision-making basis,refers to the process of finding implicit valuable information from a large amount of data.Traditional data mining technology is mostly based on offline knowledge database,using the algorithm of clustering classification to process data.Although it can well explore the value of knowledge base,but did not give full consideration to its scalability in real-time environment.While in the production environment,a lot of data processing requirements are real-time.In order to solve this problem,this paper uses the advantages of storm flow processing platform,put forward the concept of real-time data mining,of which the main idea is to combine the topology structure of storm and data mining algorithms in a bolt.According to storm whose processing logic is processing data timely,when the data access to topology,data analysis can be token as soon as possible.In order to make better use of the advantages of distributed computing environment,the problem of unbalance load of multi-topology in storm cluster is solved.Based on the original algorithm,we added the weight ratio of the existing spare resources of every node,which can make the resources of the cluster fully utilized.In order to verify the feasibility and necessity of real-time data mining,this paper selects the short text in social platform as the object of processing,and analyzes the emotional dynamics and trends of hot events timely.Because short texts have different semantics and higher feature dimensions than those of traditional texts,the method of convolutional neural network is adopted to extract the semantics of short texts and the semantic vector model is used to express short texts to maximize the retention Short text message.For a large number emotion pictures Internet language and other issues appear in short text,the use of vocabulary mapping to replace unusual parts is adapted to deal with this problem.In the aspect of emotional analysis,we use the k-fold cross-validation method to train different classifiers and select the best three classifiers to form a combined classifier,and then use AdaBoosting to optimize based classifiers,and finally use the combined classifier to predict the emotion of the short text vector.The main contributions of this paper are as follows:1.Improving the storm scheduling algorithm to improve the real-time system.2.The use of emotional words instead of the expression method,making it possible to handle text and emoticons.3.The establishment of a thesaurus to deal with short text in the network language problems. |
