| With the widespread application of computers and the rapid popularization and development of the Internet,the amount of information and types of information available to people have also increased,and most of them exist in text form.How to quickly and accurately obtain the information they need has become a hot topic of current research.One of the problems.The focus of this paper is to proceed from the aspect of feature selection and analyze and discuss the text classification technology in more depth.First,based on the entire process of text classification,the necessary introduction of its basic theory is mainly explained from six aspects,including data collection,text preprocessing,model representation,feature selection,feature weighting,and classifier evaluation.Wait.Then,taking feature selection in text classification as the key research object,this paper introduces the feature selection methods commonly used in text classification,and analyzes its advantages and disadvantages.The feature selection method used in this paper is Chi-Square Statistics.Through the study of Chi-Square Statistics,we find the shortcomings of the method,and then improve it from two aspects.The first is to improve the chi-square statistics based on the probability distribution distance.In order to ignore the weight of different categories of text containing feature terms in chi-square statistics,JS divergence is introduced.The second is to improve the chi-square statistics based on the feature distribution.Because the focus of the first improvement method is on the frequency and proportion of texts containing features in different categories between different categories,the feature distribution is not considered.Therefore,the first The two methods improve the problems ignored by chi-square statistics and JS divergence,specifically:(1)the statistics of the number of occurrences of feature words between categories,this article introduces the word frequency correction coefficient;(2)the distribution of features within the category The text frequency was introduced;(3)The problem of insufficient utilization of information between classes introduced the frequency between classes.And the first improved method and the second improved method are comprehensively used for experiments.Finally,in order to verify the feasibility and effectiveness of the improvement of the chi-square statistics proposed in this paper,the text uses the Chinese corpus of Fudan University compiled by Professor Li Lurong as the data set,experiments are performed using Bayesian classifiers,and accuracy and recall are used Evaluate and compare with F value and other indicators.The experimental results show that the improved chi-square statistical algorithm using text has a better dimension reduction effect than the traditional chi-square statistical algorithm and improves the accuracy of text classification. |
