Study On Term Semantic Relationship And Its Application In Text Categorization

Text categorization is one of the basic tasks in information retrieval. With the explosive growth of web information, people have difficulty in finding the required information from massive information. In order to solve the so called"information confusion"problem, Research on text categorization gradually seemed to be more important.This paper design and implement a module-based scalable automated text categorization framework. We also did a comprehensive survey on each important step in the framework. Based on this framework, we bring up a method that integrating the term semantic relationship into classic text categorization task. This method can solve the inherent irrationality in the assumption of Vector Space Model that terms are treated independently. Meanwhile we show that the deep association between terms can be used to improve the result of our current experiment.Term semantic relationship can be obtained by using sentence parsing in natural language processing and statistical method in information theory. We presented the deep term relationship in the form of thesaurus which can make the document vector more informative and effective. When combined with the classification power of SVM, this method yields high performance in text categorization.We compare this technique with SVM-based categorization and other term relationship model on 20NG and Reuters-21578 dataset using the simple minded bag-of-words (BOW) representation. The comparison shows that our method outperforms others model in most cases.Finally, we bring out some future research on using term semantic relationship in information retrieval area.