The Research And Implementation Of Web Text Classification Based On Web Text Segmentation

Internet has become one of the most important information resources. With the enormous growth of information on the Web, how to get the useful information from this enormous data is becoming an important research topic. In order to effectively organize and analyze the massive web text resources, the data mining technology to web text is becoming more and more important. Web text classification is one of the most important research topics. The exist of the noisy content block and the semi-structural feature of the web page makes the web page classification have some difference to the traditional text classification.There are three steps to the text classification based on machine learning, which are text representation, classification methods and the evaluation of the classifier. Vector space model is the most common model of a document, which is influenced by feature selection and reduction. The theory of Bayes and support vector model is usually used in the construction of a classifier.Most commercial web pages contain informative block and some noisy blocks such as advertisement block, navigator block and copyright block, obviously the latter has a bad infect to the web page information processing system such as information retrieval, web page classification etc. This paper proposes an algorithm which can identify the informative block based on some heuristic HTML tags and the frequency of a block appearance in a web page set. The experiment shows that our algorithm has a better result than its competitor.An Chinese web page classifier is implemented based on the blocking algorithm. The experiment shows that the classifier has better precision.