Basic Education In The Search Engine Pages Document Feature Extraction Research

The fast development of Internet increases significantly the number of basic educational resources in webs. More and more people are using Internet to study and search engine has been playing a pivotal role in acquiring educational resources. Automatic sorting of large number of webs is desired in order to provide users with more effective educational data and increasing precise in search. Meanwhile, automatic sorting rests on feature distillation. This article is developed according to webpage feature distillation and several key problems involved are discussed systemically.More details of the first part are about the background of the study, and say something about the correlative document for the feature selection of the theory, and automate classification in summary. For the second part, we pay more attention on the feature selection of the WEB documents: First of all, we expatiate the sign pretreatment of the WEB document and expatiate the method of word segment, after that, we build a document collection for the basic education, then, do more search on the feature selection and weight calculation which are included in the feature distillation according to this groundwork. During the research of feature selection we compare the DF, IG, ECE, MI,X~2, WET and make a new algorithm named MIPW based on the experimental result, by the result of experiment we find that MIPW is more valid than MI. In the part of weight calculation, we compare TFIDF and algorithm based on entropy and find that TFIDF is more valid than others. Finally, we design and develop a webpage-feature distillation system which include the above algorithm of feature selection and weight calculation, now, the system is using in the Basic Education Resources Search Engine.