Research And Implement Of An Adaptive WebPages Information Filtering System (AIFS)

The main object of WWW based information filtering is webpage, the most important expression means of which is text. Nowadays, the restriction of the widely used webpage information filtering system is keywords,net ports and URL given by administrator manually, and these systems simply match the information by character string. Such filtering mechanism with poor recall ratio and precision is obviously too clumsy .In order to fulfill the need of efficient information filtering system, this paper introduced an intelligent information filtering system based on natural language process. The component element,current employment,critical technology,existing problem and development trend of IF(information filtering) are presented .We designed a system named AIFS as an experiment, which sniffs and filters certain theme knowledge of Chinese webpage through text processing. Four major applications of artificial intelligence —Knowledge Acquisition,Knowledge Representation,Natural Language Understanding and Machine Learning are combined closely in the system to achieve these goals. The key technologies used in AIFS are devoted in detail in this paper. The chapter about net data processing introduces WinPcap and describes the detail how we get text data from Ethernet data frame. Taking Maximum Matching,Retrorse Maximum Matching,Maximum Probability and Vector Space Modal as examples, The chapter about text processing explains some algorithms about chinese segmentation and text representation, and an improved algorithm of web document representation based on VSM is put forward. As machine learning is in the key position of IF, especially in intelligent system, the chapter about adaptive processing shows practical application, where machine learning by GA is carried out for AIFS.