| With the rapid development of Internet, information resources in the network are growing exponentially, which results in the generation of the “information explosion†era. The quick proliferation of information brings new challenge to information retrieval. More and more research focused on how to find the information requisite conveniently and accurately from the tens of billions information.From a certain extent, search engine solves the difficult of information retrieve. When user enters search keywords, the relevant information will be found. But sometimes a search keyword may return hundreds or even thousands of results. These results are mainly presented to user in the form of title lists, the user still needs to browse the pages one by one to know if it is the desired resource. Summary is an overview of the contents of a document, including the document theme. If search engines can return both title and summary at the same time when retrieve, the time spends on browsing pages one by one will be reduced. For summary manually need much time, the automatic technology is presented. Automatic summarization means to use computer to extract document summary automatically, which is an important branch of Natural Language Understanding. Great progress on automatic summary technology has been made in recent years. This paper proposed a method based on improved TF*IDF algorithm and sentence similarity to generate summary automatically for Web pages.First, the paper introduces the meaning and necessity of automatic summarization. The problems and limitations of the existing technology are presented according to the study of domestic and foreign research status. The significance of the topic and the main content are put forward.Then, it analyses the related theory of automatic summarization, including the classification and methods. In view of the particularity of Web pages, the paper introduces the principle of Web de-noising and regular expressions. Several major technology for summary system to realize in this paper are discussed, such as Web pages text extraction algorithm based on block distribution function, subheadings identification method based on divide-and-conquer, the principle and classification of Chinese word segmentation, the improved TF*IDF algorithm and method to compute sentence similarity based on semantic and structural similarity.And then, from the perspective of software development, the design and implement of the system realized in this paper are described in detail, containing system function, database table design and sub-module design. For the key modules in the system, logic flowcharts and pseudo-code are given to deepen the understanding of its function and implementation.Finally, experiments on each module are carried out and the results are analyzed. Through the example, user interface of the system developed in this article is shown. Statistical experiment is done for the system. The experimental results show that approach to obtaining summary automatically based on improved TF*IDF algorithm and sentence similarity is practical and efficient. |
