News Thread And Theme Detection

The internet is the main source for people to gather information and know the world now. It has the same problem of "information overload" or called "information sea" as the other internet resource. The search engines, which are successful applications of information retrieve --, have taken the first step to help people to find their required information. With the aid of many news web sites, people can get news rapidly. However, when hot events or break-in news appear, it's not enough for people to understand the whole event if only list the related news by time. How to organize the news and find the structure of topics is a research hotspot. The aim is to help people to know sequence, cause and effect of events.In this thesis we first give the definition of the topic structure, we define: a topic consists of several developing threads; every thread has several themes in it; There are two relations between event themes: causation and elaboration; different theme has different importance. According to this definition, we propose a method to get the thread and theme of a topic based on two-level clustering results. The first clustering uses single-pass method based on Named Entities recognition. In this clustering, we propose a new hybrid linkage method , which combines single-linkage and complete-linkage. The second clustering method is NMF (Nonnegative matrix Factorization) based clustering. Then we calculate the relevance between themes by comparing their content and check the co-appearance. At last we calculate the importance of each theme based on the number of news belong to the theme and the number of related themes.We implement a prototype which combines the RSS technology and open source full text search project: Lucene. Our experiments show that the clustering methods we proposed are very suitable to our system.