The Research Of Theme Analysis Technology On Short Texts

The rapid development of internet and communication technology not only promotes the spread of short texts such as micro-letter and microblogs throughout the world, but also has brought a profound influence on the people’s daily life. The large scale of information makes people hard to acquire information effectively. It demands us to find a way to mine the topic information from the large scale of internet short texts. We call this process theme analysis technology.The work of this thesis mainly focuses on the research of hot topic detection algorithm. The work is as follows:Firstly, the project puts forward a topic detection algorithm based on improved BM25algorithm. It improves the readability of topics. It includes a term similarity based on BM25algorithm, bursting term detection and a keywords clustering algorithm.Secondly, we proposed a LDA-BM25algorithm which combines the LDA model and out topic term-BM25algorithm. It improves not only the readability of the topics, but also the independence of topics.Finally, this thesis proposed a design scheme of distributed processing system based on large scale short texts. It includes distributed preprocessing, distributed storage and efficient topic detection technology.