Research On A LDA-based Incremental Topic Detection Method

In the era of big data,we enjoy rich information resources,but are faced with information overload in the meantime.Competitive intelligence that can provide organizations with great competitive strength attracts more and more attention today,but its big volume and sparse semantics characteristics make it hard to analyze especially when it is quickly updated and continuously accumulated over time.British Telecom is an international telecommunication company faced with the challenge of web text topic detection.Traditional manual approach in competitive intelligence is not only time-consuming but also expensive,and batch processing will also bring computing and storage load to computer hardware.Therefore,BT needs an automatic incremental topic detection method to help employees quickly and accurately process web texts.This dissertation proposes a LDA-based incremental topic detection method which makes full use of LDA results and reduces dimension of massive texts via effective and reasonable compression techniques.Furthermore,in order to highlight the importance of new-coming texts,we endow the old texts a decay function so that our result can better reflect the tendency of topic focus.In order to verify the effectiveness and the feasibility of the proposed method,we experiment on real-word data provided by the BT Sales Department,which proves that our method can achieve a satisfactory accuracy and high efficiency when applied to large-scale dynamic web texts.What's more,after adding an exponential decay function to old corpus,our method can achieve a higher level of accuracy and keep high efficiency at the same time.The research not only makes contribution to topic detection theory but also has high practical value.