Algorithm And System For Real-Time Bursty Event Detection Based On Public Opinion Analysis

Using big data technology to improve government governance and public livelihood services is one of the important tasks for implementing the national big data strategy.With the rapid development and popularization of informatization of government services,government services have accumulated more and more data.Among them,the 12345 public service hotline platform has accumulated a large number of public opinion data that need to be analyzed.It is very essential and important to improve the quality of government services by using intelligent analysis methods to mine potential events from these massive public opinion data.However,research and application of bursty event detection in public service hotline are still scarce,and it is inappropriate to use the existing bursty event detection methods based on social media for detecting bursty events in public opinion data.Therefore,considering the above challenges,an algorithm for real-time bursty event detection,called RAEDetection,was proposed to overcome such obstacles.The primary contributions of this paper are highlighted as follows:(1)We proposed a bursty event detection algorithm called RAEDetection based on the public service hotline.First,an improved incremental Kleinberg model was proposed to identify the bursty words from the real-time data stream,which provides clues for the detection of bursty events;then,an algorithm based on hierarchical semantic analysis was put forward for detecting candidate bursty events,which gradually refines the clustering results of the public opinion data from the topic semantic layer and the event semantic layer to recognize candidate bursty events;with candidate bursty events as clues,the event region tree is constructed to accurately recognize the regional patterns and the specific geographical area involved in events;finally,we propose a real-time bursty event detection and trend analysis method to dynamically discover and track the development process of bursty events.(2)In order to improve the efficiency of the algorithm to process large-scale public opinion data,we design and implement the parallel bursty event detection algorithm for RAEDetection based on Spark.The parallel bursty event algorithm can effectively improve the efficiency and scalability of the algorithm while maintaining the detection accuracy.(3)Based on the algorithm proposed above,an efficient system for real-time bursty event detection was implemented in this research.The system integrates online realtime bursty event detection and offline full-scale bursty event detection methods,which can detect bursty events from real application scenarios in a timely and accurate manner.The proposed system has been successfully applied to the civic public service hotline platform of Jiangsu Province.(4)Based on the experimental evaluation results on the datasets of large-scale civic public service hotline in real-world application scenarios,the RAEDetection algorithm and the implemented application system proposed in this paper can effectively improve the accuracy of bursty events detection.At the same time,it has good computing efficiency and scalability,which can meet the needs of real-world application scenarios.This research won the “Best Application Paper Award” of the CCF Big Data 2019.