Real Time Web Event Aware System Based On Distributed Framework

With the increase in volume and dynamism,people cannot physically and mentally filter and select knowledge and information on their own.Under this circumstance,event detection provides a similar solution to the above data volume problem by partly mimicking humans that read text and integrate new information with what is known of the past to help people know the whole view or make decision.Nowadays,some mature systems rely on large-scale computing clusters with using steam and batch way to realize the application of large scale data.This thesis focuses on the aim of real-time access to event cluster results and fast query response.To achieve this,the system use stream-oriented computation to increase the overall processing efficiency and reduce the negative effect on cluster algorithm.In this thesis,based on above-mentioned requirements,we design and develop a distributed processing platform.The main work is as follows.1)With the analysis of the input and output of the event detection application,and the system is divided into three modules.2)In the storage module,we store the structured and unstructured data together,including NAF expression and two-level index data in MongoDB.3)In the processing module,under the condition of streaming and cluster processing scene,we have improved Storm topology scheduler original and developed computing components to meet the requirements of fault tolerance of incremental computation.4)Analysis and service module designed for different types of query response strategy,and design a closed cube query generation and improvement in the distributed memory environment in the background query.Finally,the usability and performance of the system are verified by the application of practical inspection and quarantine.