Design And Implementation Of Real-time Alarm System Based On Spark

In recent years,the rapid development of computer technology and the rapid spread of the Internet,mobile Internet,data gathering between the amount of the data explosion of data storage and traditional data analysis techniques in the practical application bottleneck gradually exposed more and more,especially the analysis,data processing and related knowledge on the order of PB mining,intelligent search and other large data services.For massive data,using the traditional database technology not only in throughput and storage capacity can not meet the demand and poor reliability,fault tolerance,low efficiency and high cost,so the construction based on the big data processing and analysis platform,the public security organs are in urgent demand.The birth of Hadoop is a good solution to the analysis of massive data storage and offline demand,while Spark uses Resilient Distributed Datasets(RDD)and memory operation mode,which has a good rapid analysis of large data mining,constructing real-time computing framework based on Spark Spark Streaming is calculated in real time with high efficiency and reliability analysis of flow capacity for the massive high-speed data.The combination of Hadoop and Spark as a public security data analysis and storage platform,can effectively support the police handling cases.Based on the construction project of public security network work platform,this paper expounds the design and implementation of the real time early warning function module system.Through the analysis of the security business demand for real-time data processing,based on fully research on real time computing framework Spark streaming and relevant technology,design and implementation of a real-time warning system based on Spark,provide real-time data analysis services for public security work.