Design And Implementation Of Police Communication Data Processing Platform Based On Big Data Technologies

In the era of big data, with the growing amount of data, the size of storage and computing clusters gradually expanded, hundred thousands of cloud computing environment is not uncommon. The problems clusters now needed to solve are not just high performance, high reliability, high scalability, but also the easy maintenance and data sharing within the platform, and many other challenges. Excellent data processing platforms provide rich sound business applications, and centralized management of each component of data platform to facilitate daily monitoring system operation and maintenance personnel to improve the efficiency of operation and maintenance, and can feedback system status to the system developers. Big Data era of business and operation and maintenance will be closer together.Big data has begun to affect and apply to all areas of human commitment to the development of the field of public security is no exception. Massive structured big data technology based police communication data processing platform to massive structured and internet communication data generated by various components, semi-structured log data for the study, design and implement a business system integration and operation and maintenance system, integrated real-time data stream processing engine, big data management engine, distributed memory computing architectures and data exchange systems solutions.From the Public Security Information actual work starting, real-time data streaming interfaces, design and implement a real-time data stream processing engine, for different operators, real-time communication of data according to different signaling decimated the province’s public security departments to provide mobile communications, After cleaning, filling and other steps related to private data warehouse loading, and integration of GIS and WEB system, on multiple dimensions of space, time, number, etc. of communications data query and analysis. In order to achieve high reliability communications data processing platform, easy maintenance, and data platform between the internal components of efficient data exchange, design and implementation of the operation and maintenance system, by integrating data collection tools, distributed message queues, and full-text search engine distributed memory computing architecture for data processing platform for communications log data collection, processing and analysis, the system in the Jinan Municipal Public Security Bureau has been successfully applied in the "police training" played an important role in providing for the analysis of the case based communications Efficient analysis of large data, but also greatly improve the efficiency of system development and operation and maintenance personnel.Firstly, data center construction of public security under the massive data and log data communications application environment subject background research, analyze the situation of public security needs of large-scale data-based communication data processing platform and the current main areas of public safety data integration platform. Second, based on needs analysis, through pre-research and technical architecture selection, gives an overview of structural design of public security communications platform business data processing systems and operation and maintenance systems, including system architecture and technical architecture design, system architecture design of the main press functions each component definition data processing platform, the starting point is that the technical architecture design system scalability, high availability, maintainability, the existing mainstream open source big data technologies in the research, describes the ELK Stack, Spark and integration of Hadoop Ecosystem architecture by the data acquisition module (Flume, Logstash), distributed message queue (Kafka), Distributed Data Stream Processing Framework (Spark Streaming) composed of real-time data stream processing engine of the design basis, and by the distributed column data warehouse (Hbase) and full-text search engine (Elasticsearch) consisting of large data management engine design and characteristics. Third, given the detailed design and business systems and operation and maintenance system of each module, based on the detailed design of the given implementation and testing of each module, including the use of public security to provide real-time data stream interface for real-time data stream handler development, program development related backfill learning application development and data flow based Spark number of relationships, and the development of other software modules.Finally, to build and application production environment for detailed analysis, and further improve the system’s recommendations.