Streaming Data Processing System Design And Implementation Based On Mobile Communication

With the increasing popularity of mobile devices,massive mobile communication data is collected.These data contain a large amount of time,space and other information,which can be used as a reference for major decisions of the public security department.However,the existing systems are unable to meet the normal business needs of increasing mobile communication.Firstly,the timeliness of the system is poor,and the index calculation cannot be completed within a prescribed time.Secondly,the system adopts hard coding mode and has poor scalability.Therefore,this thesis proposes a general solution for streaming data processing system in the context of mobile communication.Firstly,after fully studying the development of real-time business processing technology and the characteristics of real-time data sources of operators,it is determined to use Kafka as data cache middleware.Secondly,according to business requirements,the data processing part is divided into real-time ETL module and rule engine.Modules are used for data conversion and matching respectively.Finally,in order to provide second-level data result stream query capability,the HBASE component is used as the result database.The main work of this thesis is to complete the design and implementation of real-time ETL module and rule engine module,and test the system with real-time monitoring of human traffic as an application case.Firstly,this thesis builds a set of SQL-like parsing engine for streaming data ETL on distributed distributed processing engine Storm.The business personnel only need to submit SQL-like statements in real-time ETL module.It can complete the operations of data cleaning,filtering,aggregation,conversion,etc.Secondly,this thesis designs and implements a rule engine for data matching.Users only need to define rules and operators to get the required data.The design separates the rules from the specific implementation,which greatly increases the reusability of the code and the flexibility of the business.Finally,this thesis through the design of SQL statements,rules and operators on the system to complete the real-time monitoring of human traffic applications,and completed testing of the entire system.