Design And Implementation Of Real-Time Data Processing System For Tickets

Theme of this paper is the design and implementation of real-time data processing system of tickets which originates from the company’s actual needs of big data department. The company’s core business is service ordering plane tickets. Online ticket ordering system will generate a lot of trading data every day, and stored data in the MySQL and log files. Since the sub-library and sub-table strategy, ticket transactions in more than 3500 MySQL databases involve nearly fifty thousand tables. In order to meet the needs of operators and decision-makers, a large number of data have been processed and displayed in real time by the system.The system is divided into four parts, namely, real-time data synchronization, data processing, data storage, and data presentation. In order to ensure real-time data acquisition, the system uses the Canal to complete real-time data synchronization of MySQL database. Depending on the storage and presentation, Data processing section divided into three parts, namely, Elastic Search(real-time full-text search engine) handler, isomorphic data handler and real-time processing program of Storm(distributed real-time streaming data frame). ElasticSearch handler is used to integrate ticket information with business rules. All the information set into one table and stored in the Elastic Search for real-time query of large amounts of data in any field. Isomorphic data handler in accordance with sub-library and sub-table strategy integratea different databases with the same table name information into the same table. The full amount of data stored in HBase. A real-time query of HBase specifies a field and does not support real-time queries of any field. At the same time, the data will be stored in two days to My SQL for message reporting. Real-time processing program of Storm is to achieve a real-time dashboard of ticket transaction information. The use of real-time computing power of Storm for online transactions statistics show the results in a line chart, tables and other means.Prior to the present system, the company is based on Hive off-line analysis system with large delay in the timeliness. Time-critical needs directly query the online databases and an additional logic to handle complex database issues. This system unifies all data sources. After the only integrated into the data warehouse, providing real-time data query and analysis be able to guarantee the accuracy, timeliness, and stability data. In the case of 200 items of per second, the system ensures that the delay in seconds and stable operation of more than one month.