Research And Implementation Of The Parsing Engine For Subway Log Data

With the development of information technology, the explosive growth of data volume brings new challenges to data processing. In the subway system, each subway line contains thousands of devices, each device will generate a large amount of log data while running. The subway log data has features of large amounts, multiple formats, multiple message formats and variability. How to parse the subway log data quickly and efficiently becomes an important problem to be solved in the operation and management of subway system. Subway log data requires a lot of codes to support the parsing job, the method of writing code manually can not cope with the ever-changing multi-format mass data, this method becomes the bottleneck when dealing with subway log data.To solve these problems, a data description language for subway log data is designed, and a parsing engine based on this language is developed.By analyzing the data description files which are written by the data description language, the engine can generate codes automatically and provide support for deploying these codes.Firstly, the conception of data description language and the technology of data parsing is introduced, and several solutions for generate code automatically are compared. Then, the requirement analysis section examines the subway log's features and the analysis process, and puts forward the demand of data description and developing a parsing engine. In the preliminary design section, the parsing engine based on the data description file is regarded as a code generation and deployment system.Based on the demand, the system develops structural design, modular design, architecture design and visual interface design. The system is divided into code generation module, code merging module, code application support module and code deployment module. In the detailed design and implementation section, the design of the data description language for subway log data is introduced, and the lexical, grammatical and semantic design of the language are given. We also provide some examples to explain how to use the language to describe the subway log data. Then, each function module is described in detail by the implementation of the module with the flowcharts and core codes. The performances of these modules are tested in the test section. Finally, we summarize the main achievements, and analyzes the shortcomings and prospects for the future work.