Research And Application Of Stream Processing For Railway Operation And Maintenance

At present,we are in the era of big data and the railway industry as well.China has entered the world's leading level in the high-speed rail industry and has mastered many core technology of high-speed railway.In the high-speed railway operation and maintenance process,a large amount of data has been accumulated by advanced sensor technology,data acquisition equipment and computer storage devices.How to analyze and deal with these data is of great significance to the maintenance of high-speed railway.In the face of the characteristics of large capacity and diverse and rapid accumulation of the data in high-speed railway operation and maintenance,the traditional data processing methods have been difficult to effectively deal with the problems,mainly reflected in the time-consuming and they are difficult to meet the demand on timeliness.In this paper,we will explore the technology and method based on stream processing,and applies it to the data analysis of high-speed railway operation and maintenance.In this paper,the characteristics of railway operation and maintenance data are analyzed,after that the differences between flow processing technology and traditional processing technology are compared,and presents a solution for data processing based on streaming framework.Based on the Spark Streaming framework,the paper realizes the data processing system of log data in train operation and maintenance.The influence of various parameters on the processing is studied and optimized.The main work of this paper is as follows:(1)Based on the analysis of the core technology of stream processing framework,according to the data characteristics and requirements of the railway operation and maintenance process,the solution of operation and maintenance data based on the flow processing framework is proposed.At present,the flow data in the railway industry is growing rapidly.However,the traditional data processing technology is still used in the process of railway maintenance and operation,and the timeliness of data processing is not strong.For this reason,this paper proposes a scheme based on stream processing technology,which solves the problem that the traditional processing technology takes too long to deal with data which grows fast.The experimental results show that the flow processing mode is greatly improved in terms of timeliness compared with the traditional processing methods.(2)After analyzing and comparing the common stream data processing framework,the Spark Streaming environment is set up,and the data stream processing system is realized.Firstly,a distributed flow processing experiment environment is built.Then,the memory based processing of the log files is carried out using the streaming processing framework.Then the key fields from the log files are extracted and saved them in the data warehouse.Finally,the interactive query tools are used to analyze the extracted data.(3)After the above solutions are implemented,the scheme is optimized according to the practical application scenarios,and the performance of the system is improved.First,the distributed message queue Kafka is integrated on the architectureIn this paper,we propose a stream processing solution and implement a stream processing system to meet the specific requirements of the operation and maintenance.It can quickly process the data accumulated during the operation and maintenance,and improve the efficiency of data processing in high-speed railway operation and maintenance,so that the parallelization of data reading process is realized.Then the parameters are optimized to improve the efficiency of log data processing.In this paper,the proposed flow processing scheme is experimentally verified.The experimental data used are monitoring data accumulated in the production environment.Different experiments are designed to compare with traditional data processing methods.Experimental results show that the proposed scheme can perform the processing of log files more quickly.The distributed system architecture has good scalability,and the performance of the system will be further improved as the number of nodes increases.The flow processing system realized in this paper meets the requirement of timeliness in operation and maintenance,and can quickly process the accumulated data,and improve the efficiency of data processing in railway operation and maintenance.