Research And Application Of Stream Data Processing Based On Memory Calculation Over Flight’s Big Data

Large aircraft flight tests, could generate vast amounts of test data. Traditionally, data offline process is inefficient, leading to delayed test cycle, and increasing flight costs. To be able to efficiently process and analysis the data, a flight data processing system based on a stream data processing of memory computing is designed and implemented in this thesis. The flight data processing system in this thesis, uses real-time data processing approach, relying on the efficiency of in-memory computing and stream computing, significantly improves the efficiency of data processing. In addition, this thesis presents a dynamic memory allocation method based on Markov and a stream data processing method based on queuing by time series, and combines Spark distributed memory computing framework, stochastic processes and queuing theory and other related knowledge, improving the original handling of memory computing and streaming computing, so that could accelerate processing and transfer speeds of large flight data, as well as enhance the value of the data.This thesis proposes a dynamic memory allocation method based on Markov, by using multiple Markov chain, and according to state transition probabilities,so that obtains the highest probability of converting the circumstances to predict the most likely demand of memory size, thus will save query time of memory allocation, reduce memory fragmentation rate. This thesis also proposes a stream data processing method based on queuing by time series, through making the timing data stream queueing according to the timestamp, so as to be treated in accordance with the requirements of the sequential processing flight data, also improve the processing efficiency of the flow of data.The flight data processing system based on a stream data processing of memory computing designes and implemented in this thesis, includes four major subsystems: data acquisition subsystem, data preprocessing subsystem, the data analysis subsystem, and integrated management subsystems. Data acquisition subsystem is mainly responsible for the airborne collection of raw data, combined with a timestamp, forming a series of raw data; data preprocessing subsystem is mainly responsible for the original data format conversion, through the frame format conversion, quantities conversion and stream data conversion, ultimately forming the desired stream data; data analysis subsystem is responsible for data collation and numerical computation. It can be combined with various algorithms, complete the training data and calculated data, and finally get test results; integrated management subsystem includes system development, system management and data management in three areas. The system can be effective and comprehensive data management and use, as well as secondary development.In this thesis, the flight data processing system combined to Spark, which is a distributed memory computing framework, takes advantage of Spark for massive data computing, and uses a dynamic memory allocation method based on Markov to enhance efficiency of memory allocation in Spark memory computing, as well as uses a stream data processing method based on queuing by time series to enhance the efficiency of data processing timing stream in Spark Streaming, therefore accelerating the speed of real-time processing of data.