Design And Implementation Of Communication In Massive Experimental Data Management System Client Communication

During the process of collection, storage and analysis of model test data in aerospace field, there brings a problem that the existing data processing and analysis method is inefficient and ineffective for larger scale of data, more diversified data types and much more unstructured data. The existing file system and traditional relational database are unable to meet the demands of real-time storage, query and extract. Besides, for that testers are located in different places, efficient data interpretation cannot be concentrated. This status quo is in an urgent need of a data management system to manager test data effectively and efficiently.The study is excerpted from an important cooperative project of Xidian University High reliable software lab and a Aerospace science and technology research institute. This paper focuses on the design and implementation of the data communication module of the GUI client in the massive experimental data management system. The client can adapt to the large amount of experimental data's processing and displaying, which brings a more convenient way to count and display graphics by using the data. It means that the client has the cross-platform function by using the Qt framework to write programs. The client, using a three-layer architecture design idea ensures a low coupling degree between layers and layers as well as a strong expandability.In the design of data communication module, first, we need to encapsulate the class of network communications in Qt framework. The business code can complete network access and data access by simple configuration. It builds the task queue and internet pooling connection while packages network communications and improves the concurrent ability of all tasks. It completes the abstract of access to network tasks to make the tasks can unified processing and scheduling. Furthermore, The transmitting of business logic tasks will not affect the tasks communicate by using asynchronous communication implementation of the user interface and the executing of following logical code. For that, in the design of communications, there are two types of special communication tasks, one is a higher frequency of update files synchronization directory structure, and the other is the transmission and display of massive data, In order to response to high frequency large volume of data communication we use the incremental updates, lazy loading and sampling data points to reduce the amount of data communication. Finally, batches of files are transfered by this communicaion function, which relies on the design of transter watcher, fragmented files and memory buffers packaging. The file transfer watcher split the functions of file transfer and its progress-update-feature to make it easy to manage batch-file-transfer tasks. In addition, the fragmented files transmitted through package. When writing a file to the disk, we use a memory buffer to reduce the burden on disk and the usage of network bandwidth.Massive Experimental Data Management System achieves results that data upload and parsing efficiency was improved 150 times, hundreds of millions of data can be displayed and plotted, real-time synchronization of tens of file organization structures include thousands of meta data information in each can be realized, file packaging transfer can be performed among 10 1-GB files or hundreds of 1-MB files. As so far, the system was put into service successfully. It solved short backs of inefficient data processing, cumbersome authority management and counter-intuitive data presentation during numerous experimental data management. This system was involved in the key researches and trial fires of rocket and missile during recent 2 years, which played a significant role in data management and achieved a satisfied result.