Design And Implimention Of Access Service System For Massive Data

Research on massive data has been emphasized by people with increments of global data quantity and data-intensive applications. The storage of massive data from remote sensing satellites needs strong backend storage service, which has stronger capacity and security in handle massive data, to offer higher quality service. Local File System cannot meet the needs of massive data, so Distributed File System gradually been recognized.In order to solve the issue of increasing performance on access for massive data storage and data management, this thesis design and implement a system for massive data access and elaborate processing massive data real-time and the management of massive small files.The main contributions of this dissertation can be concluded as the following three parts:(1) The architecture of access service system for massive data from satellites is designed. Based on the actual business, this paper describe the characteristics of several of storage architectures and Distributed File Systems, test and analyze access performance on some Distributed File System for data of the specified size.(2) A solution of real-time processing for mass data is designed. Based on the technology of memory database, this paper respectively use Timesten and Redis memory database for structured and unstructured data to achieve process. Then prove that the scheme can satisfy the needs of real-time processing of massive data by simulation experiments(3)A scheme of dealing with massive small files is proposed. Analyze the impact on the file system management and storage performance due to the excessive block the massive occupied. Propose a scheme of merging small files by the method that database records the metadata of small files. Finally, make program and test to prove the feasibility and effectiveness.