Design And Implementation Of Distributed File System For Large Scale Storage For Modern Service Industry

With the development of internet technology, the entire amount of information on the internet showed the explosive growth. Some researchers have paid their attention to finding an effective way to store and manage these data. Some solutions such as file servers based on Linux have been designed to store and manage these data. These servers were connected through high-speed local area network to form a cluster which offers efficient and reliable service to clients.After studying and summing-up the outstanding distributed file systems at home and abroad, combined with the project of key technologies of modern service industry platform supported by the Ministry of Science and Technology of PRC, we design and implement a distributed file system for large scale storage.Firstly, the thesis analysis's the system requirement from three aspects: user requirement, external interfaces and the performance and reliability requirement. Then, we propose overall system architecture which was composed of Master Server, Chunk Server and Client. Master Server is responsible for the management of metadata which consists of namespace, mapping of file onto chunk and mapping of chunk onto Chunk Server. Chunk Server is responsible for chunk IO operations. It can create, delete or replicate a chunk according to the Master Server's instructions. Client provides file operation APIs which include file creation, reading, deletion etc.Secondly, we give an introduction to the key technology of distributed file system which consists of execution program, cache design and load balance. According to the actual needs, we use of the share file system program and store the data in the memory of Client. Besides, write-through algorithm was used to ensure the data consistency and the performance measurement of load balance was carried out.Thirdly, we particularly introduced the detailed process of data interactive and the mechanisms of chunk clone, data validation and garbage collection were proposed to ensure the data availability and consistency.Finally, we present the class diagram of prototype system which contains metadata type based on IO stream, Master Server, Chunk Server and Client. The performance of upload, down load, load balance and fault torrent were tested in the end.