Key Technology Research And System Implementation Of Distributed File System Adapted To Massive Small Files

Traditional storage systems are not capable to store huge amounts of files in different sizes. Therefore, various kinds of distributed file systems have been designed all around the world. This paper tries to figure out the storage problems about massive small files, and then analyzes data routing policy and architecture of distributed file system. Three problems about designing distributed file system to handle massive small files has been highlighted.To avoid the conflict among disaster recovery, capacity expansion and high-speed operation in metadata processing, this paper proposes a combination of in-memory distributed database and traditional database. By storing metadata in in-memory distributed database, this system could not only efficiently store huge amounts of small files but also could get rid of the memory constrains on the number of files. Further, this system effectively supports disaster recovery by backing up the database on SSD.Moreover, to achieve the coexistence of small files and large files, this paper proposes to merge small files and split large file. While optimize storage problems about massive small files, this system would not affect and even improve the storage of large files. Besides, this system has better I/O performance in large files. Thus, this system is highly available.In addition, to balance the load on storage nodes, this paper comes up with a dynamic data routing strategy based on peer clustering architecture. Each copy of the data can be distributed into the storage nodes in different status, presenting strong consistency. Besides, this system is able to solve the single point of failure effectively, and also supports disaster recovery and capacity expansion.Based on these three key technologies, this paper designs an architecture of distributed file system adapted to massive small files, and explains the mechanism of storage, capacity expansion and disaster recovery. Then, each component of this system has been designed and implemented.Finally, experiments has been taken to test the distributed file system implemented by this paper, including performance about storing file in different sizes, disaster recovery and capacity expansion. The results show that this system can effectively manage the vast amounts of files in different sizes.