Research And Implementation Of An Approach To Store Massive Small Files For Urban Rail Transit Network

With the growth of urban rail transit network system,the demand to establish urban rail network data center has become increasingly prominent, so the urban rail network data cen- ter faces massive file data storage requirements. Urban rail line network file data including large files(such as video and documentation) and small files(such as images and reports), at present, large file storage technology has been mature, however, because of the metadata management, data I/O access pattern and data layout, small file storage technology has not been mature.At present, the Internet companies such as Facebook respectively developed mass small file storage system for specific application scenarios.However, compared with massive small file storage system for urban rail transit network,these systems have different design require- ments, so it is difficult to directly applied them in urban rail network data center.In this paper, according to the data characteristic and the data access method of urban rail transit network system, an approach of storing massive small file for urban rail transit network based on Fast DFS distributed file system and Redis key-value database is proposed,which includes: 1) Store the relevant small files into a same file in Fast DFS;2) Put the index of the big file,the offset of the small file in the big file and the length of the small file to form a global index, and store the key-value pair of the filename of small file and the global index into Redis; 3)Adopt data prefetching mechanism to prefech the adjacent files, and put prefetched files into the cache in the client;4)Use lazy method to delete small file by setting the file status value; 5) To implement small file modification, delete the original small file and append new small file to data file.The experiment was carried out function tests for small file merged writing, small file a- ppending, small file read, small file deletion, small file modification on mass small file storage system. The performance tests has also been carried on, and the experimental results show that when the prefetched file number is five and the capacity of client cache is 500 MB, small file read throughput is maximized; compared with Fast DFS distributed file system, the read throughput and write throughput of Fast DFS-Redis storage system increase by13.26% and 10.47% respectively, which significantly improves the small file access efficiency for urban rail transit network.