Research And Implement Of Distributed Massive Small File Storage Access Optimization

With the rapid development and increasing popularity of the mobile Internet,people are gradually entering the era of the Internet of everything,generating massive amounts of mobile data every day.Especially short video social APP and e-commerce APP that have been rising in recent years.These APP generate a lot of short videos and pictures every day,the size of their data is relatively small,mostly around 10 M.However,the current distribute file storage systems such as HDFS and GFS are designed for large files.When faced massive small files,their performance will be drastically reduced,and even fail to provide service.Therefore,designing a massive files storage access system is a hotspot in the current research of distributed file systems.This paper takes the FastDFS that is specially designed for small file storage as the underlying storage system,proposes a new optimization scheme for massive small files access storage,and performs a series of optimizations,which include: 1)For this problem of the high concurrent access in the short video social APP,this paper proposes a new small file access load balancing algorithm.The algorithm dynamically adjusts the weight of each node according to the performance of each node in the backend server cluster,and fully considers the dynamic usage of the back-end server resources.2)The number of inodes in the local file system is limited,and the number of small files stored will be limited,so that disk space cannot be fully utilized.Based on the perspective of short video APP and e-commerce APP,this paper improves the small file storage process and merge algorithm of FastDFS according to the user attributes of short video files and product images,and merges the files of the same attribute into one file,so that sequential disk reading can be used to speed up access to files.3)In view of the shortcomings of the FastDFS record file synchronization process,this paper optimizes it by periodically generating checkpoint files,then synchronizing the checkpoint files to other nodes,and finally compressing the log file with the checkpoint file.For the above optimization methods,this paper carries out experimental simulation.The experimental results show that the optimization method of this paper has 11% performance improvement when dealing with high concurrent write files and 25% performance improvement when reading files.Finally,this paper implements an optimized distributed massive small file storage access system,which can be used to store a large number of short videos and image files,which can make up for the limitations of FastDFS.