| Due to the rapid development of big data technology in recent years,the amount of data has grown bloated.This poses a great challenge for data storage and access technology.In response to this situation,many big data processing systems have emerged.Currently,the Hadoop Distributed File System(HDFS)is a leader in distributed data storage systems with high fault tolerance,high scalability,high throughput,and high reliability,and has become the most popular distributed file storage system in the industry and academia.However,because of the architectural features and the unique read-write mechanism,HDFS only performs well on reading and accessing for large files.It has natural deficiencies in dealing with large-scale small file scenarios: 1)The master-slave architecture and the storage mechanism for storing files make it possible to bring huge memory consumption to the master node of the system when storing a large number of small files,thereby reducing the file storage efficiency and 2)frequent file access operations bring lots of requests and calls between different nodes,so that the file access efficiency is reduced.Among the existing researches,the main ideas for solving the problem of small file access efficiency of the HDFS platform are as follows: To improve the storage efficiency,the way is to reduce the number of files is achieved by combining files,thereby alleviating the memory pressure of the master node;For the improvement of access efficiency,the cache mechanism is added to the HDFS architecture is a way to reduce the communication spending between nodes and improve access efficiency.Based on previous researches,this paper has developed a method and technology for optimizing the access efficiency of small files in HDFS to improve the access efficiency of small files in HDFS.The main contributions are as follows:(1)Designed and implemented a small file merge storage strategy and mechanism based on file correlation and file distribution.First,the two factors that affect the quality of file merging storage are summarized: the correlation between files and the distribution of files after merging.By analyzing a large number of historical access logs,the degree of association between files is obtained and quantified.In combination with the quantized results,the small files are merged.The quantified correlation degree and file size distribution are comprehensively considered and weighed during the merge process.(2)Design and implement a layered cache strategy and mechanism based on the resident value of small files.Considering multiple factors and attributes,the current file's cached resident value is defined to provide a basis for cache replacement.In order to manage the life cycle of files in cache,the cache space is proposed to be layered and the structure of data item in cache is designed.The resident value of part of files in the cache is calculated in a timed manner.The resident value will be basis of file sorting and file removing,which provides space for new files.(3)Add service nodes to the HDFS architecture to optimize storage and access efficiency.A series of performance evaluations were performed on the optimized storage and accesses of small files.Including the overall time cost of file storage,memory consumption of Namenode,space locality after merge storage,cache hit ratio,and file access time cost.The experimental results show that the efficiency of both the storage and access of small files has been significantly improved in the HDFS system optimized by this paper. |
