Research On Small File Storage Technology For WEB Application

With the development of science and technology,the Internet industry has developed rapidly,the country also vigorously develop the Internet +.Including large-scale servicearchitecture,server cluster,cloud computing,e-commerce,social communication has become the most popular topic of science and technology.The development of these technologies brings the massive data storage.The processing of large data is represented by Hadoop's HDFS,and universal file storage is represented by Google's GFS file system.However,when we face the massive picture in the mobile Internet,becauseinformation issmall filesof a few kb to tens of kb,their storage performance cannot meet the requirements.Most of the common file storagesare based on the GFS architecture design,GFS architecture belongs tothe typical centralized scheduling file storage,through the central node Master to complete the system's metadata storage and overall scheduling.The Master maintains the metadata information of all the files and is also responsible for scheduling read and write requests tothe specified data node,data service node registration,and consistency repair calls,etc.,Which leads to the problem of massive disk space when macros are large for small files,a serious waste of disk resources,and all requests need to master to schedule,the entire system will be unavailable when the Master fails.In order to solve the shortcomings of the general storage system in the small file,mainly for the following three aspects of in-depth study: first for the massive metadata bring theproblem that disk space is wasted,through building the file storage key value,the file metadata information is curedon the key bya certain agreement,non-metadata storage is realizedby this way indirectly.Then putting forward a non-central architecture,whichprovides the service node information through the service node that the registration center to record,such as IP,ports and so on.the system is divided into Server-Client structure,Client is the basic component provided to the business side,and the business system uses Client to carry out data transmissionwith the Sever,through the Client integrated load balancing,protocol analysis,data transmission and other modules,the business systemreads the service node on the registration center and local cache,through using the monitoring mechanism,to refresh the local server listreal-time,the business systeminteracts with the Server aloneby the Mechanism,and without having to access the registry more than once,even if the registry cannot be accessed for some reason,which does not affect the Client to access the server ina short period of time,really no center structure,when any nodehas failed in the system,which has no effect on the availability of the entire system.In order to solve the problem that a large number of small files causethe growth of operatingsystem metadata,and the IO performance problems is caused by the head seek and the disk rotation,a merge queuing mechanism based on write queues is built,to make full use of the IO performance caused by the continuous storage space ascension.Finally,the weight-link dynamic weighting list load balancing algorithm is proposed by optimizing the system load balancing algorithm.At the same time,a triggering consistency detection mechanism is proposed,which is optimized locally from each technical point,and then the quality change is improved from the whole,which enhances the overall performance of small file storage.Finally,compared with the open source WeedFS file system for concurrent experiments,the stability of the small file system optimized by this paper is much higher than that of the WeedFS system,and the performance is higher than that of WeedFS.