Research On Massive Picture Storage Technology

With the rapid development of the Internet and the rapid growth of various information resources,various industries will generate massive amounts of information at all times.As one of the information sources,pictures have the advantages of convenient transmission,rich information,and intuitive feeling.The number of pictures has grown exponentially.The number of pictures stored on sites such as Taobao,JD,and Weibo,which are well known to the public,has reached the petabyte level,and is still growing.Faced with such rapidly growing picture resources,traditional file storage architectures and retrieval technologies have gradually exposed their shortcomings and cannot meet the current storage space and processing time requirements of massive picture resources.Therefore,finding an efficient picture storage technology that can satisfy high concurrent access has become an urgent problem in many information systems.This thesis analyzes and compares the current representative domestic and international massive picture storage solutions,and selects the object storage structure and distributed architecture to solve the above problems.Based on the analysis of the storage characteristics of massive picture data and the study of object storage structure,this thesis proposes a mass picture storage framework MISF based on the object storage structure.The framework takes the object storage structure as the core,and the underlying layer uses a distributed architecture to ensure high capacity and high throughput at the physical level.It uses virtualization technology to hide the tree-level hierarchical structure of the underlying file system and map it into flat storage space.In addition,a new data organization structure is adopted to organize the picture data compactly,and an internal index file is generated for quick access by clients.The research content of this thesis mainly includes the following three aspects:Firstly,by studying the organization and characteristics of the object storage structure,a mass image object storage model based on the object storage structure is proposed.The bottom layer of the model uses a distributed architecture to provide physical support for high capacity and high performance.At the same time,the entire system model separates the control path from the data path to improve system throughput.In addition,the virtualization technology is used to hide the underlying distributed architecture and file hierarchy,and the storage space is virtualized into a flat structure,so that users can more easily and quickly obtain picture data.Secondly,by analyzing the data structure of the B + tree,an on-demand loading indexing strategy based on the B + tree is proposed,which not only solves the metadata management problem of massive picture data,but also provides technical support for the underlying virtualization.At the same time,the UID of the picture object is designed so that the physical storage addresses of similar pictures are as close as possible,thereby improving query efficiency.Thirdly,through the analysis and research of the current typical cache replacement strategies,a cache replacement strategy of a custom value model is proposed,which fully considers the spatial and temporal locality,long-term popularity,and short-term popularity of massive picture objects.Improve the cache hit rate and the system query speed.This thesis designs a prototype system based on the researched massive picture data storage framework,and conducts experimental analysis on the prototype to verify the usability and efficiency of the prototype system,and illustrates the feasibility and effectiveness of the storage framework.