Research On Correlation-based Distributed Metadata Query Mechanism

Continuous development of information technology has brought about the harshrequirements for high capacity and performance in information storage systems. Alarge-scale storage system is becoming more and more extensive, and storage capacity hasincreased from TB (Terabyte) to PB (Petabyte), even EB. While using the massive storagespace, users find that data is becoming more and more difficult to search and manage.Existing metadata management methods have shortcomings in both scalability and queryefficiency.In view of these shortcomings, a correlations based metadata search method isproposed, which takes full advantage of multi-dimensional metadata correlations toimprove query efficiency. Metadata is aggregated with Locality Sensitive Hash (LSH).Global index is divided in the way of every global LSH bucket into small classes, not inthe way of whole LSH table into small parts of buckets. This partition method makes oursystem more flexible and alleviates data migration. An index is maintained hierarchically,which provides convenience for system management. Metadata are stored with local filesystems instead of database and updated periodic. Every query request is allocated to oneserver, which works as an agent of client. That is, this server is responsible fortransmitting query request, collecting query result and returning it to client.Our experiments show that this method of organization improves multidimensionalquery efficiency by an order of magnitude with respect to the traditional one-dimensionalsearch method. Our system’s response time increases linearly as it scales. At the same time,we propose an simplified version based bath index update strategy, which greatlyimproves index updating efficiency, compared with MySQL.