Key Technologies And Applications For Distributed Bitmap Indexing For Scientific Data

The volume of scientific data has been increasing dramatically over the past decades.Due to its large data size,it becomes rather hard to efficiently organize the archived scientific data so as to provide fast data access capability.The indexing techniques,especially the multidimensional indexing techniques which can handle complex data,are heavily demanded,provide the ability for fast data access and thus have been playing a critical role in many modern scientific applications.Because scientific data has a large amount of data,resulting data in fast,no update or delete operation,the majority of the characteristics is numerical data type,is very suitable for using bitmap index to manage and retrieve data.Most bitmap indexing techniques build indexes under a single node,which has the problem of slow indexing,poorly scalable storage.Scientific data stored in the distributed storage system,so the field of scientific data research urgently need a distributed environment in the rapid construction of the bitmap index method.In order to meet this requirement,this paper designs a method to speed up the construction of bitmap index based on distributed environment.The bitmap indexes are constructed on each node,and then the nodes are co-processed for merging into the global bitmap index.A system is designed and implemented to ensure the consistency of multiple copies.When using the bitmap index query,it is necessary to ensure the data consistency of the index file.The design of the query to ensure the consistency of the index data method.In the process of running the system,if the storage node downtime or data file damage,will affect the normal use of the system.In order to improve the system stability and data security,the data fault tolerance system in the distributed environment is designed and implemented.When the storage node is down or the data file is corrupted,the lost data can be retrieved by redundant copies.The main research contents of this paper include the following four aspects:(1)Using the distributed environment to speed up the bitmap index construction;(2)Designed and implemented a system which can ensures the data consistency of bitmap index;(3)Designed a method for checking and recovering data consistency in the query execution;(4)Designed and implementation of a distributed environment in the data storage fault-tolerant system.