Research And Application Of Distributed Index In The Cloud

Currently, in the distributed data warehouse based on MapReduce framework, forrange queries, it is required to traverse the entire dataset to get the right data regardlessof the size of returned data. This leads waste a lot of I/O cost. In this study, we presenta novel tree based index scheme for efcient indexing and serving large datasets in thecloud. It incorporates and extends the functionality of distributed system to create afully parallel index system. The system is focus on distributed data warehouse in thecloud, and our contribution in the design and implementation can be summarized asfollows:Firstly, we apply the non-distributed tree-like index to a distributed system. Tree-like indexes are very efcient on data retrieving, and distributed systems can providepowerful computing capabilities. We combine these advantages to make C-index playa powerful role in data processing on large datasets. Then, to reduce the I/O overhead,we change the index’s storage structure, and make it suitable for distributed storagesystem. These changes include the node structure of the tree and the fle structure ofthe index. This will make C-index use less I/O operations to get the data. In addition,we add an cache mechanism to improve our system. This reduces I/O time to someextent. Last but not the least, for the system implementation, we make the C-index hasa low coupling between modules, in order to migration between diferent platforms.WedoseriesofexperimentsforC-indexunderdiferentplatforms,includingindexcreation test, one-dimension or multi dimension query test, KNN query test and cachetest. The result of these experiments fully illustrate the C-index has great usability,efciency and portability.