Design And Application Of Computing&Query System Based On Hypertable

With the development of big data technology, NoSQL database is becoming popular, the expectation for this kind of technology is more and more high. When the non relational database was originally proposed, just for the sake of the problem of mass data storage solution of stable and simple concurrent query. But the development as the data size increases and the data types of diversity, users began to care more about the relational operation of data. In order to obtain the relational operation’s querying&computing ability on NoSQL database, the usual practice is based on the actual situation, in addition to realize related system or algorithm as auxiliary. This kind of practice is not common and convenient, there is no uniform standard which may cause high learning costs. Therefore, the realization of how to get the relational operation and also get the properties of NoSQL database is facing the challenge, this paper is going to study and solve problems.This article take investigation and study of related open source solutions for the case, design and implement a SQL query&computing system based on Hypertable. Solve that keeping the performance of NoSQL database at the same time achieve relational data operation; using a SQL like language, more in line with the user’s habits, reduce the learning cost. This paper completed the following work:(1) The system introduced in this paper use the Hypertable as the core of the system and the underlying storage module; using Storm instead of MapReduce model which used in the existing solutions for the programming model, to realize the built-in function to expand query and calculation of Hypertable; design the system using the query calculation language HHQL and realize its parser to achieve the query computing tasks; scheduling, according to the query automatically constructing task code and execution; construct an easy to use, efficient distributed large data query computation system.(2) Investigate the current main distributed stream computing system--Apache Storm; by comparing the computational efficiency, the abundant degree and friendly programming interface etc., determine to use Storm as the calculation layer framework and operator model for the query computation system; and using the DRPC model in Storm, realized the calculation operator built in the system in.(3) The HHQL query language based on the standards Sql, designed for the query computing system. Through the development of JSqlParser, realized a HHQL language parser.(4) The design and implementation of the query computation task execution manager. Task execution manager is responsible for generating task execution code and the maintenance task instances during the entire life cycle, including the execution of a task, kill task and failure tolerance.(5) The design of the distributed system of master-slave structure, provides a remote procedure call development kit (SDK), realized the underlying transparent query computation request. That is, users need not operate directly cluster system, or implemented their own Storm operator. To solve the easy to use, cross platform, multi-user concurrent control problems.Through the above work, implemented a complete large data query computing system, make the expansion of the non relational database, provides efficient, easy to use tool.