The Parallel Loading Technology Of Column Data Index In Distributed In-Memory Database

With the high-speed development of the Internet industry and wide spreads of concepts like "Big data", "Cloud computing", the explosive growth of data in terms of large capacity, diversity and high growth, is challenging the data processing ability of modern enterprise. For the mature traditional database technology, the rapid development of computer network technology and the expansion of the industry application requirements makes the traditional database technology faced new challenges in many situations. In this situation, the concept of distributed database and in-memory database arises at the historic moment. Compared with the traditional centralized database, distributed database has a good flexibility and extensibility, which has more advantages on the performance and reliability when dealing with huge amounts of data. In-memory database stores data in RAM(Random Access Memory) rather than disk, in this case, the performance of read and write is much higher than disk access, which greatly improves the performance.Compared to the disk, RAM is a very valuable resource, so in most scenarios distributed in-memory database in the business of production is mainly used as computing platform instead of primary database stores data. The massive data is still stored in traditional disk database. How to load mass structured data stored in the traditional disk into distributed memory database quickly is the first problem to solve.To solve this problem, in this paper, we provide a solution to quickly load mass structured data stored in disk to distributed in-memory database. First, against the original structured data, we provides a new set of quick index model, which can achieve efficient of data storage and query quickly. Then, convert the structured data to in-memoty data index with distributed system, and load the data index into distributed in-memory storage engine. In addition, this solution can support user preferences data filtering, and provide incremental data update.The main work and innovations as follows:1. Designed an in-memory database data index model, which implement high efficient data storage and quickly query. The index model based on column data storage, the original data can be compressed and provide quick query efficiency.2. According to the provided data index model, we design a fast parallel data loading scheme for distributed in-memory database, which will load external traditional structured data stored in database into the distributed in-memory database engine. This scheme can support user preferences data filtering, and then create data index. After this, load data index into the distributed in-memory database engine with certain strategy.3. Increment data update with certain strategy to synchronize data between distributed in-memory database system and the external data in the database, and solve the problem of data consistency.4. Adopts distributed system for large amounts of structured data loading in parallel, which can improve the data processing speed, alleviate the pressure of the nodes and the safety of the nodes. The distribute system can also improve the speed of data loading and stability.