A Hybrid Storage Engine Based On The Architecture Of Read/Write Separation

With the increasing complexity of Internet business models and the growth of application data,traditional online transaction processing(OLTP)-oriented databases and online analytical processing(OLAP)-oriented databases can't satisfy diverse business needs at the same time,due to the differences between architectures and independence.Hybrid transactional/analytical processing(HTAP)-oriented databases have emerged as one of the directions for the development of the database systems.Traditional OLTP or OLAP-oriented databases use single storage for ”row-oriented”or ”column-oriented”.It is impossible to take account of the advantages of row and column storage formats(row-oriented storage is conducive to efficient transaction processing,while column-oriented storage can effectively optimize complex queries),so hybrid storage engine is one of the key technologies for constructing HTAP-oriented databases.How to be compatible with the storage formats and access interfaces of the two types of databases,to ensure efficient transaction processing and complex query analysis is a major challenge for the construction of hybrid storage engines.The distributed databases with the architecture of read/write separation have good writing performance and scalable storage capability.Combined with the advantages of this architecture,this paper mainly proposes a hybrid storage engine for HTAP applications.The main contributions of this paper are as follows.1.Solving the problem of data integration in HTAP-oriented hybrid storage engine.A row-column mixed data organization based on batch update is proposed.It integrates the management of the OLTP-oriented and the OLAP-oriented storage files.2.Implementing efficient transaction processing and data access methods under the hybrid storage engine.Leveraging advantages of the architecture,transaction processing is not affected by the underlying hybrid storage;A unified access interface for row and column data,a method of scanning column data based on row groups,and rule-based data reading strategy are designed to optimize complex query.3.Proposing a query expansion method of hybrid storage engine.It shares data storage files with other big data processing systems,and designs an independent extended component to enable the upper application to call the computing interface of the big data processing systems in the storage engine with SQL statements.It expands the abilities of query analysis,data mining and so on.4.Demonstrating the validity of hybrid storage engine in a database under the typical architecture.This paper carries out a series of benchmarks for the hybrid storage engine,including storage space,transaction processing,query processing and other application scenarios.Compared with the industrial database system,its effectiveness is fully proved.The hybrid storage engine proposed in this paper has some innovative and academic significance.It provides an effective solution to solve compatibility problems of data management,transaction processing and complex query analysis in HTAP-oriented databases.The work of this paper has practical significance for the research of HTAP-oriented database systems.