Column Storage Design And Query Optimization For Nested Records

With the rapid development of the information society,the word "Big Data"exists in all aspects of our lives.With the using of "Big Data" in the internet,we can inquire for massive data online,shop on shopping sites and instantly share things on social networks etc.Nowadays,with the explosive growth of Big Data,the requirements for data access and updating are becoming more and more complex.Also,the challenges faced by database management systems are becoming more and more difficult.The traditional row-based storage database cannot satisfy those various requirements.In the context of this development,column-based storage emerged.Because of the columnar storage mode,column-based storage systems can perform complex queries more quickly and efficiently than a row-based one.The data of the same column being stored together can better use various compression technologies to save storage costs.The column-based indexing technology also makes it more efficient to access and manage databases.By the strengths above,column-based storage is more suitable for the applications of OLAP(Online Analytical Processing).This paper takes column-based storage and nesting structure as two major characteristics.Based the above features,paper makes optimization on storage,query,and updating.In terms of data organization,paper studys the storage structure of Trevni and then optimizes it.For example,the intra-block index is added to the columns with variable length data,and the block bitmap is designed for the UNION type(which column can have different data types)to indicate the data types,and the GROUP type can be supported to store data from multiple columns together.In addition,paper uses the B+ tree to implement the key-value store for each nested level,and also establishes the index of the primary key between nested levels.Based on the nesting strucrture and the indexes of its nested primary keys,a series of update operations,such as insert,update,delete,are fulfilled,and the association of nested relationships can be maintained through the update operations.In terms of query optimization,paper designs a scheme that pushes down the filtering operations,using a lightweight data structure like bitset to reduce the IO and assembly costs of unnecessary units.