The Design And Optimization Of Select Operation In Column-storage Database Based On Multi-core Processor

With the rapid increasing of data, according to the U.S. Internet data center that the datacreated by the Internet will grow by50percent annually.The total amount of data will getexponential grouwth every to yeas. Compared to the traditional data warehouse applications,the large data analysis has different features such as huge amount of data, very complex queryand so on. However, it is hard to deal with by the traditional DBMSs. Column database isproved to be a powerful weapon for complex queries. Contrast to the line database,columndatabase has more benefit like compression,deferred materialize and block iterarion..It hasbeen widely used in large data fields.At the same time, the delelopment of multi-core CPUmake more possiblity to algorithm performance optimization.Based on the two key points above, our lab implementd a column database prototypesystem gSQL v1.0,which is aimed to deal with complex queries. To further enhancegSQLv1.0system execution performance, this paper analyzes the bottlenecks of the system’sperformance at first. Then, find that the reason of the bottlenecks is related to the database’sselect operation. This analysis comes out a method to improve the system’s performance,which includes:1、Coarse-grained primitive design2、Introduce indexing mechanism3、Reuse of the intermediate resultsAnd also re-design adn re-define the primitives set of the select operation. The newprimitives set contains multi-field and multi-condition selection of full data primitivesMULTI_SELECT, multi-field and multi-condition selection of part data primitivesMULTI_SELECT_P, single field selection primitives SELECT_P, single-field index selectionprimitives SELECT_PI, single-field intersection selecting primitives INTERSECTION₁and multi-field intersection selecting primitives INTERSECTION_N. This paper usesmulti-thread technolo, SSE instruction set, cache optimization, and other optimizationinstructions to implement these primitives. Final test results show the effectiveness of t heoptimization strategy on these primitives, prove that these optimization strategies of theselection operation is correct, and give the direction of further optimizati...