| With the advent of the era of big data,how to quickly mine potential information from massive data is an important challenge in the database field.In order to solve this problem,data warehouse came into being.It can solve the performance bottleneck problem of traditional database when dealing with massive data and simplify the calculation process of distributed system.Hive is the mainstream data warehouse based on Hadoop framework,which can be used for large-scale data analysis in SQL language.However,Hive data warehouse also faces two challenges: on the one hand,Hive builds Hadoop-Map Reduce tasks on Hadoop,and Hive’s query performance is limited by Hadoop-Map Reduce parallel computing framework,which leads to high delay and resource consumption,so Hive will face the problems of high query delay and poor concurrency tolerance in practical business applications;On the other hand,Hive appeared earlier and has a high popularity in the industry.Even though the emerging OLAP(Online Analytical Processing)engine has a faster query speed,it is expensive for people to learn different OLAP engines,and it is not easy to completely replace Hive.In view of the above situation,this paper proposes an adaptive acceleration method based on OLAP engine.Under the condition of not changing Hive QL query habits,this method adaptively converts Hive QL queries into high-performance OLAP engine queries through SQL dialect conversion,and dynamically adjusts the priority of OLAP engines according to the cluster busyness to automatically select the best engine to perform query tasks,thus improving Hive QL query efficiency.The main contents of this paper are as follows:(1)Propose an adaptive query acceleration method based on OLAP.So that Hive QL can be converted with SQL statements of different OLAP engines;The dynamic scoring strategy is adopted to dynamically prioritize the clusters.(2)Optimize the OLAP engine used in this method.Aiming at the problem that Kylin pre-calculation will bring Cube explosion,a construction adaptation algorithm is proposed to better build the cube.Aiming at the problem that the query data of Presto and Click House is too large,a partition checking strategy is proposed to reduce the query data of Presto and Click House.Finally,in order to reduce repeated queries,a result-level cache is provided for repeated queries and time-span queries.(3)This paper uses TPC-DS authoritative data set for comparative experiments.The experimental results show that the adaptive query method proposed in(1)and the optimized adaptive query method in(2)have obvious speed-up effects on Hive under single-table query and multi-table associated query.The average query time of Hive before speed-up is 1680.20 seconds,and after speed-up,the average query time is 46.15 seconds,and the comprehensive query time is reduced by about 1634.05 seconds,which finally verifies the feasibility and effectiveness of the adaptive acceleration method proposed in this paper.The query acceleration method proposed in this paper has been analyzed,designed and implemented for nearly a year,and has been gradually replaced in the online environment since last September.So far,the stability and high performance of this method have been fully proved. |
