Database System Performance Evaluation And Quality Management

As an important basic supporting software,database management system is widely used in key applications of finance,communication,e-commerce and other fields.In recent years,the rapid development of Internet has spawned a large number of new applications.Driven by the new applications,the database system has developed rapidly.At the beginning of this century,NoSQL databases emerged,and nowadays,NewSQL databases play an increasingly important role.Currently,there are more than 300 wellknown database systems.At the same time,with the development of new hardwares,the design of database system has more choices,and the system architecture is constantly reconstructing.The emergence of new applications and fast development of new systems bring new challenges to database system performance evaluation and quality management.Benchmark is a ruler to measure the database performance,which guides the development direction of database systems.However,benchmark is an abstraction to a class of applications,serving for general performance evaluation.For a specific application,especially a new one,the evaluation results may not be instructive because the workloads of the benchmark may vary greatly from the actual ones.Due to the weak application relativity of generated test data and test workloads,the evaluation results based on traditional test techniques may not have reference value for specific applications.At the same time,for the quality management of database systems,the benchmark workloads are of limited size,and current test techniques cannot achieve the automatic generation of large-scale transactional workloads,so database systems lack sufficient testing of transactional workloads,especially high-pressure and highconflict ones.In view of the above problems of database system performance evaluation and quality management,this thesis proposes application-oriented database performance evaluation technologies and designs a large-scale transactional workloads generator with automated correctness verification.The main contributions of this thesis are summarized as follows:1.Database performance evaluation technology for analytical applications.We design and implement Touchstone system to support workload-aware test database generation and query instantiation.In Touchstone,a new query instantiation mechanism is proposed.Based on the data distribution of attribute values,binary search and random sampling algorithms are used to quickly locate the query parameter values that meet cardinality constraints.To achieve parallel test database generation,it designs new data structures,i.e.,constraint chains and join information tables,to decouple the dependencies among attributes.In order to reduce memory consumption,it proposes an efficient compression algorithm over join information tables without affecting the quality of generated test database.2.Database performance evaluation technology for transactional applications.We design and implement Lauca system to support application-oriented transactional synthetic workload generation.In Lauca,the key workload characteristics that need to be paid attention to in transactional workload generation are given for the first time.Lauca proposes to define transaction logic by using transaction structure information and parameter dependency information for characterizing potential business logic in transactional applications.In order to capture the skewness,dynamics and continuity in the data access distribution of real workloads,the system proposes three data access distributions,i.e.,skewed data access distribution,dynamic data access distribution and continuous data access distribution.Finally,the architecture design of Lauca guarantees the protection of data privacy.3.Large-scale transactional workload generation and execution correctness verification technology.We design and implement Thor system to support random transactional workload generation and execution correctness verification.To generate comprehensive and controllable random transactional workloads,Thor abstracts a set of basic database operations,and then generates large-scale transactional workloads by randomly combining them based on the specified random space.In order to verify the correctness of database under high-pressure and high-conflict workloads,it proposes the phenomena and anomalies detection algorithm and the correctness verification algorithm for read result sets based on transaction execution history,which effectively realizes the correctness verification of database isolation level.Moreover,for massive transaction execution history,Thor achieves controllable memory consumption during the verification process.In summary,this thesis deeply studies the performance evaluation and quality management of database systems,and designs and implements three systems,i.e.,Touchstone,Lauca and Thor.Firstly,in order to solve the problem of database performance evaluation for analytical applications,Touchstone achieves efficient query instantiation and fully parallel test database generation.Secondly,Lauca can generate synthetic transactional workloads that are highly similar to the actual application workloads in order to solve the problem of database performance evaluation for transactional applications.Lastly,to solve the problem of insufficient database testing for transactional workloads,Thor supports the automatic generation of large-scale transactional workloads,and can verify the isolation level of the database under highpressure and high-conflict workloads.Additionally,a large number of experiments have been carried out to verify the effectiveness of each system.Future work includes: developing peripheral tools of the three systems;open source and promoting applications,in which Touchstone and Lauca have been introduced into TiDBs quality management system;building a library with massive test cases to support automated evaluation of database systems.