Distributed Database Fault-tolerant Fault Injection Test Tool

Distributed databases have higher scalability and performance than centralized databases.Due to the complexity of the distributed database,software and hardware failures such as network interruption,packet loss,and node downtime are unavoidable in the real world.The distributed databases should continue to perform fault injection testing,that is,proactive triggering of faults through coding simulation during normal system operation in order to reduce the loss caused by the failure problem to the distributed database,improve the reliability of the distributed database and verify the correctness of the fault tolerance protocol in the system.This thesis designs and implements a fault injection test tool AFTest for fault tolerance of distributed database systems.The main contributions are as follows:1.Flexible task scheduling design: it supports manual submission of test cases and automatic traversal of existing event combinations at the same time.Testers can query the test progress,history and real-time running status of the tested system in real time through the front-end interface.The back-end task scheduler can periodically generate test cases and generate execution scripts based on the cases,assign specified worker threads to execute the tests and record the corresponding log information and system state changes during the test.2.Implemented a fault injection framework: we first analyze the various types of software and hardware failure problems that have occurred in the distributed database in the actual application process.Then we implement a variety of software and hardware failure problems and Cedar's unique system operations through coding simulation.3.Reduce state space explosion: we deeply traverse the combination of most fault events in order to avoid tedious test case writing and perform fault injection testing of distributed database systems more efficiently and quickly.We then reduce redundant test cases through algorithms that reduce the state space to avoid problems with too many test cases.This dissertation conducted a fault injection test on the open source distributed database system Cedar with AFTest.This paper demonstrates the effectiveness of AFTest through experiments,which can realistically simulate the occurrence of various distributed fault problems.AFTest can find potential compound bugs in distributed databases by traversing all event combinations,at the same time it can reduce the impact of redundant cases on the testing process and speed up the test progress.