Design And Implementation Of Distributed Database Transaction Processing With Shared Buffer Based On RDMA

With the rapid development of Internet technology and big data,traditional centralized database systems have been unable to meet the demand for high concurrent processing loads.As a result,distributed database systems with different architectures have emerged to address the demand for high concurrent loads.The distributed database with shared buffer supports multi-node writing,and multiple nodes share a storage,and the buffer is also shared.Compared with a database with shared nothing architecture,the network overhead is higher due to the need to share the buffer.In response to the above problems,in order to effectively solve the problem of high concurrency load and low latency requirements in the context of big data,we designed and implemented a shared buffer distributed database based on remote memory access(RDMA)to give full play to RDMA technology in network transmission.RDMA's lowlatency feature optimizes the data transmission efficiency in the shared buffer.The new storage hardware persistent memory(PM)technology is also used to improve the persistence efficiency of commit log information in the global transaction manager(GTM).The database system designed in this paper can support the characteristics of transactions,and proposes a set of conflict resolution solutions encountered in the process of transaction processing.In addition,the shared buffer database has high scalability and can extend the buffer horizontally to avoid the occurrence of disk IO.The main contributions of this article can be summarized as follows:1.Based on Open Gauss,a distributed database system with shared buffers was designed and implemented,and optimized by using RDMA and persistent memory technology,a customized and lightweight network communication framework was proposed to give full play to the low-latency characteristics of RDMA.2.Based on the shared buffer distributed database system,this paper proposes an effective cache management mode to coordinate the conflict of buffer access between different nodes.Since modifying the buffer will cause the invalidation of the buffer copy in other nodes,this paper also proposes a global cache service method to solve the problem of cache consistency.3.In order to meet transaction consistency in a distributed environment,this paper proposes an efficient cluster transaction management mode,which effectively solves the global timestamp problem and provides unified snapshots and commit timestamps for different computing nodes.At the same time,this paper proposes a multi-node write concurrency control algorithm based on a shared buffer distributed database,which uses an optimistic lock method to resolve write-write conflicts between transactions and effectively avoid the network overhead of distributed locks.4.A large number of experiments are carried out on the realized shared buffer distributed database system,which further verifies that these methods are correct and efficient.