Batch Insertion Optimization Oriented Parallel Database Storage Engine

With the avent of multi-core, using the multi-core to enhance the performance of database system is an important research topic in database field. Aiming to the bad parallelism performance and weak expansibility of MySQL existing storage engines, this paper researchs and develops a parallel storage engine named MTPower to make full use of multi-core.As one of the most popular open source database systems, MySQL is pluggable storage engine architecture, which divies into two layers:the SQL layer and the storage engine. The SQL layer is responsible for query analysis, optimization and execution while the storage engine manages data storage, index management, transaction and recovery for MySQL. However, the existing storage engines always adopt sigle thread model, like MyISAM, or fixed threads number, like Falcon, this is hard to develop potential parallel computing capacity of multi-core, and especially index insertion often becomes the bottleneck.Based on multithreaded dynacim scheduler interface (MSI), MTPower has strong expandability and ability of dynamic scheduling. This storage engine mainly includes storage engine interface, parallel batch linear hash index module and disk buffer. The storage engine interface totally follows MySQL storage engine interface standards and the parallel batch linear hash index can effectively use parallel computing capacity to improve performance of index insertion in batch record insertion. The disk buffer supports multithreaded parallel access and LFU replace strategy.Comparing with the classic single thread storage engine MyISAM with hash index in testing batch insertion, MTPower performance is 6 times than MyISAM at best. MTPower can achieve the best performance, when the total number of threads is close to the core number of processors. With the increase of the number of processor cores, the performance of MTPower increases subsequently.