Research And Implementation Of Column-Store In DWMS

As a data analytical environment, more and more attention has been paid to data warehouse, which is divided from the operational database environment. The analytical environment puts forward higher requirements for query speed than operational environment while column-stores have much more advantages than row-stores in Read-optimized and analytical environment. To this end, almost one year ago, the author has already begun the research and development of column-oriented data warehouse management system in the laboratory.This paper, combing the research and development in actual project, studies the storage technologies of database and data warehouse as well as the related technologies of existing relational database management systems and data warehouse management system, especially the column-store technology. It designs and implements the storage structure and management functions of column-oriented data warehouse, including external medium and cache. In the aspect of cache management, an adaptive page replacement algorithm called HA, based on the double management links, is proposed.First of all, this paper studies the storage parts of traditional RDBMS and row-oriented DWMS. It studies the detailed implementations of row-stores and column-stores, especially analyzes and summarizes the data storages of two open-source database, PostgreSQL and C-Store.The paper makes an intensive study of the storage part in column-oriented data warehouse based on the analysis of rational database. It accomplishes the preliminary design of the overall structure and function modules in column-oriented DWMS. It defines the design and implementations of the file manager, the buffer manager and the access method manager. The storage in DWMS is divided into two layers, logical and physical. According to the hierarchical management thought, file manager can be divided into three parts, the operating system file management, virtual file management and file management. Virtual file manager uses LRU algorithm to evict file handles and hash algorithm to find files. Buffer manager is the core of the entire system and the pivot of data flow.Based on double management links, this paper proposes HA(Hybrid Adaptive Algorithm) on general buffer management algorithm. As an improvement of DMC (2c) Algorithm, HA introduces the dynamic replacement points and decides the working link by the number of page faults. It can also choose a more efficient page replacement strategy by the local features of page access sequences. The experimental results show that the rate of page fault of this algorithm is nearly 30% less than that of before. B+ tree index is one of the important index approaches of access method management. This paper mainly introduces its structure and its operations such as create, search and update.Finally, the paper analyzes the performance of DWMS storage manager, and puts forward the cost evaluation model. Combining the cost evaluation model with the concrete implementation, the paper proposes a two-aspect optimization from the angle of data storage, which are storage structure optimization and data buffer management optimization. The specific optimization methods are as follows. To save space, the ROWID of data item is stored in the head of data block; To avoid the data loss, setting operation of dirty blocks is included in the block management; In favor of the data scan in segment, the extent information included in a segment is stored in the segment control block; To save the hash searching time as well as to ease the update operation, hash chain structure is designed to be a double list; To reduce the cost of data buffer management when bulk data is imported, the status setting is performed in the data buffer management.