Design And Implementation Of Metadata And Data Buffer In Column-oriented DWMS

With the development of ERP and electronic commerce, more and more companies are paying attention to the data warehouse recently for getting their useful information. DWMS has three characteristics, subject-oriented, compositive, nonrenewable. It is a decision system for analysis large quantities of data. When DWMS do the data analysis, ad-hoc query operation is usually used which needs system read large quantities of data in a short time. So compare with the traditional DBMS, it pays more to the performance of read operation. Column-oriented storage is a new skill for reading large quantities of data in a short time. In the column storage structure, each column is a unit to store in the disk, then it needs read some useful column from disk when system does the query operation. This process reduce the disk I/O and improve the reading efficiency. So the Column-oriented storage is the main storage structure in most of the DWMS.Metadata and Data Buffer is both the important factor in DWMS. How to design and realize both of them has relationship to the efficiency and stability of DWMS. Metadata store any basic data in data dictionary. Data Buffer which is a buffer structure is to reduce the I/O operation. So a good design of Metadata and Data Buffer impact the efficiency of DWMS.As for DWMS, there is a little operation to update a data. But to the Metadata, the system has a lot of operation for updating and deleting. As for the limit of Column-oriented storage, the system can't support the updating very well. This gives a big problem to realize DWMS system. The target of the author is to design the logical and storage of Metadata. The another target of the author is to design a data buffer management which fits the storage of Metadata.The author's research as follows:1. Analyzing the management, realizing data storage and designing a good logical and physical storage management as for the definition of the DWMS.2. The author just researches data dictionary in Metadata and designs the data dictionary in detail which meets the upper module in DWMS.3. As for the shortcoming of Column-oriented storage and the needs of high efficiency of updating and searching, the author designs a suitable storage structure for DWMS-Hash_B+ index which meets the efficiency of updating and searching.4. The author designs a suitable management of Data Buffer which meets the Hash_B+ management of data storage.5. Through a lot of experiments which shows the efficiency of Hash_B+ which meets the demand of the project.This paper designs and researches the Metadata and Data Buffer in DWMS and puts forward a index structure called Hash_B+ index. For a lot of experiments, the result shows that the design of Metadata and Data Buffer is effective which improves the efficiency of searching certain value.