Research And Application Of The Partition Technology In Real-time Data Warehouse

With the increasing of real-time processing data requirements in enterprises, the traditional data warehouse technique has been unable to meet the requirements. The appearance of real-time data warehouse technique provides real-time or near real-time data information for enterprises and organizations. As the improvement of real-time data, the daily mass of data is stored in the computer, so the problem how to efficiently manage the data in data warehouse is emerged. It is one of the approaches solving the problem to implement the efficient data management based on database management techniques.In this thesis, for the characteristics of real-time data warehouse and the purpose improving storage and query effectivity of data warehouses, the contribution of partition technique for storage and query effectivity of real-time data warehouses and the problem of partition storage for increment data are studied. From the traditional characteristics of the database model, the partition model based on zoning district table and systematic modeling processes are proposed, including model creation, the mode extraction and data migration. The advantage of the partition table is proven from the theoretical basis, experimental basis, universal and expansibility. Aim at the characteristics of real-time data warehouse, an improved algorithm of horizontal partition and an improved algorithm based on server-side are proposed, and the improvement for query effectivity is discussed. Moreover, the partition engine based on above partition techniques is designed and utilized in the development for data warehouse in national marine environment.The experiments show that partition strategy based on partition table and the dynamic partition algorithm in data storage and query had superior performance and achieved the expected goals.