Improvement Of Dameng Data Interchange Platform Oriented On Real-Time Data Warehouse

Data Warehouse not only provides the service of strategic predicting analysis for BI,but also need to provide the service of real-time tactical analysis, in the era of fierceeconomic competition. the service of real-time tactical analysis requires that the data in theReal-Time Data Warehouse ware updated in time and quickly, but the traditionalperiodical scheduling strategy of ETL in Data Warehouse can’t. in view that Dameng datainterchange is a professional ETL products, but it couldn’t provide the service of real-timetactical analysis, Therefore, the professional ETL improvement to adapt to theenvironment of real-time data is meaningful.Based on the in-depth analysis of Real-Time Data Warehouse and the newrequirements for ETL, a kind of Real-Time Data Warehouse architecture was proposed.After the in-depth research of the implementation principle of Dameng ETL, there shouldmake Dameng ETL better according to improving the two aspects of the schedulingstrategy and executive process of ETL. Firstly, the traditional Data Warehouse is generallyupdated by day, week or month in term of periodically scheduling, and the schedulingstrategy has the shortcoming of large time delay and cannot fit the environment ofReal-Time Data Warehouse. So the event trigger scheduling strategy is employed on theplatform of Dameng ETL, the strategy is a flexible scheduling manner, it can implementthe setting of trigger condition from multiple perspectives and well combine the changeddata’s characteristic in the data sources with the requirements of the user, and can updatesthe data in Data Warehouse in time. Secondly, because of the requirement of the higherETL work efficiency in Real-Time Data Warehouse and the fact that the processes of dataextraction, data transformation, data loading run in their own thread in the process of ETLwork, the schema of multi-threaded task execution decomposition for the execution ofETL work was proposed, and in the schema multi-thread execute the processes of dataextraction, data transformation, data loading, and the schema promotes the ETL workefficiency by means of the improvement of concurrent degree.Lastly, the experimental results show that the schema of multi-threaded taskdecomposition can improve the ETL work efficiency and the strategy of event trigger scheduling is effective, can load data into the Real-Time Data Warehouse in time from theincremental data set in the situation of meeting conditions, and the schema ofmulti-threaded task decomposition can improve the ETL work efficiency in the conditionthat the system CPU resources is not used completely.