Design And Implementation Of Real-time Data Extraction Mechanism In Data Warehousing

In order to improve the level of scientific decision, China telecom enterprises have successfully built Business Analysis Systems (BAS) to provide the decision makers with management information based on data warehousing techniques. However, BAS can't provide real-time information based on changed data. Further, it can not support automation of the structured decision making process which is routine for the enterprise. And it can not provide the right person with the right information in the right time. As a result, when more and more management information grow up in BAS, it is more difficult for the decision-makers to use them. Active Real-time Data Warehousing techniques can be utilized to solve the problem. CDC is the key one of them. By means of the technique of DBMS logging, a CDC framework is proposed in this paper based on Message Queue, which can support real-time CDC in business system, transmit formatted data to target system through Message Queue, and process the received messages by data apply-agent. In addition, an event management model is designed following the framework. The model can produce events triggered by the changed data, which can actively drive ETL process to produce real-time management information. Integrated with Publish and Subscribe mechanism, the model can properly support real-time information producing and pushing actively. Furthermore, it can support automation of routine structured decision making process. The whole process is a real-time making-decision closed loop based on event-driven. The loop is a dynamic process which continuously observes and collects events from a business environment, converts the event data into meaningful business information, discovers and analyses business situations and exceptions, automatically selects the most appropriate actions for a response to the business environment, and executes the business actions based on the decision that has been made.This framewok is expanded based on traditional data warehouseing techniche. It not only reservers traditional data warehousing framework which is easy to put into effect in reality, but also adds the mechanism of processing data in real-time and active behavior based on event. It can support automation of the structured decision making process which is routine for the enterprise.