Research And Application Of Near Real-Time Update Mechanism In Data Warehouse Based On SOA

With the development of web technologies and increasing use of information systems in enterprises, the frequency to update data has increased so much that traditional data warehouse could not catch with up the transactional systems. Traditional periodical data updating methods might not provide latest reports and data analysis to decision makers in time. In order to decrease the delay of data transmission, the ability to detect data changes, load data continuously and update data in a fast speed are introduced to real-time data warehouses, which, in turn, increase the amount of data transferred in the network. By near real-time data update mechanism, change data could be stored temporarily and transferred in a batch to save the network resources.To solve the problems that data in transactional systems and data warehouse could not be synchronized in real time, the paper researched related technologies about service oriented architecture (SOA) and data services and proposed the client/server model and data transmission mechanism on top of SOA. It took use of event-driven architecture to detect data change and transmit continuous data. Data service and data compression was used to transfer data across multiple platforms while change data capture and incremental data update were used to improve the speed to update data. The paper then gave a detailed design and implementation on the solution and tested it with some examples.This paper has accomplished the following tasks:①It investigated current status of the development of near real-time data warehouse and researched related technologies about near real-time data warehouse. It analyzed kinds of different change data capture methods and compared their advantages and disadvantages.②It researched related technologies about SOA and enterprise bus and analyzed the strong points of using SOA in data warehouse. It took research into data service based on SOA and the relationship between SOA and event-driven architecture. Different XML data compression methods were also investigated to compare their different applications.③It designed the near real-time data update system using client/server model. The client was used to monitor data change and capture changed data, the event-driven architecture could supply active data transmission in real time and the server was used to convert data format and update data incrementally.④It implemented the prototypes of major modules of this system, including the event-driven architecture and change data capture on client side, the data access based on data service and incremental data refresh on server side. XSLT was used to convert SOAP messages, compression by gzip were used to compress data in XML formats and enterprise service bus was employed to register and integrate services.⑤Elementary tests had been taken on this system based on some examples, which validated the feasibility and rationality of the design. It analyzed the performance of data compression and compared the system with other systems using similar technologies.