Source Of Information In The Data Integration System Monitor Realization

To access information on Internet and most new information application such as DSS and OLAP will need the support of data integration technique. Data warehouse is a representative application of data integration. Significant work has been done on research of data warehouse maintenance by the WHIPS project in U.S.A Standford University, in which the architecture is proposed and the relevant components-monitor and integrator are consummately developed. In these researches it has not been thoroughly considered how to make the information sources more positive and efficient.Continuous Intelligent Information Support is that the information sources initiatively and continuously send new information to its subscriber according to the predefined demands and the data change on itself. In data integration system, if the information sources have the capability of continuous intelligent information support, the complicated demands of integration-end can be well meet and it will increase the efficiency of data propagation and integrating.In data integration system, the function of monitor is to detect the change in source data and propagate it to integrator. As to make the information sources capable of continuous intelligent information support, we meliorate the implement of monitor as following: 1. The monitored objects are changed from base tables to source views as to avoid propagating the unnecessary source data change; 2. We provide the monitor rules according to which monitor detects, analyzes and propagates the change in source data, and these rules can be predefined to meet thecomplicated demands of integration-end such as the monitoring period.According to these meliorations, we propose the new architecture of monitor. The relevant research and implement work including three aspects:The first is connotation and management of monitor rules. We design the structure of monitor rule database, list the attributes and explain their meaning. We also provide the rule management interface and the front tool to facilitate client to manage the monitor rules.The second is detection-analysis of the change in source data, or how to implement detector-analyzer. We propose the software structure, in which threads work concurrently to ensure efficiency and response time of detecting-analyzing processes. Then we focus on the two detecting-analyzing methods. For detecting-analyzing method based on snapshot difference, we implement these snapshot differential algorithm: sort merge algorithm, partition hash algorithm and window algorithm. For detecting-analyzing method based on log, we discuss the analyzing method of log on two relation database systems-Oracle and SQL Server. The relevant functional components and data structures are also stated.The last is propagation of the change in source data, and the crux is how to ensure the robusticity of this data transfer process. We use TCP/IP protocol to ensure the reliability of the process. By the checkpoint-log technique, we can ensure the recovery of the process.