Research And Implement Of Configurable Data Migration Framework

In the informatization process, large organization and companies always deploy multiple IT applications when they implement their internal information management process. And there exists the need of migrating data across these applications. Therefore, effective data integration among heterogeneous systems is a very important and practical issue.Today, the research topics of data migration among heterogeneous systems in China are focus on uniform relationship model based on XML and business object mapping based on hibernate. But these methods cannot solve the problem of how to extract data smoothly from database with complex data persistence model. And they could neither solve the problem of how to reduce the development cost of data migration middleware. We should analyze and improve these methods.This thesis analyzes the data migration and ETL technology, and concludes the base tasks and bottleneck of each stage in data migration. And combined with the configurable theory, this thesis proposed the designment of data extraction and data loading interface based on Web Services provided by the source and target systems. This designment separate data interface with the database and it reduce the development cost of data interface.Through introducing the staging area and business rule engine, this thesis also proposed the designment of a configurable data cleansing and data transformation model. And this model improves the reusability of the data migration middleware.This thesis finally implemented the configurable data migration framework with Kettle, which is an open source ETL tool. And the framework has been tested in a simulated business environment, and the feasibility of it has been verified.