Research And Implementation Of Universal ETL Technology

With the development of computer network technology, the management of manipulation and information is gradually depended on computer in many departments. Enterprises start to develop many application systems based on various software and hardware platform; consequently many abundant data sources are accumulated. These data sources are valuable to the enterprise; they are eager to integrate these heterogeneous data sources, which are separated in locations and self-management in management and heterogeneous in data module.Firstly, the research status of data integration in our country and abroad is depicted. The background of the article is the data center project carried out by Fujian state power. According to the character of data sources with different architecture, a common ETL system with common technology of data accessing (OLE DB), management of repository, abundant data cleaning functions, friendly user interface and multi-thread data manipulation is provided. The design and implement of this system is completed in the article.Each functional module in this system has a model. The innovation in this article mainly consists in the model of repository, data buffering, and incurrent manipulation. Furthermore, the functions of data transformation are enriched by plug-ins, which is defined by users. The model of repository has a standard hierarchy. Upper layer provides the depiction of lower-layer and the implement of upper-layer is provided by lower-layer. The control of data errors, the detection of data quality and the definition of ETL rules are simplified by repository model. The design of buffer model, which stores data in the relative same form, conforms the various forms of data sources. The incurrent data manipulation increases the efficiency of ETL system and the plug-in enriches the functions of data transformations.All these modules are implemented in the article. The Algorithmic flow of the most of functions is presented in detail and a prototype of ETL system is implemented. The test of this system is carried out in end of the article.