Design And Implementation Of Massive Heterogeneous Data Customization Platform

In recent years, domestic banks have realized data concentration of the core system, and have developed a series of business management systems, such as financial, loan, audit, cards. Because of heterogeneous of the software and hardware platforms, which have caused many difficulties in data centralizing, sharing, analysis, and using. "Data supply and demand" has become an increasingly contradiction. In order to solve this problem, large banks generally build data warehouse and business intelligence projects. With limited human and financial resources, small banks’ information construction starts late, and doesn’t implement these projects timely. Simultaneously, building a warehouse construction is a gradual long-term process. Therefore, conforming the law of information technology, under the principle of "urgent need first", small banks urgently need to build a forward-looking and scalable data integration customization platform to meet the demand of sharing data with using heterogeneous systems and personalized data customization for a large number of users. This is one of the core tasks for small banks to speed up the construction of information technology.This thesis is in the background of the "data platform project of a bank in Hebei Province". According to the demand of a large number of users to personalize the data from heterogeneous data sources, this thesis proposes a massive heterogeneous data customization model, and carries out the design and implementation. To solve the banking data customization problems, this thesis carries out the following research:(1) This thesis carries out a detailed analysis for the characteristics in data customization process which include mass user, asynchronous subscription, loosely coupled. This thesis points out the necessity of the implementation of data integration as a precondition, and analyses the heterogeneity, distribution and autonomy problem which data integration is facing. Finally this thesis gives the basic framework consists of customization and integration of two core modules.(2) This thesis proposes the data customization model based on publish/subscribe, and designs the event model and subscription model based on the XML tree structure and rich expression. The thesis also designs the matching algorithm which uses the events generated by the XML parser to drive the transitions in NFA, and gets the subscriber information quickly and efficiently, which meets the demand personalized customization data for a large number of users. This thesis combines all of the sharing NFA query paths into a single NFA, and improves the matching in terms of time efficiency and space efficiency.(3) This thesis proposes the data integration model based on ETL-ODS, solves the problem of extracting data from heterogeneous system and massive data storage. This thesis applies ETL to achieve on extracting data from heterogeneous system, according to the bank data characteristics, the ETL takes a four-step measures, and generates a global unified view of data finally. This thesis applies ETL to achieve on massive data storage, and carries out the analysis and optimization for ODS.At present, the data integration customization platform has been put into operation in a bank in Hebei Province. Practice has proved that the overall design is reasonable, and has effectively solved the problem of personalized data customization and massive heterogeneous data integration, and satisfies the data application requirements of the whole bank.