Design Of Data Warehouse Of Historical Data Archiving For Commercial Banks

A large number of financial data is generated each day with transactions in commercial banks.In addition to transaction data,many structured and unstructured data,such as customer operation actions,receipts scanning,videos and recordings,are stored dispersedly in internal banks' systems,so it is not easy to analyze and utilize them uniformly.Besides,most of the data come from trading systems which will not store operation traces and status as well as other process data for a long time,but will regularly clear them up due to storage capacity and performance.It is difficult to trace specific data in a certain time after the cleaning.With the high requirement of regulatory and bank management,it requires highly for the integrity of historical data.Historical data should be stored completely for the sake of applications' query relevant to big data analysis and mining like customers' precision marketing and risk model construction in commercial banks.With the development of data storage technology and big data technology,it is a trend for commercial banks to build archiving database for commercial banks based on big data technology,and realizing the storage and processing platform of massive historical data of commercial banks,in order to support queries from various application situations like historical data and original transaction data to assist the processing and application of big data.The paper studies the application orientated big data supporting by taking archiving database of some commercial bank as a case study.This paper establishes archiving database for historical data storage,management and application via adopting data acquisition,data processing,data storage,data access,scheduling and monitoring functions.Technically,the distributed framework of Hadoop includes projects like HDFS,Hive and HBase,is used to offer archiving and query based on massive historical data.This paper builds a big data processing platform based on Hadoop which provides management,monitoring and diagnosis of Hadoop cluster,and offers administrative webpage for monitoring,operation,configuration,log viewing and performance reports of big data platform.On this basis,the paper designs the data warehouse of bank archiving data,to archive important system data include T + 1 data,accounting and current data tables in core business,mobile banking and credit,and maintain long enough to meet the demands from various queries of historical data.The paper packs up common functions by adopting Perl Language and provides common components like file transferring,Oracle data unloading,hive data loading / unloading and others.This achieves a multi-version of the data access service and provides data accesses of original table structures and data tables at any time.The system testing shows that the data warehouse designed in this paper is able to provide online inquiry service of data sheet,and snapshot information like accounting and transaction logs.All of these will meet the requirement on historical data from judicial office's query,internal-external audit and supervision etc,provide bulk data supply and output services of historical data.