The Optimization Design And Realization Of Mass Data Parallel Processing Architecture In A Financial System

In many industries and institutions, there are a wide range of application requirements to do a fast batch processing of mass data sets of large and medium-sized database and data warehouse. How to realize fast interactive batch processing of mass data isboth an increasingly prominent problem for a management information system and a critical problem that needs resolving in the data concentration projects. A dynamic and scalable storage calculation mode is urgently needed for mass information technology architecture to achieve quick response mechanism.This thesis studies the industry practice of mass data processing technology such as database based on MPP, cluster technology and virtualization technology. Based on the study, this thesis analyzes the inefficiency of mass data processing of a financial system and finds that the main reason is that the problem of data architecture design causes the inability to do a large scale parallel processing and influences the extensibility and the processing efficiency of the system, and the parallel scalability of the server is restricted and overall system processing efficiency cannot meet the needs of business development since the technologies adopted at the IT infrastructure level are the traditional one. Based on an analysis of the problem and an reference of the industry best practice, this thesis redesigns the data architecture of this financial system by appropriately doing data defining, partitioning and resolves the problem of mass data parallel processing in the perspective of logic level. This thesis introduces cloud computing technology at the infrastructure level and builds computing resources pool and storage resources pool, achieves the flexible configuration of resource and lays a foundation for the efficiency and flexibility of mass data processing of this financial system. Finally, in order to verify the effectiveness of the solution, this thesis develops a prototype system, sets up a testing environment based on GreenPlum database that adopts MMP architecture, WAS cluster and IBMvirtualization technologyand conducts testing. Testing results show that the solution better solves the inefficiency of mass data processing of the current financial system, owns strong expansibility, and fully meets expectations. Based on the research of mass data processing technology, through appropriately classifying the data, introducing the idea of multi-channel pipeline processing and combing cloud computing technology, database based on MPP architecture and cluster technology, this thesis effectively solves the problems of mass data processing efficiency of a financial system, builds an extensible, configurable, and scalable technology support platform and provides strong technical guarantees for a variety of business innovation and development of the financial system.