Research And Application In System Architecture Of Parallel Data Processing Middleware

With the development of the increasing centralization of database and the network technology, the information flow is becoming larger and larger. As far as the applications of the data management system, with the data scale going to TB or PB level, it is hard to meet the demands for the performance requirement in terms of CPU and I/O, even if a large amount of users and data are operated on large-scale main host with high performance. Therefore, there is an urgent demand for the data management system of parallel technology to extend data processing application. In order to solve the management and calculations of the massive data and heterogeneous data, the concept of data processing middleware, which can be used to regulate the database to work together and achieve interoperation, is put forward. The system of middleware is characterized by structural flexibility, simple development and the realizing of parallel operation of database with comparatively low cost. The research in this thesis is mainly focused on the system structure of data processing middleware and the main relative components.Based on the existing PC, advanced network technology and cluster of databases technology, the study on the larger structure of the middleware system, the structure of hardware, the interface communication technology, the architecture of middleware and the key techniques are performed. It is founded that includes the model of cluster system based on SN, the system module model and the resource distributes. The analysis on the resource data of middleware system aad the research and the design of the management system of the data of middleware system are also performed. The knowledge of the basic principle of inquiry, conduct and control systems and the comprehensive process are also achieved. This thesis uses data placement and cache system to improve system performance, the contents of the cache system, organization and management. In the actual application environment, achieving the main function modules of the parallel data processing middleware and principles and work steps of the middleware are presented. According to the results, the parallel data processing middleware system performance was evaluated.At present, this middleware system has been used in data placement strategies and semantic cathing and has been applied to some large engineering projects. The research and implementation of data processing middleware, shows that this approach can provide users with a unified service interface and management platform, improve system performance efficiently, enhance system availability and maintainability, protect the users' existing investment.