The Design And Implenmentation Of Big Data Integration Platform

With the continuous spread of cloud computing, data correlation technology and the concept. The traditional centralized storage replaced by distributed storage, calculation and analysis of traditional structured database data replaced by distributed computing based on Hadoop cluster. Then, storage capacity, computing power system has made rapid development, all kinds of software are also springing up one after another.Provides a strong support to the development of the scale of the one hand system for massive data storage and computation of the sea. But, operation and maintenance of business system on the other hand, has brought the huge challenge.The traditional operational system mainly limited to two aspects: one is the dynamic scalability; One is the monitor size. This article is based on plug-in technology, using the configurable design concept, to build a distributed dynamic operation maintenance system. Through the resource management configuration,dynamic register new monitoring resource types, no longer limited and later monitoring resources type; through the configuration of collection, summary node register, dynamically extensible system monitoring, processing, computing ability, can realize the monitoring and control for large scale system.Firstly, This articleis used to analyses the demand of the system, and explore the construction scheme of the system based on distributed technology, through dynamic acquisition nodes and sink processing node configuration register, to achieve a dynamic scalable distributed systems, in order to meet the needs of operation monitoring ability of large-scale system.Then, the most important part of this paper is to focus on the research of monitoring resources model, through a highly summarized and decomposition, through the establishment of resources, index, threshold, state, event and so on five elements model, realizes the dynamic expansion of monitoring resources, can be based on five elements to register the new monitoring resource types, not limited by the inherent monitoring resource types, meet the demand of soft, hardware monitoring emerge in an endless stream.Finally, in order to enhance the real-time monitoring system of large-scale system introduces the cache technology, constructs a cache computing architecture, greatly enhance the real-time monitoring system.The contents of this paper have been applied to real practice, and verified in the project, to further prove that the system is feasible and the research content has the certain practical engineering value.