Collcetion,Storage And Analysis Of Time Series Data In Large Scale Cluster

At present,the cluster has been more and more widely used in Internet field.But with the system architecture being increasingly complex and the amount of data being growing,the cluster size becomes larger and larger,which invisible brings operation and maintenance staff great difficulties to process failure in time and ensure the efficient operation of the cluster.So a good monitoring system is needed to help operation and development engineer timely detect the source of the problem and position performance bottlenecks.The server metrics collected by the monitoring system are generally time series.There are many drawbacks using traditional relational database to storage time series.For example,in the case of large amounts of data query speed is slow which is not friendly to the user,the storage being not optimized takes up a lot of disk space and so on.Therefore,the cluster monitoring system designed in this thesis adopts a new database-time series database.In order to amend the existing cluster monitoring system of a number of shortcomings and meet the needs of a wide range of monitoring,the cluster monitoring system is designed in a new architecture.The ultimate goal is to ensure the stable operation of the cluster,and to help developers know the real-time state of business,find performance bottlenecks and locate the problem code,thereby enhancing the efficiency of development and operation to ensure that users get good service.This thesis first introduces the significance of cluster monitoring and the shortcomings of the existing monitoring system,and then studies the working principle of time series database InfluxDB and tests the performance of InfluxDB compared with relational database.And then describes what metrics the cluster needs to collect and summarize the monitoring principles of the technology used in our laboratory.Then,a new cluster monitoring system for system,application and service is designed with InfluxDB as the core.The architecture of the whole system and the design concept and concrete realization of each module are expounded.Finally,we deploy the monitoring system on the existing cluster and analyze the time series data collected by it.