Design And Implementation Of Monitoring And Management Platform For Elasticsearch

Elasticsearch is a distributed full-text search engine based on Lucene.In many companies,the Elasticsearch engine is used to retrieve data,and its application fields are also extensive.However,how to monitor and manage Elasticsearch clusters to improve cluster stability and how to control user queries to avoid mutual influence are problems faced by many companies.Therefore,it is necessary to build a monitoring and management platform for Elasticsearch to control user queries,manage Elasticsearch cluster resources,establish monitoring system,improve cluster stability,reduce the cost of operation and maintenance,and facilitate users to retrieve data using Elasticsearch.The monitoring and management platform for Elasticsearch mainly includes gateway module,metadata management module,and cluster monitoring module.The gateway module mainly contains authority checking at API level,rate limiting,downgrading,disaster tolerance,and query analysis and auditing.The gateway module uses Ehcache to cache user query data,and supports both TCP and HTTP protocols for handling user query requests.The metadata management module mainly includes Template life cycle management,resource dynamic adjustment,user application management,work order management and so on.The metadata management module analyzes the required resources according to the TPS and QPS corresponding to each Template,and adopts a greedy algorithm to perform resource capacity planning on the cluster to improve resource utilization.The cluster monitoring module collects cluster indicator information and configures monitoring graphs to visually observe various indicators of the cluster,such as write rate and query rate.In addition,from the perspective of users the module can monitor whether the data can be queried,analyze cluster unhealth check items and then generate health reports.The metadata management module and the cluster monitoring module all adopt the B/S architecture.Users interact with the server through web client.The background uses the SSM framework for development,uses Tomcat as the server and saves the data in MySQL.The platform has been launched.User query is well controlled.The unreasonablequery will be rejected.And the platform operation and maintenance personnel will evaluate the rationality,and then optimize the query before approval,which avoids the impact of the query on the cluster.The cluster operation and maintenance work is greatly reduced.And the resource utilization rate has been increased from the original 30% to50%.Also the stability of the cluster is greatly improved.At present,the peak reading and writing TPS of monitoring and management platform is 10904/s and 1316w/s respectively.