Research On Operation And Maintenance Management Of Big Data Platform Based On ELK

Traditionally,a data management system is built with a relational database,which requires column names to be explicitly specified during data retrieval.The retrieval efficiency of this approach is low when the scale of data is large.Now with the ELK stack,which supports full-text search,the retrieval latency can be reduced to milliseconds.And to solve the problems such as abnormality detection is not timely,log analysis is not efficient,and statistical retrieval is cumbersome,which are all caused by log files such as application logs,system logs,and error logs being stored in a distributed fashion,the ELK stack,in combination with role access control,offers log analysis on big data applications and visualization on the analytical result and the effectiveness of operation and maintenance management.This paper mainly focuses on the following two aspects.(1)A real-time log search and analysis platform for big data applications is built with the ELK stack.The platform consists of four modules,namely,log collection module,Kafka message publishing and subscription module,log processing and indexing module,and log analysis and visualization module.By taking advantage of Kafka's throttling and ordering features,the platform is able to ensure the stability of massive data and further reduces the pressure on the Elastic Search engine from a large number of requests.The experimental result demonstrates that the platform is of certain applicability.(2)Three system operation and maintenance solutions are proposed for the real-time log search and analysis platform.In operation and maintenance,it is possible that any user can access any index of a cluster.When multiple services use the same cluster,one service may expect its data to not be seen,written,or deleted by another service.This requires user access control,which only allows certain users to access certain indices.The operation and maintenance solution proposed by us can guarantee the access to a cluster is restricted to legitimate users.These three solutions are role-based access control,X-packbased system communication encryption implementation,and specific cluster management methods.They can ensure the stability and security of the log analysis platform.The paper studies the operation and maintenance management of big data log platform based on ELK and role access control technology.As a search engine framework for industry processes,ELK is not only capable of searching user data at a high speed in mass data,the fastest speed can even reaches milliseconds,effectively filters the data unrelated to user requests;but also it can display the data through user-defined methods to users in a variety of colorful chart forms such as bar charts and line charts.Therefore,ELK can solve all kinds of operation and maintenance related tasks and problems such as traffic early warning,personalized log collection,and user behavior analysis.Applying ELK to the operation and maintenance management of big data platforms can not only integrate resources,but also quickly and thoroughly view the role information for log analysis.In the context of the rapid development of big data,the research has certain application value.