Design And Implementation Of Log Analysis System Based On Cloud Encryption Platform

In the era of big data,various application systems generate a large amount of log data every day.And with the development of cloud computing and distributed technology,these log data are often scattered on different server nodes.This makes it difficult to collect,store,and visually retrieve these log data.Furthermore,it becomes more and more difficult to solve system failures and tap the value of these massive log data.In view of the above problems,this dissertation studies the related theory of the log system,and designs and develops a real-time log analysis system based on the cloud encryption system.The cloud encryption-based real-time log analysis system is based on the B/S architecture and adopts the design concept of hierarchical processing.According to different functions,it is divided into five functional modules: log collection processing,log buffering,log analysis,log storage visualization,and abnormal alarm.The process of collecting,transmitting,and analyzing and processing logs in real time,and finally storing and visualizing them is realized.Relevant technologies involved in the implementation of this system include a content association key algorithm,a cloud encryption system,an ELK log framework,a Kafka data buffer queue,a cosine similarity algorithm,and a Flink real-time data analysis and processing framework.According to the characteristics of cloud encryption system logs,the logs are divided into five categories: cloud encryption system logs,deployment server logs,database logs,reverse proxy server logs,and cache middleware logs.Log collection is performed through two different collection methods: HTTP interface and SDK embedded.After preliminary filtering by the Logstash cluster and the transfer of Kafka message queues,the data stream is split and copied through the Flink real-time computing platform,and a series of log cleaning,classification and other operations,and finally use Elasticsearch and MySql databases to persist log data.This dissertation designs and implements a background data query interface,performs statistical processing on database log data,and realizes visual query of log data.