Design And Implementation Of Web Application Front-end Performance Monitoring System Based On Spark

The rapid development of the Internet has brought earth shaking changes to people’s lives.By June 2021,the overall scale of Internet users in China has exceeded 1billion.In the face of such a large number of Internet users,it is very important to ensure the website performance of web applications.At present,there are the following problems in the performance monitoring of web application websites: first,how to deal with and analyze massive logs to monitor the performance of websites;Second,because different websites have different requirements for website performance,it is impossible to monitor abnormal performance indicators by setting a fixed threshold;Third,it is very inefficient and resource consuming to manually design regular expressions to detect abnormal log sequences.In order to solve the above problems,this dissertation proposes and implements a web application front-end performance monitoring system based on spark.The web application front-end performance monitoring system based on spark consists of four modules: log collection and reporting module,report statistics and analysis module,log detection and alarm module,log and alarm query module.The log collection and reporting module collects front-end logs of web applications through the SDK and sends the logs to the kafka message queue for cache.The report statistics and analysis module uses the spark SQL framework to calculate reports and supports the generation of daily,weekly,and custom performance reports.The log detection and alarm module realizes monitoring of abnormal performance indicators of website through single indicator anomaly detection algorithm,and realizes abnormal detection of log sequence through circular neural network LSTM.LSTM has excellent performance in long sequence and short sequence data processing.In addition,the log detection and alarm module also uses spark streamimg framework to calculate real-time alarms.To avoid alarm fatigue,The log detection and alarm module uses spark streamimg framework to perform alarm filtering and alarm convergence calculation.The log and alarm query module is responsible for real-time log query,log link tracing,alarm record query,and alarm related rules setting.The real-time log query function is implemented by the elasticsearch search engine.At present,the web application front-end performance monitoring system based on spark has been officially put into operation.The implementation of single indicator anomaly detection algorithm and log sequence anomaly detection algorithm based on LSTM solves the problems that there is no unified standard for performance index monitoring,log sequence anomaly detection is difficult and inefficient.At the same time,the report calculation and alarm calculation using spark framework realize the processing and analysis of massive logs,which has been highly praised by all parties since the system went online.