The Design And Implementation Of Log Data Analysis System

With the rapid development of enterprise informatization,the number of the information system continues to increase.In large company,the number of hardware and software resources deployed internally is increasing.In the process of maintaining the system,the data processing pressure will be harder and harder.The workload and difficulty of system maintenance are also increasing.The network log,the system log and the application log generated by routers,switches,PC servers,Ali cloud servers,a variety of large-scale application systems,databases and other computer network equipment continue to increase.In terms of log analysis,the traditional technology in dealing with those log datas will encounter a very large bottleneck in computing and analysis.As the recording tool for system application,network equipment and user operation,log data plays an important role in ensuring system security,eliminating system error,monitoring system and network operation and many other aspects.Through the log analysis,it can be summarized to see the system error source,the user operating behavior and the equipment running status.Such as by viewing the user registration log records,user event records can be found to judge whether the user is in normal state.Through the network configuration,equipment parameters,interface changes and other behavior of the record analysis,you can get information about the network running status.If the log can be real-time collected when it is generated,user can carry out large-scale data analysis or set the data alarm by setting the system alarm and monitoring.The method of displaying analysis results in the data dashboard can be easily accepted by enterprises,and it can more clearly reflect the value of the data.The project described in this paper is the actual project that the author participates in the design and development process during the internship in Beijing Jiuqi Zhitong Data Technology Co.,Ltd.In this period,the author have completed the whole process of system requirements analysis,design development and deployment on-line of the project.The author do the system general design and the detail design of accessing the log data to the system module,the search module,the data dashboard module,the alarm module.Finally,the author finish the function of the real-time acquisition of log data,real-time search,timing monitoring alarm and log data analysis dashboard.This paper firstly analyzes the requirements of the system according to the actual business requirements,and determines the functional requirements and nonfunctional requirements of the system.On the basis of this,the author completes the general design of the system.The server uses the Ali cloud server as the development test environment,and the system is diveded into the search module,access data into the middle layer module,the middle layer data access to the ElasticSearch index module,the dashboard module and the Alarm module.After completing the database design,I also do the system design and implementation of the detailed part of the module and I take the responsible for the functional description and class design,and responsible for the implementation of the key functions of the logic and the realization of the process of calling the relationship between the actual operation and the effect of a detailed description.After the system running on-line,for the massive log data search and analysis,it not only greatly enhance the efficiency of the maintenance personnel,but also save the staff time cost.By the alarm and dashboard module,they show the rich resources contained in the log data,and greatly enhance the monitoring of the system and equipment operation stability,and the location of the problem in the system becomes very clear.