Design And Implementation Of Host Anomaly Detection System Based On ELK

With the rapid development of computer technology,the network security and host security of enterprises and governments are facing the ever-changing challenges.However,the simple improvement of border protection technology has been unable to cope with the current severe security situation,so the in-depth defense technology of the local area network is also being taken seriously by people.Traditional intrusion detection methods mainly rely on signature recognition and regular matching.As machine learning becomes more and more popular in recent years,people pay more attention to using machine learning algorithm for intrusion detection in the current big data environment.However,the existing detection methods still have the disadvantages of long detection time,low detection accuracy and poor classification effect.Computer log records a large number of users' access and operation information to the system.If we use the log information generated by each host,we can clearly grasp the security status of each host in the current intranet.In this thesis,a cluster host anomaly detection method based on ELK is proposed.Through ELK to achieve the collection and management of logs,the main content is divided into data collection,data preprocessing,visual analysis,exception detection and alarm notification.The main work and innovation of this thesis are as follows:1.Based on Libpcap,C + + program is used to obtain the security data set for anomaly detection generated by network traffic.By comparing the general security data set,similar security data set for detection is extracted from the host,and the security data set and system log are simply filtered.2.After different data sources are marked by the filebeat installed on the host,they are transmitted to the server installed with logstash for regular matching,and preprocessing methods such as field replacement and normalization are carried out for the secure dataset protocol.Index the data document through elastic search and use kibana to build the chart according to the index results.At the same time,the logstash node is equipped with an exception detection mechanism.The hierarchical exception detection model is established by the improved self convergence PCA and OCSVM algorithm.When an exception is detected,the detection results can be sent to the internal mailbox of the manager.3.The implementation of the anomaly detection module is mainly based on the multi-level anomaly detection model constructed by the combination of self convergence PCA and OCSVM algorithm.Through the use of attack feature analysis method to preprocess the collected data set,at the same time,data cleaning and feature selection are carried out for the training set.The improved self convergence PCA algorithm can significantly reduce the noise contained in the data.The difference between abnormal data and normal data is highlighted,and the influence of irrelevant features on training model is weakened.The improvement goal of PCA is to reduce the noise in the data set quantitatively.By calculating the minimum characteristic contribution rate m,the optimal dimension of PCA dimension reduction can be obtained automatically.It not only improves the traditional method which needs frequent input of k value,but also has more pertinence to noise filtering.4.Through different data characteristics training different models correspond to DOS,R2 L,U2R and Probe attack types respectively.The OCSVM algorithm based on RBF kernel is used for model training,and the defect of poor classification effect of OCSVM algorithm is avoided by the improved hierarchical detection framework.In the end of this thesis,KDDCUP99 data set is extracted,and more attack features are tested after feature screening,and compared with the previous detection research using KDDCUP99 data set,the results show that the method proposed in this thesis is more superior.The main work of this thesis is distributed in data collection,data preprocessing optimization,traditional algorithm improvement for security requirements,exception detection system structure design optimization and so on.Based on the open source ELK distributed log management scheme,an anomaly detection scheme with hierarchical self convergence PCAOCSVM as the core is added.