Research On Bigdata Threat Perception System Base On Log Analysis

With the rapid development of Internet technology,network security issues are becoming more and more serious.Faced with increasingly complex network structures,cyber threats are becoming more diverse and destructive.The traditional security tools,such as Snort and firewall,matching and analyzing on a single log through rules,can not capture the full attack behaviors,often appear false positives and omission.Moreover,rule writing relies on the expert knowledge of experienced security technicians,so the system's generalization ability is weak and the implementation cost is high.In this paper,a big data threat perception system based on log analysis was proposed.The machine learning model was used to build two lines of defense of the system,so that the network threat can be perceived quickly and accurately.The interpretability of the system model was studied,and the prediction results and interpretability results were displayed.In the threat perception system,the training sample is the basis of the whole syste,and the completeness and validity of feature and accuracy of label are related to the performance of the whole system.This paper studied and designed a method for constructing behavioral,state features,and dimension reduction and parameter reduction methods for features.Aiming at the problem of inaccurate label and less labeling samples,a semi-supervised learning model based on adaptive loss was designed to re?label(purify)and label(expand)the samples.The specific work of the whole paper is as follows:1.The literature to summarize the research background and significance of the threat perception system and the research status of the corresponding technology was reviewed,and the design goals and ideas of the system were analyzed.2.The implementation principle of the key technologies were studied to realize the system and summarize its characteristics and use skills.Key technologies include big data platform technology,traditional log analysis techniques,machine learning models,interpretability models,and presentation layer technologies.3.The shortcomings of traditional construction features methods were analyzed,and the methods of constructing behavior features and state features were proposed,and the methods of reducing the dimension of sparse behavior features and the number of network parameters were proposed.the two problems in data acquisition was aslo analyzed in this paper,a semi-supervised learning model based on adaptive loss function was proposed,and the performance of the model was evaluated.4.The architecture,training and prediction process of threat perception system was designed.Log acquisition module,log matching module,log detection module,log analysis module and result display module are designed and implemented in detail.5.Through the experimental training and evaluation system,the reliability and stability of the system were guaranteed.And the process of system training,the idea of parameter adjustment and some experimental results were introduced,and finally the research results of the interpretability of the model were analyzed.Finally,this paper implements the big data threat perception system based on log analysis.Experiments show that the system can accurately and efficiently locate threats,and has the characteristics of low false alarm rate and missing alarm rate.The results of the model interpretability study,on the one hand,prove the reliability of the system,on the other hand,reveal the vulnerability of the system and the attack means of the attacker,and help the security personnel to manage the operation and maintenance server.