Security Data Compression And Fusion For Efficient Network Security Analysis

Network security analysis is one of the most important technical methods to manage and maintain cyberspace security.It analyzes host cardinality,network flow and network behavior in order to understand network communication status and measure the level of network security.However,with the continuous expansion of network scale,big-volume security-related data(in short security data)makes network security analysis inefficient in terms of analyzing and storing security data,which finally affects the accuracy of security analysis.Recently,data compression and fusion have been widely applied into network security analysis to eliminate the impact of big-volume security data by compressing and fusing data on demand.Data compression and fusion can reduce resource consumption when analyzing and storing big-volume security data,thus has great advantages in terms of time and space complexity.However,there are still a number of open issues related to data collection,storage,traceability and analysis when applying data compression and fusion into big-volume security data processing and analysis.First of all,current data compression and fusion models suffer from high false positives and high computational overhead when reconstructing abnormal addresses.Second,existing data compression and fusion models cannot support distributed and parallel host cardinality collection and analyze.Third,current data compression and fusion models cannot effectively balance between memory usage and accuracy of super host identification.Fourth,so far,data compression and fusion model based anomaly detection methods cannot realize adaptive and protocol-independent abnormal behavior detection.At last,we are still facing the challenges caused by the shortages of training data and the effects of heterogeneity of security data.Therefore,in order to improve the accuracy and efficiency of network security analysis,novel data compression and fusion models and data analysis methods should be proposed in order to ensure that network security analysis has the ability to effectively process and analyze big-volume security data.In order to solve the above open issues,we proposed a number of methods in this Ph D dissertation.In summary,the contributions of this dissertation are described below:(1)In order to address the problem of reversible calculation of compression and fusion model and the challenges brought by big-volume security data to abnormal behavior detection,we first propose a new two-dimensional compression and fusion model with low reversible computation complexity.Based on this model,we design two adaptive and protocol-independent abnormal behavior detection methods for detecting Distributed Denial of Service(DDo S)attacks and amplification attacks.The first method uses a modified multi-chart cumulative sum algorithm to dynamically monitor the changes of related security data that can fully describe the characteristics of DDo S attacks.The second method can accurately detect amplification attacks by monitoring the unbalanced relationship between request traffic and response traffic without collecting other traffic features.It realizes anomaly detection with high efficiency.(2)To address the challenges caused by big-volume security data to host cardinality analysis,based on the two-dimensional reversible model proposed in Contribution 1,we propose a three-dimensional reversible compression and fusion model to realize distributed and parallel analysis of host cardinality.This model has the following advantages: multi-dimensional,that is,it can monitor multiple categories of host cardinality in parallel and then identify multiple super hosts simultaneously;mergeable,it can collect security data in a distributed manner and merge them together for super host identification with the assurance of the integrity of host cardinality collection;invertible,it can accurately reconstruct the super host addresses.(3)To achieve economic consumption for storing huge host cardinality information,based on the compression and fusion model proposed in Contribution 2,we propose an extensible and reversible compression and fusion model to realize both memory efficiency and accuracy of host cardinality analysis.This model can dynamically adjust memory usage of data compression according to the distribution of host cardinality,so as to ensure efficient utilization of memory when monitoring low cardinality hosts and at the same time to provide accurate cardinality analysis in case that high cardinality hosts appear.Based on this model,we further propose an accurate and efficient method for identifying super hosts.(4)To address the challenges caused heterogeneity of data feature and the shortages of training data,we propose a new distance learning method based on data fusion with the purpose of improving the accuracy of data analysis model.The method can learn distance functions to fully reflect data characteristics according to Group-level prior knowledge that carries local information of data.Concretely,a linear distance function is learned through semi-positive definite optimization,and a nonlinear distance function that can deal with nonlinear data is realized by making use of the virtue of deep learning.The proposed method can mine background knowledge under the condition of limited prior knowledge,and make full use of the weight information of prior knowledge,thus greatly improving the accuracy of data analytics.