Adaptive Gibbs Sampling Method Based On Network

Today's network traffic presents a large-scale, high-speed characteristics, the development of software and hardware hasn't keep up with the network situation and update rate, so the whole collection of network data packets isn't possible. Traditional statistical methods and measurements aren't meet the current situation and face great challenge, then we are required to take an effective measure to reduce the load burden, thus sampling technique becomes a hotspot and important research direction in the Internet field, it can be used in intrusion detection, network measurement and other good development prospects directions.Sampling, by definition, is taking samples from the overall sample according to some rules or methods to reflect some of the key features, while the distribution function which the sample obey has a direct impact on the results of sampling. Network sampling, it's a process which extracting the key samples from the whole network data, then reflect the required information though analyzing and dealing with the sample data. Network sampling technique is a interdisciplinary, we can get the network traffic characteristics or key performance indicators by it, this could provide the necessary basis for the allocation of network resources and equipment. According to the proposal of PSAMP, the network packet sampling model should as simple as possible, and meet various application requirements, making the sample data can be used to network behavior research. However, the complexity of the network environment makes the traditional network model reveal its drawbacks gradually: such as bad randomness, adaptive capacity isn't strong, and it can't solve the high dimensional samples property and so on, thus the results can't completely reflect the network environment.This paper is departure from the existing shortcomings in the network sampling technique field, emphasizes the Gibbs sampling algorithm based on the Markov-Monte Carlo method (MCMC), and its network packet sampling environment application. In this paper, we define a network packet sampling algorithm (NPA-Gibbs) based on the Gibbs sampling algorithm, and it suitable for the typical network environment, at the same time, put the information entropy method into the choice of the key attributes, and in light of the relevant literature to develop an assessment strategy.The major research work includes three sections:The first part is the first two chapters of this paper. At first, we introduce the Internet large-scale, high-speed characteristics, and then describe the network sampling technology and measurement technology, including its composition, a simple classification and the classical method, in addition, also involve a number of algorithm evaluation criteria, emphasis on the randomness as an indicator of the sample results, and play an important role in the sampling technique.The second part is the third chapter. We introduce Markov chain-Monte Carlo method based on the theory of Markov chain and Monte Carlo method, and then further explain the principle and superiority of the Markov chain-Monte Carlo (MCMC) method, MCMC methods is a special Monte Carlo method, it adds the Markov process into the Monte Carlo simulation, so implement dynamic simulation (ie, sampling distribution changes with the simulation process), using the current value of random sampling generates the next sample value, then generates a Markov chain. Essentially, MCMC method is a kind of Markov chain Monte Carlo integration. MCMC algorithm for estimating should remove some iteration what used to before convergence, they only eliminate the impact of the initial value, and use the back value for the results. Finally, the focal point is the most classical Gibbs sampling algorithm which belongs to MCMC, describe the essence of their thinking and algorithm steps. Because of principle simple, randomness strong, research value significant, applications range broad, so use this algorithm as the core algorithm of this paper.The third part is the fourth and fifth chapters, and they are the focus chapters. After the Gibbs sampling algorithm is applied to network sampling, we carry out some improvements on it, propose network-based packet Gibbs sampling algorithm (NPA-Gibbs), which adapt to the current complex network situation better, then prove that this algorithm has certain advantages in theory and in experiment. This section explains the Gibbs sampling algorithm thinking, principle, the sampling method selection rules and the overall algorithm description. When selecting key attributes of network packets, we use the information entropy theory, and analyze the randomness of IP packet fields in line with it, the experimental evaluation show that the algorithm availability. Finally, in the summary and outlook section, we describe the consequent and shortage of this work as future reference.In this paper, we describe relevant content about the classic Gibbs sampling algorithm, illustrate it's essence that construct a Markov chain making Markov chain converges to the distribution of sample target. The biggest advantages of Gibbs sampling is simpler than MCMC other methods about simulate joint distribution. In this paper, the Gibbs sampling method distribution model are simulate normal and uniform distribution of the mathematical model, and make Gibbs sampling algorithm could adjust to segment case, the current segment's distribution can by a number of single-variable distribution simulate the joint distribution, so reduce the computational complexity. Through the efforts of many experts and scholars in the field of mathematics and physics, they have proved that Gibbs sampling can be widely used in statistical problems, and it's convergence is better than other sampling methods.In the experimental part of the algorithm, we use a period of time real network data set in order to show the authenticity of the experiment. Experimental evaluation of the algorithm, we first evaluate the randomness of the algorithm by use a typical evaluation criteria. In order to make the results easy to understand, we use open source software-WinBUGS to analyze and present results of the analysis; then we analyze the amount of information about the algorithm, compare the data sets what have been sampled with the original data sets about the information quantity and so on; Finally, the experiment shows NPA-Gibbs algorithm's advantage of the extraction rate and stability.This paper intends to apply Gibbs sampling algorithm to the network sampling environment, we obtain a new algorithm named NPA-Gibbs, the experiment show that the proposed algorithm owns availability during the analysis of a real network packets, if the model definition is appropriate, this algorithm could adapt to different network situation as complex, it could provide reference for intrusion and network measure, etc.Direction for the research sample, this work also appeared to be very narrow. By studying the Gibbs sampling algorithm and its application to the network data transmission system, to achieve reduction of the sample data, reducing the bandwidth burden of purpose. However, due to the diversity of network behavior, consider the case of this paper is quite narrow, an adaptive method of thinking, depending on network traffic to identify appropriate to select the field, thus faster, better meet the needs of the sampling system, The purpose is to select a sample of cases as little as possible down the overall reaction network traffic for intrusion detection, network measurement, etc.By studying the Gibbs sampling algorithm, we can reduce the sample data and extenuate the bandwidth burden. But at the direction of the sample research, this work also appears insufficient. Because of limited time and personal capacity, the paper isn't yet fully considered, so I hope that the subsequent researchers could improve the idea, and do more basic preparation for intrusion detection, network measurement, etc.