Network Traffic Anomaly Detection Based On Entropy Estimation

Because of its openness, nature of resource sharing and increasing interconnectivity, in particular, the emergence of the Internet, network is getting more important in all aspects of people's social lives. It is reported that various network attacks have made serious damages recently. Without built-in protection mechanisms in TCP/IP protocols, the networked computers are exposed to all kinds of network attacks. As an effective network information protection measure, Intrusion Detection Systems (IDS) become an important tool protecting network resources from being abused. An IDS using little resources, able to detect unknown attacks is promising in the market.This thesis discusses the application of network information entropy in detecting network traffic anomaly based on the analysis of attack characteristics. A hybrid algorithm is introduced to estimate the entropy of stream data, as an extension of Alon-Matia-Szegedy algorithm. This hybrid algorithm separates the high-frequency items and the low-frequency items in a data stream and calculate their contributions to the stream entropy respectively, giving a more accurate estimate of entropy than AMS algorithm.The experiments are designed with the DARPA dataset from MIT Lincoln Labs. Then, network information entropy is evaluated using the uniform random sampling, AMS algorithm and the proposed hybrid algorithm respectively. The effectiveness of the proposed algorithm is verified by the experiment results.