Online Adaptive Network Anomaly Detection System

With the extensive usage of computer networks, security becomes a critical issue. Network intrusions can cause severe disruption to networks. Therefore there is an urgent need for a solution that can actively defend networks against the growing security threats. The Intrusion Detection Systems (IDSs) can automatically scan network activity and recognize intrusion attacks to protect computers against unauthorized uses and make them secure and resistant to intruders. This is where network IDS comes in to offer security in addition to that provided by traditional anti-threat applications such as firewalls, antivirus software and spy-ware detection software. For the last decade, misuse detection has been the dominant strategy for IDSs for the reasons that it is easier to implement. However, anomaly detection attracts more and more interests to researchers because it has the advantage of detecting novel intrusions without any prior knowledge. In addition, most intrusion detection algorithms are lack of efficiency with the increasing traffic data in high speed networks. Also we require better performance and adaptive ability of intrusion detection technology because of the improvement of hacker technology and the changing of network application environment.In order to increase the running efficiency, enhance the self-adaptive ability, improve the detection rate and decrease the false positive rate, the dissertation focuses on the design of system model, the combination of both misuse detection technique and anomaly detection technique and the light weighted intrusion detection algorithms. And propose some innovative solutions as the following.1. A light weighted online adaptive network anomaly detection system model (OANAD) is proposed. The system can process network traffic data stream in real-time, gradually build up its local normal pattern base and intrusion pattern base under a few supervising of the administrator, and dynamically update the contents of the knowledge base according to the changing of the network application environments. The system architecture of OANAD exhibits the characteristics of both the anomaly based and misuse based intrusion detection systems. The model builds the network application patterns with a grid based form and the operating mechanism of the model is compact and efficient which makes the model qualified for online network intrusion detection.2. A novel pattern influence based anomaly detection algorithm (PIAD) is provided. The algorithm, which combines both misuse detection technique and anomaly detection technique, can detect not only the learned intrusion patterns but also the unseen intrusion patterns. It also greatly decreases the false positive rate while keeping a high detection rate in checking intrusions. The PIAD has the ability of online incremental learning by the employment of a lazy learning strategy. Also the detecting algorithm costs little computational time and memory space. We test the algorithm with the OANAD model on the KDD99 intrusion detection datasets. The system scans the training dataset and the testing dataset only once. Within 40 seconds our system finished the whole learning and checking tasks. The experimental results show that our model achieves a detection rate of 91.32% and a false positive rate of only 0.43%. It is the lowest false positive rate on the whole test set of KDD99 with the equivalent detection rate in the literature. It is also capable of detecting new type of intrusions.3. A novel clustering algorithm, named k-Cubes, is proposed for network anomaly detection and network application pattern clustering. The algorithm directly processes grid-based network application patterns. The number of clusters is automatically decided by dynamically merging and splitting of clusters. The semi-supervised version of k-Cubes is also presented. It takes advantage of a small portion of labeled samples to supervise the merging and splitting process. Detection rules are produced according to the clustering result. This method is suitable for processing large amount of high dimensional datasets with a lot of symbolic attribute values. It also requires a few inputting parameters. Experimental result on the KDD 99 intrusion detection datasets shows that the algorithm achieves a detection rate of 95.82% with a false positive rate of 1.25%, and it detects 15 out of 17 new types of intrusions.4. A supervised ISODATA clustering algorithm is proposed for network anomaly detection. We improve the original ISODATA algorithm mainly in three aspects. First, the modified algorithm can directly process mixed attributes of symbolic and numeric values. Second, the algorithm can process both labeled and unlabeled samples. The labeled samples are used to supervise the clustering process in the splitting stage. Third, the initial parameters needed to be input into the algorithm are reduced to only two. Experimental result on the KDD 99 intrusion detection datasets shows that this algorithm has high detection rate (95.62%) while maintaining a low false positive rate (1.29%).In order to verify the practicability of the proposed system model, we developed a prototype system of OANAD, which monitors the HTTP packages of a local network. The experimental results prove that this system model is practical and efficient.